A quick guide on how to create a Data Connector using DT Studio or our APIs.
Overview
To illustrate the process, we will here create a new Data Connector that forwards all events to dweet.io, a free hosted service that fulfills the same role as a server. If you already have a hosted HTTPS POST endpoint set up, feel free to use that instead.
Prerequisites
Service Account
Creating, deleting, and interacting with Data Connectors require that your User or Service Account has been granted the role of Project Developer or higher.
HTTPS Endpoint
Data Connectors sends events as POST requests over HTTPS. In order to receive these events you need to have a service up and running that can accept incoming HTTPS connections, HTTP is not supported. In this guide, we will use dweet.io as an example.
Please note that Disruptive Technologies and dweet.io are in no way affiliated and that the data sent there is publicly available.
Creating a New Data Connector
Data Connectors can be created using either the DT Studio interface or our REST API. For simplicity, most of the configurations will be left default in this example. For those interested in a more custom Data Connector, the Advanced Configurations page contains the explanations for the rest.
DT Studio
REST API
Python API
In DT Studio, navigate to your Project. In the left menu, locate Data Connectors and press Add Data Connector. This will open the Data Connector configuration menu.
Edit the following parameters.
Display name
Give the Data Connector some identifiable display name.
Endpoint URL
The URL to which the Data Connector will forward events. As we use dweet.io in this example, simply use an URL on the form https://dweet.io/dweet/for/<SOME_UNIQUE_NAME>.
Remember to save your Data Connector configuration at the bottom of the page.
A request body with the following parameters is required.
1
{
2
"type":"HTTP_PUSH",
3
"httpConfig":{
4
"url":"<ENDPOINT_URL>"
5
}
6
}
Copied!
A list of all available parameters can be found in our REST API Reference.
Example Usage
Using cURL with a Service Account for authentication, the following example creates a new Data Connector with a given name "new-dcon" and a dweet.io endpoint which can be monitored at https://dweet.io/follow/<SOME_UNIQUE_NAME>.
1
curl -X POST "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors"\
Once the package is installed and authenticated as described in the Python API Reference, a new Data Connector can be created by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example creates a new Data Connector with a given name "new-dcon" and a dweet.io endpoint which can be monitored at https://dweet.io/follow/<SOME_UNIQUE_NAME>.
1
import disruptive as dt
2
​
3
# Authenticate the package using Service Account credentials.
4
dt.default_auth = dt.Auth.service_account(
5
key_id='<SERVICE_ACCOUNT_KEY_ID>',
6
secret='<SERVICE_ACCOUNT_SECRET>',
7
email='<SERVICE_ACCOUNT_EMAIL>',
8
)
9
​
10
# Create a new HTTP Push Data Connector called "new-dcon".
For dweet.io, navigate to your generated URL, now on the form https://dweet.io/follow/<SOME_UNIQUE_NAME>. Under the Raw tab, the event body should become visible as they are forwarded. Any errors will also be shown here.
When you are certain an event has been emitted, the Data Connector metrics will be iterated.
DT Studio
REST API
Python API
The metrics can be found under the same page on which your Data Connector was created.
The response contains the metrics for the specified Data Connector over the last 3 hours.
1
{
2
"metrics":{
3
"successCount":44,
4
"errorCount":0,
5
"latency99p":"0.663s"
6
}
7
}
Copied!
Example
Using cURL with a Service Account for authentication, the following example requests the metrics for the specified Data Connector in the specified Project.
1
curl -X GET "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>:metrics"\
Once the package is installed and authenticated as described in the Python API Reference, Data Connector metrics can be fetched by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example requests the metrics for the specified Data Connector in the specified Project.
1
import disruptive as dt
2
​
3
# Authenticate the package using Service Account credentials.
4
dt.default_auth = dt.Auth.service_account(
5
key_id='<SERVICE_ACCOUNT_KEY_ID>',
6
secret='<SERVICE_ACCOUNT_SECRET>',
7
email='<SERVICE_ACCOUNT_EMAIL>',
8
)
9
​
10
# Fetch the metrics of the specified Data Connector.
11
metrics = dt.DataConnector.get_metrics(
12
data_connector_id='<DATA_CONNECTOR_ID>',
13
project_id='<PROJECT_ID>',
14
)
15
​
16
# Print the content of metrics.
17
print(metrics)
Copied!
If the Error counter is iterated instead, this indicates that a status code other than 2xx has been returned.
Verify that your Data Connector configuration, especially the Endpoint URL, is correct.
View the logs on your endpoint which should contain the error.