Overview

To illustrate the process, we will here create a new Data Connector that forwards all events to dweet.io, a free hosted service that fulfills the same role as a server. If you already have a hosted HTTPS POST endpoint set up, feel free to use that instead.

Prerequisites

• Service Account Creating, deleting, and interacting with Data Connectors require that your User or Service Account has been granted the role of Project Developer or higher.

• HTTPS Endpoint Data Connectors sends events as POST requests over HTTPS. In order to receive these events you need to have a service up and running that can accept incoming HTTPS connections, HTTP is not supported. In this guide, we will use dweet.io as an example.

Creating a New Data Connector

Data Connectors can be created using either the DT Studio interface or our REST API. For simplicity, most of the configurations will be left default in this example. For those interested in a more custom Data Connector, the Advanced Configurations page contains the explanations for the rest.

DT Studio

REST API

Python API (Beta)

DT Studio

In DT Studio, navigate to your Project. In the left menu, locate Data Connectors and press Add Data Connector. This will open the Data Connector configuration menu.

Edit the following parameters.

• Display name Give the Data Connector some identifiable display name.

• Endpoint URL The URL to which the Data Connector will forward events. As we use dweet.io in this example, simply use an URL on the form https://dweet.io/dweet/for/<SOME_UNIQUE_NAME>.

Remember to save your Data Connector configuration at the bottom of the page.

REST API

Send a POST request to:

https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors

A request body with the following parameters is required.

{
    "type": "HTTP_PUSH",
    "httpConfig": {
        "url": "<ENDPOINT_URL>"
    }
}

A list of all available parameters can be found in our REST API Reference.

Example Usage

Using cURL with a Service Account for authentication, the following example creates a new Data Connector with a given name "new-dcon" and a dweet.io endpoint which can be monitored at https://dweet.io/follow/<SOME_UNIQUE_NAME>.

curl -X POST "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors" \
    -u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
    -d '{"displayName": "new-dcon", "type": "HTTP_PUSH", "httpConfig": {"url": "https://dweet.io/dweet/for/<SOME_UNIQUE_NAME>"}}'

Python API (Beta)

Once the package is installed and authenticated as described in the Python API Reference, a new Data Connector can be created by calling the following resource method.

• ​disruptive.DataConnector.create_data_connector()​

Example Usage

Using our Python API with Service Account credentials for authentication, the following example creates a new Data Connector with a given name "new-dcon" and a dweet.io endpoint which can be monitored at https://dweet.io/follow/<SOME_UNIQUE_NAME>.

import disruptive as dt
​
# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)
​
# Create a new HTTP Push Data Connector called "new-dcon".
new_dcon = dt.DataConnector.create_data_connector(
    project_id='<PROJECT_ID>',
    config=dt.DataConnector.HttpPushConfig(
        url='https://dweet.io/dweet/for/<SOME_UNIQUE_NAME>',
    ),
    display_name='new-dcon',
)
​
# Print the newly created Data Connector.
print(new_dcon)

Test Your Data Connector

The simplest way of testing your connection is by forwarding an event in one of the following ways.

• Wait for a sensor in the project to naturally emit an event by its periodic heartbeat.

• Touching a sensor to force an emitted event.

• Use the Sensor Emulator to emulate an emitted event.

For dweet.io, navigate to your generated URL, now on the form https://dweet.io/follow/<SOME_UNIQUE_NAME>. Under the Raw tab, the event body should become visible as they are forwarded. Any errors will also be shown here.

When you are certain an event has been emitted, the Data Connector metrics will be iterated.

DT Studio

REST API

Python API (Beta)

DT Studio

The metrics can be found under the same page on which your Data Connector was created.

REST API

Send a GET request to:

https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATACONNECTOR>:metrics

The response contains the metrics for the specified Data Connector over the last 3 hours.

{
  "metrics": {
    "successCount": 44,
    "errorCount": 0,
    "latency99p": "0.663s"
  }
}

Example

Using cURL with a Service Account for authentication, the following example requests the metrics for the specified Data Connector in the specified Project.

curl -X GET "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>:metrics" \
    --user "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>"

Python API (Beta)

Once the package is installed and authenticated as described in the Python API Reference, Data Connector metrics can be fetched by calling the following resource method.

• ​disruptive.DataConnector.get_metrics()​

Example Usage

Using our Python API with Service Account credentials for authentication, the following example requests the metrics for the specified Data Connector in the specified Project.

import disruptive as dt
​
# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
    key_id='<SERVICE_ACCOUNT_KEY_ID>',
    secret='<SERVICE_ACCOUNT_SECRET>',
    email='<SERVICE_ACCOUNT_EMAIL>',
)
​
# Fetch the metrics of the specified Data Connector.
metrics = dt.DataConnector.get_metrics(
    data_connector_id='<DATA_CONNECTOR_ID>',
    project_id='<PROJECT_ID>',
)
​
# Print the content of metrics.
print(metrics)

If the Error counter is iterated instead, this indicates that a status code other than 2xx has been returned.

• Verify that your Data Connector configuration, especially the Endpoint URL, is correct.

• View the logs on your endpoint which should contain the error.