Advanced Configurations
Data Connectors can through configuration be tailored to fit more custom use-cases.
Before diving into how to configure a Data Connector, there are a few things worth keeping in mind when working with them.
- Multiple Projects Data Connectors are created and configured on a per-project basis. If you want multiple projects to send data to the same Endpoint URL, you have to create one Data Connector per project.
- Access Control To create, update, and delete a Data Connector, your DT Studio User or Service Account must have the role of Project Developer or higher.
- Multiple Data Connectors All Data Connectors in a project will receive events from all the devices in that project. It is recommended to filter to event types when possible.
It is possible to enabled and disable a Data Connector at will.
DT Studio
REST API
Python API
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then toggle the Data Connector enabled switch. Remember to click Update Data Connector.
Send a PATCH request to:
https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>A request body with the following parameter should be included.
{
"status": "ACTIVE"
}
The status field takes either
"ACTIVE" or "USER_DISABLED".Using cURL with a Service Account for authentication, the following example disables the speficied Data Connector by setting
status to "USER_DISABLED".curl -X PATCH "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>" \
-u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
-d '{"status": "USER_DISABLED"}'
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example disables the specified Data Connector by setting
status to "USER_DISABLED".import disruptive as dt
# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
key_id='<SERVICE_ACCOUNT_KEY_ID>',
secret='<SERVICE_ACCOUNT_SECRET>',
email='<SERVICE_ACCOUNT_EMAIL>',
)
# Disabled Data Connector by updating the status.
dcon = dt.DataConnector.update_data_connector(
data_connector_id='<DATA_CONNECTOR_ID>',
project_id='<PROJECT_ID>',
status='USER_DISABLED',
)
# Print the updated Data Connector.
print(dcon)
Disabled Data Connectors have no at-least-once guarantee.
When a Data Connector is disabled, undelivered events and events generated after being disabled will not be sent. Re-enabling the Data Connector will not backfill data from the period it was disabled and must be fetched programmatically using our REST API.
A Data Connector will be automatically disabled by DT Cloud if it has failed for an extended period of time without any success. A push-attempt from a Data Connector is considered a failure if the receiver responded with a non-200 status code, or didn’t respond at all. A Data Connector that is disabled by DT Cloud will get the status
SYSTEM_DISABLED.By default, all event types are forwarded by a Data Connector. If you want to avoid uneccesary traffic on your endpoint, events can be filtered by type.
DT Studio
REST API
Python API
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then toggle the Forward All Events followed by a selection. Remember to click Update Data Connector.
Send a PATCH request to:
https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>A request body with the following parameter should be included.
{
"events": ["touch", "temperature", ...]
}
Using cURL with a Service Account for authentication, the following example sets a filter so that only temperature- and humidity events are forwarded.
curl -X PATCH "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>" \
-u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
-d '{"events": ["temperature", "humidity"]}'
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example sets a filter so that only temperature- and humidity events are forwarded.
import disruptive as dt
# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
key_id='<SERVICE_ACCOUNT_KEY_ID>',
secret='<SERVICE_ACCOUNT_SECRET>',
email='<SERVICE_ACCOUNT_EMAIL>',
)
# Update the Data Connector to only forward specified events.
dcon = dt.DataConnector.update_data_connector(
data_connector_id='<DATA_CONNECTOR_ID>',
project_id='<PROJECT_ID>',
event_types=[
dt.events.TEMPERATURE,
dt.events.HUMIDITY,
],
)
# Print the updated Data Connector.
print(dcon)
You can include labels held by a device with each event forwarded by the Data Connector.
DT Studio
REST API
Python API
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then append the label keys to the list. Remember to click Update Data Connector.
Send a PATCH request to:
https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>A request body with the following parameter should be included.
{
"labels": ["label-01", "label-02", ...]
}
Using cURL with a Service Account for authentication, the following example configures the specified Data Connector to include device labels called
"floor" and "customer-ID".curl -X PATCH "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>" \
-u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
-d '{"labels": ["floor", "customer-ID"]}'
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example configured a Data Connector to include device labels called
"floor" and "customer-ID".import disruptive as dt
# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
key_id='<SERVICE_ACCOUNT_KEY_ID>',
secret='<SERVICE_ACCOUNT_SECRET>',
email='<SERVICE_ACCOUNT_EMAIL>',
)
# Update the Data Connector to include device labels.
dcon = dt.DataConnector.update_data_connector(
data_connector_id='<DATA_CONNECTOR_ID>',
project_id='<PROJECT_ID>',
labels=['floor', 'customer-ID'],
)
# Print the updated Data Connector.
print(dcon)
DT Studio uses labels with the keys
name and description to show as the display name and the description of a device. It is recommended that an integration uses these labels in the same way when displaying or updating a device.Custom HTTP headers can be included with each event forwarded by a Data Connector.
DT Studio
REST API
Python API
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then add a key- and value pairs for your custom headers. Remember to click Update Data Connector.
Send a PATCH request to:
https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>A request body with the following parameter should be included.
{
"httpConfig": {
"headers": {
"name-1": "value-1",
"name-2": "value-2",
...
}
}
}
Using cURL with a Service Account for authentication, the following example adds the header
"my-header" with value "my-value" to every event forwarded by a Data Connector.curl -X PATCH "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>" \
-u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
-d '{"httpConfig": {"headers": {"my-header": "my-value"}}}'
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example adds the header
"my-header" with value "my-value" to every forwarded event.import disruptive as dt
# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
key_id='<SERVICE_ACCOUNT_KEY_ID>',
secret='<SERVICE_ACCOUNT_SECRET>',
email='<SERVICE_ACCOUNT_EMAIL>',
)
# Update the Data Connector to include custom HTTP headers.
dcon = dt.DataConnector.update_data_connector(
data_connector_id='<DATA_CONNECTOR_ID>',
project_id='<PROJECT_ID>',
config=dt.DataConnector.HttpPushConfig(
headers={
'my-header': 'my-value',
},
),
)
# Print the updated Data Connector.
print(dcon)
For increased security in a production integration, we recommend signing each Event with a Signature Secret that can be verified on the receiving end.
- The JWT payload contains the checksum of the request body.
This allows you to both validate the origin and content integrity of the received request for a more secure connection. The process is two-fold and must be configured for both the Data Connector and endpoint.
Adding a signature secret to your Data Connector will automatically sign each forwarded event.
DT Studio
REST API
Python API
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then add a strong and unique signature secret. Remember to click Update Data Connector.
Send a PATCH request to:
https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>A request body with the following parameter should be included.
{
"httpConfig": {
"signatureSecret": "some-good-secret"
}
}
Using cURL with a Service Account for authentication, the following example adds a signature secret to the specified Data Connector.
curl -X PATCH "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>" \
-u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>" \
-d '{"httpConfig": {"signatureSecret": "some-good-secret"}}
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be updated by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example adds a signature secret to the specified Data Connector.
import disruptive as dt
# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
key_id='<SERVICE_ACCOUNT_KEY_ID>',
secret='<SERVICE_ACCOUNT_SECRET>',
email='<SERVICE_ACCOUNT_EMAIL>',
)
# Update the Data Connector with a signature secret.
dcon = dt.DataConnector.update_data_connector(
data_connector_id='<DATA_CONNECTOR_ID>',
project_id='<PROJECT_ID>',
config=dt.DataConnector.HttpPushConfig(
signature_secret='<YOUR_SECURE_SECRET>',
),
)
# Print the updated Data Connector.
print(dcon)
When synchronizing the Data Connector, the state of each device currently in the project will be sent as events. These events will be dated back in time to when they were reported from the device. Note, however, that the event id will be different for the synchronized events.
DT Studio
REST API
Python API
In DT Studio, click on API Integrations -> Data Connectors. Select the desired Data Connector, then click Synchronize Data Connector towards the bottom of the page.
Send a POST request to:
https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR>:syncUsing cURL with a Service Account for authentication, the following example syncs the specified Data Connector.
curl -X POST "https://api.d21s.com/v2/projects/<PROJECT_ID>/dataconnectors/<DATA_CONNECTOR_ID>:sync" \
-u "<SERVICE_ACCOUNT_KEY_ID>":"<SERVICE_ACCOUNT_SECRET>"
Once the package is installed and authenticated as described in the Python API Reference, a Data Connector can be synced by calling the following resource method.
Using our Python API with Service Account credentials for authentication, the following example syncs the specified Data Connector.
import disruptive as dt
# Authenticate the package using Service Account credentials.
dt.default_auth = dt.Auth.service_account(
key_id='<SERVICE_ACCOUNT_KEY_ID>',
secret='<SERVICE_ACCOUNT_SECRET>',
email='<SERVICE_ACCOUNT_EMAIL>',
)
# Synchronize the specified Data Connector.
dt.DataConnector.sync_data_connector(
data_connector_id='<DATA_CONNECTOR_ID>',
project_id='<PROJECT_ID>',
)