Receiving Events

A few things to consider when receiving events from a Data Connector.

What Is Sent?

Each event is sent as a JSON body in an HTTPS POST request to the Endpoint URL. An example touch event is shown below. See our Events Reference to read about an event's anatomy and different types of events that can be expected from different devices.

{
"event": {
"eventId": "bboqciu55u1g00c0g9n0",
"targetName": "projects/bbbk89v86c6000c19pbg/devices/bapo55k1hbj000f5l0ig",
"eventType": "touch",
"data": {
"touch": {
"updateTime": "2018-05-08T13:29:47.543486780Z"
}
},
"timestamp": "2018-05-08T13:29:47.543483560Z"
},
"labels": {
"name": "Coffee machine service button"
}
}

Acknowledging Received Event

A request-reply flow on the Endpoint URL should be implemented as follows:

  1. Your endpoint receives an HTTPS POST request.

  2. Your service processes the data in some way.

  3. Your service replies to the event request with a 200 OK response.

What is important to note here is that the request should never return an HTTP 200 OK response before you are done processing it. When our cloud receives a 200 OK, the event will be taken off the internal Data Connector queue and checked off as received.

Best Practice

Do not reply 200 OK until you have finished processing your data.

Note that the status code range 2xx will also be accepted as OK in the response.

Verifying Signed Events

When using a Signature Secret, the header X-Dt-Signature is included with each event which contains a JSON Web Token (JWT), signed by the Signature Secret. Inside, a checksum of the request body can be found and used to check for tampering.

The following steps sum up the process of verifying the received request at the receiving endpoint.

  1. Extract the signed JWT from the HTTP header X-Dt-Signature of the received request.

  2. Verify the JWT's signature with the Signature secret.

  3. Calculate a SHA1 checksum over the entire request body.

  4. Compare the body checksum with the checksum contained in the JWT.

  5. If these checksums are identical, you can be certain that the event has not been tampered with and originated from your Data Connector.

The following snippet implements the verification process. It is written to run on a Google Cloud Function.

Python 3.8
Node.js 14
Python 3.8
import os
import hashlib
import jwt # pip install pyjwt==2.0.1
# Fetch environment variables.
SIGNATURE_SECRET = os.environ.get('SIGNATURE_SECRET')
def verify_request(body, token):
# Decode the token using signature secret.
try:
payload = jwt.decode(token, SIGNATURE_SECRET, algorithms=["HS256"])
except Exception as err:
print(err)
return False
# Verify the request body checksum.
m = hashlib.sha1()
m.update(body)
checksum = m.digest().hex()
if payload["checksum"] != checksum:
print('Checksum Mismatch')
return False
return True
def dataconnector_endpoint(request):
# Extract necessary request information.
body = request.get_data()
token = request.headers['x-dt-signature']
# Validate request origin and content integrity.
if not verify_request(body, token):
return ('Bad request', 400)
#
# Further processing here.
#
return ('OK', 200)
Node.js 14
const crypto = require('crypto')
const jwt = require('jsonwebtoken') // npm install [email protected]
// Fetch environment variables
const signatureSecret = process.env.SIGNATURE_SECRET
function verifyRequest(body, token) {
// Decode the token using signature secret.
let decoded
try {
decoded = jwt.verify(token, signatureSecret)
} catch(err) {
console.log(err)
return false
}
// Verify the request body checksum.
let shasum = crypto.createHash('sha1')
let checksum = shasum.update(body).digest('hex')
if (checksum !== decoded.checksum) {
console.log('Checksum Mismatch')
return false
}
return true
}
exports.dataconnectorEndpoint = (req, res) => {
// Extract necessary request information.
let body = JSON.stringify(req.body)
let token = req.get('X-Dt-Signature')
// Validate request origin and content integrity.
if (verifyRequest(body, token) === false) {
res.sendStatus(400)
return
}
//
// Further processing here.
//
res.sendStatus(200);
};

Handling Duplicates

Every event received by DT Cloud is put in a dedicated, per-Data Connector queue. Messages are removed from this queue once acknowledged, or if the message is older than 12 hours. This means that if your endpoint goes offline for a while, you will still receive the data when it comes back up again because the Data Connector will retransmit each message for up to 12 hours.

An important side effect of this delivery guarantee is that, under certain conditions, you may receive duplicates of the same event. This is rare but should be considered on the receiving end by checking the unique event id in the body.

Best Practice

While rare, use the included eventId field to check for duplicated events.

Retry policy

The Data Connector will retry anytime it does not receive a successful (HTTP 200 OK) response from your endpoint. The retry interval is calculated by an exponential back-off policy, given by

t02n1,t_0\cdot2^{n-1},

where t0t0 is the initial interval of 8 seconds and nn the retry counter. The interval will not exceed 1 hour. For very slow endpoints, the minimum retry interval will be 4x4x the response time.

Attempt

Retry Interval [s]

1

8

2

16

3

32

...

...

9

2048