Heroku

An example integration for forwarding Data Connector events to a server hosted on Heroku.

Overview

This example uses a Data Connector to forward the events of all devices in a project to a server hosted on Heroku. When receiving the HTTPS POST request, our application will verify both the origin and content of the request using a Signature Secret, then decode the data.

Prerequisites

The following points are assumed.

Heroku

Ensure you have the following software installed on your local machine.

  • Git, the popular version control tool.

  • Heroku CLI, for interacting with Heroku.

  • A functioning local environment for the language and version you choose to use.

Create a New App Locally

Locally on your machine, create and ender a new directory for your app.

mkdir my-app
cd my-app
Python 3.9
Python API (Beta)
Node.js 16
Go 1.16
Python 3.9

In the directory, create a new file app.py where you paste the following snippet which contains the Flask server code for receiving and validating the Data Connector request.

import os
import hashlib
import jwt
from flask import Flask, request
app = Flask(__name__)
# Fetch environment variables.
SIGNATURE_SECRET = os.environ.get('DT_SIGNATURE_SECRET')
def verify_request(body, token):
# Decode the token using signature secret.
try:
payload = jwt.decode(token, SIGNATURE_SECRET, algorithms=["HS256"])
except Exception as err:
print(err)
return False
# Verify the request body checksum.
m = hashlib.sha1()
m.update(body)
checksum = m.digest().hex()
if payload["checksum"] != checksum:
print('Checksum Mismatch')
return False
return True
@app.route('/', methods=['POST'])
def endpoint():
# Extract necessary request information.
body = request.data
token = request.headers['x-dt-signature']
# Validate request origin and content integrity.
if not verify_request(body, token):
return ('Could not verify request.', 400)
# Print the request body.
print(body)
return ('OK', 200)

In the same directory, create a new file requirements.txt with the following contents. Heroku will install these dependencies in its environment before starting the server.

requirements.txt
gunicorn==20.1.0
flask==2.0.1
pyjwt==2.1.0

Finally, specify the command Heroku should run by adding a Procfile with the following snippet. This instructs Heroku that the server should run a single process of type web, where Gunicorn should be used to serve the app using a variable called app in a module called app (app.py)

Procfile
web: gunicorn app:app
Python API (Beta)

In the directory, create a new file app.py where you paste the following snippet which contains the Flask server code for receiving, validating, and decoding the Data Connector request.

app.py
import os
from flask import Flask, request
from dtintegrations import data_connector, provider
app = Flask(__name__)
@app.route('/', methods=['POST'])
def endpoint():
# Use the provider-specific validation function.
event = data_connector.http_push.decode_request(
request,
provider=provider.FLASK,
secret=os.getenv('DT_SIGNATURE_SECRET'),
)
# Print the event data.
print(event)
# If all is well, return 200 response.
return 'Success'

In the same directory, create a new file requirements.txt with the following contents. Heroku will install these dependencies in its environment before starting the server.

requirements.txt
gunicorn==20.1.0
flask==2.0.1
dtintegrations

Finally, specify the command Heroku should run by adding a Procfile with the following snippet. This instructs Heroku that the server should run a single process of type web, where Gunicorn should be used to serve the app using a variable called app in a module called app (app.py)

Procfile
web: gunicorn app:app
Node.js 16

In the directory, initialize a node application with the following command. This generates the file package.json which contains information used by Heroku when deploying the app.

npm init

Create a new file index.js where you paste the following snippet which contains the Express server code for receiving and validating the Data Connector request.

const express = require("express")
const bodyParser = require("body-parser")
const crypto = require('crypto')
const jwt = require('jsonwebtoken')
// Create the express server.
const app = express()
app.use(bodyParser.urlencoded({ extended: false }))
app.use(bodyParser.json())
// Fetch environment variables.
const signatureSecret = process.env.DT_SIGNATURE_SECRET
function verifyRequest(body, token) {
// Decode the token using signature secret.
let decoded;
try {
decoded = jwt.verify(token, signatureSecret)
} catch(err) {
console.log(err)
return false
}
// Verify the request body checksum.
let shasum = crypto.createHash('sha1')
let checksum = shasum.update(JSON.stringify(body)).digest('hex')
if (checksum !== decoded.checksum) {
console.log('Checksum Mismatch')
return false
}
return true
}
app.post("/", (req, res) => {
// Extract necessary request information.
let body = req.body
let token = req.get('X-Dt-signature')
// Validate request origin and content integrity.
if (verifyRequest(body, token) === false) {
res.sendStatus(400)
return
}
// Log the request body.
console.log(body)
res.sendStatus(200)
})
app.listen(process.env.PORT)

In the same directory, install the following packages. This updates package.json so that Heroku can replicate the environment when deploying.

npm install body-parser==1.19.0
npm install express==4.17.1
npm install jsonwebtoken==8.5.1

Create a file Procfile with the following snippet. This instructs Heroku that the server should run a single process of type web, where node index.js should be used to serve the app.

web: node index.js
Go 1.16

In the directory, initialize a go application with the following command. This generates the file go.mod which specifies version and requirements. Give any name you like.

go mod init example.com/m/v2

Create a new file main.go where you paste the following snippet which contains the server code for receiving and validating the Data Connector request.

package main
import (
"crypto/sha1"
"encoding/hex"
"fmt"
jwt "github.com/dgrijalva/jwt-go" // go get github.com/dgrijalva/[email protected]
"io/ioutil"
"log"
"net/http"
"os"
)
// Environment variables.
var signatureSecret = os.Getenv("DT_SIGNATURE_SECRET")
// verifyRequest validates the request origin and content integrity.
func verifyRequest(bodyBytes []byte, tokenString string) error {
// Decode the token using signature secret.
claims := jwt.MapClaims{}
_, err := jwt.ParseWithClaims(tokenString, claims, func(token *jwt.Token) (interface{}, error) {
return []byte(signatureSecret), nil
})
if err != nil {
return err
}
// Verify the request body checksum.
sha1Bytes := sha1.Sum(bodyBytes)
sha1String := hex.EncodeToString(sha1Bytes[:])
if sha1String != claims["checksum"] {
return fmt.Errorf("Checksum mismatch.")
}
return nil
}
func endpoint(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/" {
http.Error(w, "404 not found.", http.StatusNotFound)
return
}
// Take an action depending on the request method.
switch r.Method {
case "POST":
if err := r.ParseForm(); err != nil {
fmt.Fprintf(w, "ParseForm() err: %v", err)
return
}
// Attempt to extract the body bytes of the request.
body, err := ioutil.ReadAll(r.Body)
if err != nil {
log.Printf("Error reading body: %v", err)
http.Error(w, "Can't read body", http.StatusBadRequest)
return
}
// Validate the origin and contents of the request.
if err = verifyRequest(body, r.Header.Get("X-Dt-Signature")); err != nil {
log.Printf("Could not validate request: %v", err)
http.Error(w, "Could not validate request", http.StatusBadRequest)
return
}
//
// Further processing here.
//
log.Println("Success")
default:
http.Error(w, "Only POST methods are allowed.", http.StatusMethodNotAllowed)
}
}
func main() {
http.HandleFunc("/", endpoint)
port := ":" + string(os.Getenv("PORT"))
log.Printf("Listening on port %s.", port)
if err := http.ListenAndServe(port, nil); err != nil {
log.Fatal(err)
}
}

In the same directory, run the following command to install the required package.

go get github.com/dgrijalva/[email protected]

Build your application into a new directory bin.

mkdir bin
GOOS=linux CGO_ENABLED=0 go build -o bin/main -v .

Finally, specify the command Heroku should run by adding a Procfile with the following snippet. This instructs Heroku that the server should run a single process of type web, where the app should be server from the binary bin/main that we built in the previous step.

Procfile
web: bin/main

Deploy the App

The following steps will deploy your application to a new app in Heroku. Take note of the app name given to the application in step 4. This can be changed later if so desired.

  1. Initialize your app's directory as a Git repository: git init

  2. Add all changes: git add .

  3. Commit the changes: git commit -m "initial commit"

  4. Create the Heroku app, and add it as a Git remote: heroku create <APP_NAME>. You can pick a unique app name yourself or let Heroku create a random one by omitting <APP_NAME>. This will be used as a part of the URL for your server.

  5. Push your application to Heroku: git push heroku main

Post Deployment Configuration

Add a new signature secret configuration variable to your app using the following command. This can be read by the python instance and imported as an environment variable to use. Use a strong secret.

heroku config:set -a <APP_NAME> DT_SIGNATURE_SECRET=<YOUR_SECRET>

Your app is now ready to receive requests, but we need to know to which URL to send them. This can be acquired by looking at the Web URL field in the results of the following command. Save it for later.

heroku apps:info -a <APP_NAME>

Create a Data Connector

To continuously forward the data to our newly created Heroku server, a Data Connector with almost all default settings is sufficient. If you are unfamiliar with how Data Connectors can be created, refer to our Creating a Data Connector guide. The following configurations should be set.

  • Endpoint URL: The Web URL found in the previous step.

  • Signature Secret: The value of DT_SIGNATURE_SECRET environment variable.

Depending on your integration, it can also be smart to disable the event types you are not interested in. For instance, the NetworkStatusEvent is sent every Periodic Heartbeat and will by default be forwarded by the Data Connector if not explicitly unticked.

Test the Integration

If the integration was correctly implemented, the Success counter for your Data Connector should increment for each new event forwarded. This happens each Periodic Heartbeat or by touching a sensor to force a new event.

If instead the Error counter increments, a response containing a non-2xx status code is returned.

  • Verify that the Data Connector endpoint URL is correct.

  • You can view the logs of your Heroku app with the following command.

    heroku logs --app=<APP_NAME> --tail