Google Cloud Functions
An example integration where a Data Connector forwards events to a Google Cloud Function.

Overview

This example uses a Data Connector to forward the events of all devices in a project to a Google Cloud Function. When receiving the HTTPS POST request, our function will verify both the origin and content of the request using a signature secret, then decode the data.

Prerequisites

The following points are assumed.

Google Cloud Platform

While there are many advantages to using a local environment for development, this guide will be using the browser portal to minimize setup requirements.

Create a Cloud Function

Following this guide, create a new Cloud Function with the following configurations.
Python 3.9
Python API
Node.js 14
Go 1.13

(1) Configuration

Trigger

    Trigger Type: HTTP
    Authentication: Allow unauthenticated invocations.

Variables, Networking and Advanced Settings

Add a new runtime environment variable with the following values.
    Name: DT_SIGNATURE_SECRET
    Value: A unique password. We will need it later, so write it down.

(2) Code

    Runtime: Python 3.9
    Entry point: dataconnector_endpoint
In the Source Code, edit main.py with the following snippet. The implementation is explained in detail on the Data Connector Advanced Configurations page.
main.py
1
import os
2
import hashlib
3
import jwt
4
5
# Fetch environment variables.
6
SIGNATURE_SECRET = os.environ.get('DT_SIGNATURE_SECRET')
7
8
9
def verify_request(body, token):
10
# Decode the token using signature secret.
11
try:
12
payload = jwt.decode(token, SIGNATURE_SECRET, algorithms=["HS256"])
13
except Exception as err:
14
print(err)
15
return False
16
17
# Verify the request body checksum.
18
m = hashlib.sha1()
19
m.update(body)
20
checksum = m.digest().hex()
21
if payload["checksum"] != checksum:
22
print('Checksum Mismatch')
23
return False
24
25
return True
26
27
28
def dataconnector_endpoint(request):
29
# Extract necessary request information.
30
body = request.get_data()
31
token = request.headers['x-dt-signature']
32
33
# Validate request origin and content integrity.
34
if not verify_request(body, token):
35
return ('Could not verify request.', 400)
36
37
#
38
# Further processing here.
39
#
40
41
return ('OK', 200)
Copied!
Append the following snippet to requirements.txt.
requirements.txt
1
pyjwt==2.1.0
Copied!
For details, read our Python API Documentation.

(1) Configuration

Trigger

    Trigger Type: HTTP
    Authentication: Allow unauthenticated invocations.

Variables, Networking and Advanced Settings

Add a new runtime environment variable with the following values.
    Name: DT_SIGNATURE_SECRET
    Value: A unique password. We will need it later, so write it down.

(2) Code

    Runtime: Python 3.9
    Entry point: dataconnector_endpoint
In the Source Code, edit main.py with the following snippet.
main.py
1
import os
2
from dtintegrations import data_connector, provider
3
4
DT_SIGNATURE_SECRET = os.getenv('DT_SIGNATURE_SECRET')
5
6
7
def dataconnector_endpoint(request):
8
# Validate and decode the incoming request.
9
event = data_connector.http_push.decode_request(
10
request,
11
provider=provider.GCLOUD,
12
secret=DT_SIGNATURE_SECRET,
13
)
14
15
# Print the event data.
16
print(event)
17
18
# If all is well, return 200 response.
19
return ('OK', 200)
Copied!
Append the following snippet to requirements.txt.
requirements.txt
1
dtintegrations
Copied!

(1) Configuration

Trigger

    Trigger Type: HTTP
    Authentication: Allow unauthenticated invocations.

Variables, Networking and Advanced Settings

Add a new runtime environment variable with the following values.
    Name: DT_SIGNATURE_SECRET
    Value: A unique password. We will need it later, so write it down.

(2) Code

    Runtime: Node.js 14
    Entry point: dataconnectorEndpoint
In the Source Code, edit index.js with the following snippet. The implementation is explained in detail on the Data Connector Advanced Configurations page.
index.js
1
const crypto = require('crypto')
2
const jwt = require('jsonwebtoken') // npm install [email protected]
3
4
// Fetch environment variables.
5
const signatureSecret = process.env.DT_SIGNATURE_SECRET
6
7
function verifyRequest(body, token) {
8
// Decode the token using signature secret.
9
let decoded;
10
try {
11
decoded = jwt.verify(token, signatureSecret)
12
} catch(err) {
13
console.log(err)
14
return false
15
}
16
17
// Verify the request body checksum.
18
let shasum = crypto.createHash('sha1')
19
let checksum = shasum.update(JSON.stringify(body)).digest('hex')
20
if (checksum !== decoded.checksum) {
21
console.log('Checksum Mismatch')
22
return false
23
}
24
25
return true
26
}
27
28
exports.dataconnectorEndpoint = (req, res) => {
29
// Extract necessary request information.
30
let body = req.body
31
let token = req.get('X-Dt-signature')
32
33
// Validate request origin and content integrity.
34
if (verifyRequest(body, token) === false) {
35
res.sendStatus(400);
36
return
37
}
38
39
//
40
// Further processing here.
41
//
42
43
res.sendStatus(200);
44
};
Copied!
Edit package.json to contain the following dependencies field.
package.json
1
{
2
"dependencies": {
3
"jsonwebtoken": "^8.5.1"
4
}
5
}
Copied!

(1) Configuration

Trigger

    Trigger Type: HTTP
    Authentication: Allow unauthenticated invocations.

Variables, Networking and Advanced Settings

Add a new runtime environment variable with the following values.
    Name: DT_SIGNATURE_SECRET
    Value: A unique password. We will need it later, so write it down.

(2) Code

    Runtime: Go 1.13
    Entry point: DataconnectorEndpoint
In the Source Code, edit function.go with the following snippet. The implementation is explained in detail on the Data Connector Advanced Configurations page.
function.go
1
package triggerfunction
2
3
import (
4
"crypto/sha1"
5
"encoding/hex"
6
"fmt"
7
jwt "github.com/dgrijalva/jwt-go" // go get github.com/dgrijalva/[email protected]
8
"io/ioutil"
9
"log"
10
"net/http"
11
"os"
12
)
13
14
// Environment variables.
15
var signatureSecret = os.Getenv("DT_SIGNATURE_SECRET")
16
17
// verifyRequest validates the request origin and content integrity.
18
func verifyRequest(bodyBytes []byte, tokenString string) error {
19
// Decode the token using signature secret.
20
claims := jwt.MapClaims{}
21
_, err := jwt.ParseWithClaims(tokenString, claims, func(token *jwt.Token) (interface{}, error) {
22
return []byte(signatureSecret), nil
23
})
24
if err != nil {
25
return err
26
}
27
28
// Verify the request body checksum.
29
sha1Bytes := sha1.Sum(bodyBytes)
30
sha1String := hex.EncodeToString(sha1Bytes[:])
31
if sha1String != claims["checksum"] {
32
return fmt.Errorf("Checksum mismatch.")
33
}
34
35
return nil
36
}
37
38
// DataConnectorEndpoint receives, validates, and returns a response for the forwarded event.
39
func DataconnectorEndpoint(w http.ResponseWriter, r *http.Request) {
40
41
// Extract necessary request information.
42
tokenString := r.Header.Get("x-dt-signature")
43
bodyBytes, err := ioutil.ReadAll(r.Body)
44
if err != nil {
45
log.Fatal(err)
46
}
47
48
// Validate request origin and content integrity.
49
if err := verifyRequest(bodyBytes, tokenString); err != nil {
50
log.Println(err)
51
http.Error(w, err.Error(), http.StatusBadRequest)
52
return
53
}
54
55
//
56
// Further processing here.
57
//
58
59
log.Println("OK")
60
fmt.Fprintf(w, "OK\n")
61
}
Copied!
Replace the content of go.mod with the following snippet.
1
module example.com/cloudfunction
2
3
go 1.13
4
5
require (
6
github.com/dgrijalva/jwt-go v3.2.0+incompatible
7
)
Copied!
When configured, deploy your function.

Post Deployment

Your function is now ready to receive requests, but we need to know the target URL. In your function, locate the TRIGGER tab and copy the Trigger URL. Save this for later.

Create a Data Connector

To continuously forward the data to our newly created Cloud Function, a Data Connector with almost all default settings is sufficient. If you are unfamiliar with how Data Connectors can be created, refer to our Creating a Data Connector guide using the following configurations.
    Endpoint URL: The Trigger URL found in the previous step.
    Signature Secret: The value of the DT_SIGNATURE_SECRET environment variable.
Depending on your integration, it can also be smart to disable the event types you are not interested in. For instance, the networkStatus event is sent every periodic heartbeat and will, by default, be forwarded by the Data Connector if not explicitly unticked.

Test the Integration

If the integration was correctly implemented, the Success counter for your Data Connector should increment for each new event forwarded. This happens each periodic heartbeat or by touching a sensor to force a new event.
If instead the Error counter increments, a response containing a non-2xx status code is returned.
    Verify that the Data Connector endpoint URL is correct.
    Google provides a host of tools that monitor Cloud Functions. Check the logs for any tracebacks that could explain why an error is returned.

Next steps

Your sensor data is now in the Google Cloud environment, and you can start using it in their various services. Fortunately, Google has some well-documented guides to get you started.

PostgreSQL Database

A database should be tailored to each specific use case. However, if you're uncertain, PostgreSQL (Postgres) is a good place to get started. The following guides will show you how to create a new Postgres database, then connect your Cloud Function to execute queries.

DataStudio

Once your Database has been set up, the following guide shows you how to connect it to DataStudio for continuous data visualization and analytics.
Last modified 1mo ago