Device Communication via Device Data Proxy

Just like the key-value store for homes, key-value data can be stored in the cloud for devices as well. We call it Device Data Proxy, because it serves not only as a data store, but also as a means of communication with devices.

At its core, Device Data Proxy is a key-value storage that applications can access at the following REST API endpoint:

/devices/{deviceId}/kv

What makes Device Data Proxy interesting is that it provides a mechanism for a device to watch the changes made by others to the data store, while the device itself can also update the data. Essentially, it acts as a proxy that allows bi-directional communication channel between the device and the app. It provides an alternative to the usual commmand-and-event method of communication.

Device Data Proxy is great for sending “state-ful” or persistent values. An obvious use is for holding configuration parameters for devices. Conversely, devices can use it to hold values that represent its “last-known” state.

Using the REST API

Accessing the key-value store of devices is quite similar to accessing the key-value store of homes. For example, considering a device with ID 123, to set its key xyz to the value hello world, just make a PUT request like this:

$ curl -i -X PUT -H "Authorization: ModeCloud _YOUR_API_KEY_" -H "Content-Type: application/json" -d "{\"value\": \"hello world\"}" https://api.tinkermode.com/devices/123/kv/xyz

The types of value you can save to the key-value store are:

  • Number
  • String
  • Boolean
  • Array
  • Simple Javascript object

Similarly, you can use the REST API to read and delete key-value pairs.

Data Synchronization

The key-value pairs are stored in the cloud and therefore can be read from and written to anytime, even when the device itself is offline. Hence, this cloud-based data store acts as a “source of truth” for the key-value pairs associated with this device. However, this works only if the device implement the proper protocol to synchronize with the cloud storage. One way of doing that is for the device to establish a websocket connection to the following API endpoint:

/devices/{deviceId}/kv

The device should expect to receive “sync” messages from this websocket connection. Each message is a JSON object. By applying the actions described in these messages, the device can maintain an up-to-date local mirror of the key-value pairs stored in the cloud.

(It is also possible to do this via MQTT. We will talk about that later in this article.)

There are three types of sync messages:

  • reload
  • set
  • delete

“Reload” Message

Device should expect a “reload” message immediately after it has established the websocket connection. It contains a full dump of all existing key-value pairs in the cloud storage. This allows the device to initialize its local mirror storage. Here is an example of a “reload” message:

{
  “action”: “reload”,
  “rev”: 82,
  “numItems”: 2,
  “items”: [
    {
      “key”: “xyz”,
      “value”: “hello world”,
      “modificationTime”: “2017-11-19T01:23:22Z”
    },
    {
      “key”: “foo”,
      “value”: 22399,
      “modificationTime”: “2017-11-21T11:09:18Z”
    }
  ]
}

The rev field is a revision number of the key-value store and is incremented every time a change happens (i.e. when a key-value is set or deleted). It is important for the device to keep track of the revision number with its local mirror storage, so that it can handle out-of-order sync messages.

“Set” Message

The device will receive a “set” message when a key-value pair has been created or updated. For example:

{
  “action”: “set”,
  “rev”: 83,
  “key”: “xyz”,
  “value”: “I have changed”
}

The device should apply this change to its local storage if and only if the rev field is greater than the revision number stored locally.

“Delete” Message

The device will receive a “delete” message when a key-value pair has been deleted. For example:

{
  “action”: “delete”,
  “rev”: 85,
  “key”: “xyz”
}

The proper way to handle a “delete” message is to mark the local mirror value as deleted, and record the new rev number. This way, an out-of-order “set” message to the same key can be identified and ignored.

Using MQTT with Device Data Proxy

As in handling events and commands, devices can use MQTT to interact with Device Data Proxy. In fact, MQTT may be the preferred method, since with MQTT a device can use one single network connection to the MODE Cloud for both events/commands and Device Data Proxy, simply by publishing/subscribing to the appropriate topics.

In the case of using Device Data Proxy, a device connects to the MODE MQTT server (mqtt.tinkermode.com) and subscribes to the following topic to receive “sync” messages:

/devices/{deviceId}/kv

The device can also publish to this same topic to update the key-value store. For example, to set a key-value pair, publish the following JSON:

{
  "action": "set",
  "key": "xyz",
  "value": "this is changed"
}

And to delete a key-value pair, just publish a JSON like the following:

{
  "action": "delete",
  "key": "xyz"
}

Example Code

The official MODE device SDK in Go supports Device Data Proxy synchronization and mirroring. You may use it directly for your device software implementation, or use it as a reference for your own design.