Introduction


The Kinesis Smart Module is a Smart Module that directly forwards data collected from MODE Devices to your Kinesis Data Stream.

Concept Diagram

Preparation

Before you set up the Kinesis Smart Module, make sure 1) Kinesis Data Stream that will be the target is created, 2) An IAM Role with an IAM policy which has the appropriate permissions for putting records into the Kinesis Data Stream is created.

The IAM Policy given to the IAM Role should look like the following:

IAM Policy template

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:PutRecord",
        "kinesis:PutRecords"
      ],
      "Resource": "KINESIS_DATA_STREAM_ARN"
    }
  ]
}

NOTE: Please replace KINESIS_DATA_STREAM_ARN with your actual Kinesis Data Stream ARN.

Setting up Kinesis Smart Module

ONE | In the Developer Console, navigate to the Smart Modules page and select +NEW. Click +ADD on the section for Kinesis.

Screenshot - Console Smart Modules Add Kinesis

TWO | Give the Kinesis Smart Module an ID and Description.

Screenshot - Console Smart Modules Add Kinesis

THREE | Fill in the Role ARN, External ID, Region, Stream Name based on the information of the Kinesis Data Stream you have created in the previous Preparation section. Click SAVE.

Screenshot - Console Smart Modules Add Kinesis

FOUR | (Optional) You can specify how data is sent into your Kinesis Data Stream by configuring your Kinesis Data Stream. Click the Kinesis Smart Module you've created in the list of Smart Modules, then click the gear icon on the details page.

Screenshot - Console Smart Modules Add Kinesis

By specifying Subscribed Events, you can specify the event types that will be propagated to your Kinesis Data Stream.

  • By default, Subscribed Events is *, which means every Event occurring in your project will be forwarded to your Kinesis Data Stream.
  • If you leave it blank, no Events will be forwarded.
  • If you enter specific event type name(s), only those events will be propagated.

Please refer to the section Send MODE Event as Kinesis Data Record for more information.

On the other hand, you can enable efficient transfer of arbitrary data from your devices into your Kinesis Data Stream by specifying a Bulk Data Label. For the details, please have a look at the section Send arbitrary data as Kinesis Data Record.

Screenshot - Console Smart Modules Add Kinesis

How to send Kinesis Data Record

Send MODE Event as Kinesis Data Record

First of all, please refer to the following pages to know the way to send Events if you are not familiar with MODE Event.

MODE Event is a JSON object with following structure:

  • FieldTypeDescriptionRequired
  • eventTypestringEvent type ID.Yes
  • eventDataobjectEvent data.Yes
  • timestampstringIn RFC 3339 format.Yes
  • homeIdintegerID of home associated with this event.Yes
  • originDeviceIdintegerPresent if the event originates from a device.No (present only with Device Event)
  • originDeviceClassstringPresent if the event originates from a device.No (present only with Device Event)
  • originDeviceIpstringPresent if the event originates from a device.No (present only with Device Event)
  • originProjectKeyIDstringPresent if the event originates from a program using a project API key.No (present only with Home Event)
  • originProjectKeyNamestringPresent if the event originates from a program using a project API key.No (present only with Home Event)

Each MODE Event corresponds to a Kinesis Data Record. That is, if you send a MODE Event and it is forwarded to your Kinesis Data Stream, that event is represented as a Kinesis Data Record.

NOTE: The type of event which can be propagated to Kinesis Data Stream is determined by the Smart Module configuration "Subscribed Events".

Send arbitrary data as Kinesis Data Record

You can send any kind of data which can be represented as a byte sequence by MQTT with Bulk Data topic. Please read How to use MQTT with MODE if you are not familiar.

While the payload for the MQTT topic Bulk Data is an arbitrary byte sequence, you need to build the payload data to satisfy the format which is specific to Kinesis Smart Module.

The payload format used for the Bulk Data topic for Kinesis Smart Module is supposed to be encoded as MessagePack.

First part of the payload is a version string which represents the data encoding format version – "1.0" is the only available version string. The rest of it is an arbitrary-length array of byte arrays. Eachelement, a byte array, corresponds to a Kinesis Data Record.

  • VersionData....Data
  • "1.0"byte array....byte array

The following code snippet is an example in Golang for encoding an array of byte arrays into a MessagePacked payload for the MQTT Bulk Data topic corresponding to Kinesis Smart Module.

Example code by Golang

func encode(records [][]byte) ([]byte, error) {
    var buf []byte
    writer := bytes.NewBuffer(buf)
    encoder := msgpack.NewEncoder(writer)
    
    if err := encoder.EncodeString("1.0"); err != nil {
        return nil, err
    }

    for _, record := range records {
        err := encoder.EncodeBytes(record)
        if err != nil {
            return nil, err
        }
    }

    return writer.Bytes(), nil
}

How a Kinesis Data Record from MODE looks like

As described in the API Document, essential components of Kinesis Data Record are "Data" ( arbitrary byte sequence within 1 MB) and "PartitionKey" (arbitrary string used to determine the shard to which the record would be going to).

"Data" is explained in the previous section. "Partition Key" is automatically assigned in the Kinesis Smart Module internal system, where UUID is used for each Record.

Troubleshooting

You can view the Kinesis Smart Module activities under the "LOGS" section of your Kinesis Smart Module configuration page.

Screenshot - Console Smart Modules Add Kinesis

Check if there are log entries at the ERROR level.

One type of typical errors, especially in the early phase of development, is data format error. Please check How to send Kinesis Data Record section again.

Another common error is caused by insufficient throughput capacity in your Kinesis Data Stream. This error can only be fixed either by 1) decreasing the frequency or amount of data emission from devices or 2) increasing the number of shards in your Kinesis Data Stream so that it can acquire more throughput capacity (see Kinesis Data Streams Quotas and Limits - Amazon Kinesis Data Streams for more details about this topic).