AWS Kinesis Quick Start

Create a Kinesis stream for your Account

We have prepared a script for you that sets up a Kinesis stream with a shard count. Copy the script and save it to a file named create_kinesis_stream.sh. Make it executable with chmod 755. The script will set up the policies and roles KORE needs to write to your stream. The only thing you need to do is run the script and pass the stream name and shard count as arguments.

You will need to ensure that your terminal has valid AWS credentials before executing both this script and the script you will use later in this quick start.

Use the name kore-events for the Kinesis stream. You can use any name, but if you use a different one than kore-events, you will need to change the permission instructions later on with your custom name.

Set the number of shards to 1. This makes initial validation easier. It can be modified after the Destination has been set up.

Here's an example:

./create_kinesis_stream.sh kore-events 1 | jq . > kore-destination.json

The script uses the AWS CLI to create the Kinesis stream with the specific shard count. It will return a JSON payload with the details you need to create a Destination. In the above example, the received data will be written to a file named kore-destination.json.

The stream is created in the region specified by the environment variable AWS_DEFAULT_REGION.

You can update the shard count of the Kinesis stream after it has been validated by using this script:

Copy code block
if [ $# -ne 2 ]; then
  echo
  echo "usage: $0 <stream_name> <shard_count> <scaling_type>"
  echo "<scaling_type> possible values UNIFORM_SCALING”
  echo
  exit 1
fi

# update stream
STREAM_NAME=$1
SHARD_COUNT=$2
SCALING_TYPE=$3

aws kinesis update-shard-count --stream-name $STREAM_NAME --scaling-type $SCALING_TYPE --target-shard-count $SHARD_COUNT

Here is an example showing how the Kinesis stream shard count is increased to 6:

./update_kinesis_shard_count.sh kore-events 6 UNIFORM_SCALING

Create a Kinesis Destination at KORE

To create a new Kinesis destination, go to the navigation bar on the left side of the screen and click on the Destinations option. This will open the Destinations page. Select the +Create button to add.

Provide a friendly name and select the destination type "AWS Kinesis" then click on continue.

Provide your kinesis configuration by filling in the following inputs

• ARN: is a Unique identifier for your Amazon Kinesis resource. It typically follows this format: arn:aws:kinesis:region:account-id:stream/stream-name

  • You can find your Kinesis ARN in AWS Management Console :

    • Navigate to Amazon Kinesis => Data Streams

• Role ARN: is a unique identifier for an IAM (Identity and Access Management). This identifier typically follows this format arn:aws:iam::account-id:role/role-name

  • You can find the ARN of an IAM role in the AWS Management Console :

    • Navigate to IAM service => Roles

• External ID: is an additional security measure for stream connections. This unique identifier serves as a secret token that authorizes KORE to assume the provided IAM Role, granting secure access to your AWS Kinesis resources.

Once you have completed all required fields in the form, click the 'Finish' button . This action will finalize the setup process and create a new Kinesis destination.

(Optional) Test your Destination

This step is optional but encouraged because it ensures KORE and your destination can successfully communicate before trying to stream events to your destinations.

Review our guide on how you can test your destination.

Create a Streaming Rule

A streaming rule that tells KORE which events you streamed to which destination

To create a new rule, go to the navigation bar on the left side of the screen and click on the Streaming Rules option. This will open the Rules page. Select the +Create button.

Provide your rule configuration, filling in the required fields for the rule configuration:

• Rule Name: A unique identifier for the rule you are creating.

• Destination: The endpoint where the events will be sent.

• Product: Select the product group relevant to your events.

After selecting a product group, you'll have a list of available event types and their associated schemas. To configure your rule:

• Review the displayed event types relevant to the chosen product group.

• Select the appropriate schema version for each event type you wish to include in your rule.

• Your selections will determine which event types and schema versions will be applied to your streaming rule.

Once you have completed all required fields in the form, click the Create button. This action will finalize the setup process and create a new streaming rule. You should start receiving events sent to the destination when they happen.

Read and Parse the Data

The data you receive will follow the formats as designed:

• Event Schema

• Event Data Object Schema

Review our guide to see an example of this.