Script to Create a Kinesis Stream

You can use the following Bash script to automate the creation of a Kinesis Stream. Copy the code and save it to your computer, for example as create_kinesis_stream.sh.

Run chmod +x create_kinesis_stream.sh to make it executable.

You will also need to install jq, a command line JSON processor on which the script depends. For installation instructions for your OS, please see the jq download page.

The script also depends upon the AWS CLI, which you will need to install and configure before executing the script.

The script takes two arguments: your chosen AWS Kinesis Stream name and a shard count.

#!/bin/bash

JQ_CHECK=$(which jq)
if [ -z "$JQ_CHECK" ]; then
  echo
  echo "This script requires the jq JSON processor. Please install for your OS from https://stedolan.github.io/jq/download/"
  echo
  exit 1
fi

if [ $# -ne 2 ]; then
  echo
  echo "usage: $0 <stream_name> <shard_count>"
  echo
  exit 1
fi

# Set the stream name
STREAM_NAME=${1:-kore-events}
SHARD_COUNT=${2:-1}

# Create the initial stream
aws kinesis create-stream --stream-name $STREAM_NAME --shard-count $SHARD_COUNT
if [ $? -ne 0 ]; then
  echo "Kinesis create failed"
  exit 1
fi

# Get the ARN for the Kinesis Stream
KINESIS_ARN=$(aws kinesis describe-stream --stream-name $STREAM_NAME | jq -r .StreamDescription.StreamARN)

# Create the policy for the Kinesis Stream
POLICY_ARN=$(aws iam create-policy --policy-name kore-events-kinesis-write --policy-document '{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "Quickstart0",
           "Effect": "Allow",
           "Action": [
               "kinesis:PutRecord",
               "kinesis:PutRecords"
           ],
           "Resource": "'$KINESIS_ARN'"
       },
       {
           "Sid": "Quickstart1",
           "Effect": "Allow",
           "Action": [
               "kinesis:ListShards",
               "kinesis:DescribeLimits"
           ],
           "Resource": "*"
       }
   ]
}' | jq -r .Policy.Arn)

if [ -z "$POLICY_ARN" ]; then
  echo "Failed to create IAM policy"
  exit 1
fi

# Generate a random external ID
EXTERNAL_ID=$(openssl rand -hex 40)
if [ -z "$EXTERNAL_ID" ]; then
  echo "Failed to generate external ID"
  exit 1
fi

# This is the KORE account that needs permissions to be able to assume the role
KORE_ASSUME_ROLE_ACCOUNT=${KORE_ASSUME_ROLE_ACCOUNT:-arn:aws:iam::750607079480:root}

# Add the random external ID to the the role ARN
# More information can be found here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html
ROLE_ARN=$(aws iam create-role --role-name kore-events-kinesis-write --assume-role-policy-document '{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Principal": {
       "AWS": "'$KORE_ASSUME_ROLE_ACCOUNT'"
     },
     "Action": "sts:AssumeRole",
     "Condition": {
       "StringEquals": {
         "sts:ExternalId": "'$EXTERNAL_ID'"
       }
     }
   }
 ]
}' | jq -r .Role.Arn)

if [ -z "$ROLE_ARN" ]; then
  echo "Failed to create IAM role"
  exit 1
fi

# Finally attach the policy and the role
aws iam attach-role-policy --role-name kore-events-kinesis-write --policy-arn $POLICY_ARN

if [ $? -ne 0 ]; then
  echo "Attaching policy to role failed"
  exit 1
fi

# Print out the values needed for creating the sink in nice JSON
echo "{"
echo '"arn":"'$KINESIS_ARN'",'
echo '"role_arn":"'$ROLE_ARN'",'
echo '"external_id":"'$EXTERNAL_ID'"'
echo "}"