AWS Kinesis is an AWS product for building systems that process and analyze large amounts of streaming data such as IoT device events, website clickstreams, logs, financal transactions etc. It’s an instant-on managed solution and a popular alternative to self-managed open source solutions such as Apache Kafka.

A Kinesis-based system will have a number of streams (each of which is composed of one or more shards). Producers can push messages into a stream, which can be read by one or more consumers. The messages (or records) published to a Kinesis stream are persisted for 24h or longer and are distributed across several AWS regions for reliability.

There is no concept of a topic in Kinesis like that in Kafka or RabbitMQ. New Kinesis streams can be created as needed through the AWS SDK, the CLI or the Console, and existing streams can be scaled up and down.

Generating load on Kinesis with Artillery

With the new artillery-engine-kinesis plugin, Artillery can now act as a producer for a Kinesis stream and push large amounts of data into one very quickly.

An example Artillery test script using the Kinesis engine:

# Emulate 20 publishers sending two records per second
# for 60 seconds.
config:
  target: "my_awesome_stream"
  kinesis:
    region: "eu-west-1"
  phases:
    - arrivalCount: 20
      duration: 1
  engines:
    kinesis: {}

scenarios:
  - name: "Push to stream"
    engine: kinesis
    flow:
      - count: 60
        loop:
        - putRecord:
            partitionKey: "testPk"
            # data may be a string or an object. Objects
            # will be JSON.stringified.
            data:
              eventType: "view"
              objectId: "ba0ec3de-26fe-4874-a74d-b72527160278"
              timestamp: 1492975372004
              location: "London, UK"
        - putRecord:
            partitionKey: "testPk"
            data:
              eventType: "view"
              objectId: "ff0ed9de-26be-7654-f90c-b974117161954"
              timestamp: 1492975372004
              location: "London, UK"
        - think: 1

Why load test Kinesis?

Kinesis was designed to scale horizontally and elastically, so why bother load testing it at all?

The reality is more complex than that. Essentially, Kinesis makes extra capacity accessible to you, but it is up to you to:

  1. Provision enough capacity - the write throughput of a stream will be determined by the number of shards. Each shard provides capacity for 1000 record writes per second / up to 1MB per second. How many writes per second and of what record size on average are your producers going to be performing and how many shards are you going to need to support that? More importantly, only 5 transactions per second are supported for reading from a shard for read rate of 2MB/second. Will your consumers be able to keep up with the amount of data streamed to them with a low enough latency - especially important if you need more than 5 consumers accessing the entire stream and will have to sacrifice some latency already by polling less frequently than once per second.
  2. Scale capacity up and down as required - while a Kinesis stream can contain up to 50 shards (by default, the max value can be increased further), streams need to be scaled explicitly through the API, the CLI or the Console as needed, and a stream cannot be scaled up by more than twice its current capacity. Scaling up a busy stream is not an instantaneous operation, and it’s far better to start off with a number of shards that will accomodate the amount of data flowing through the system.
  3. Make the best use of provisioned capacity - each write operation needs to specify a partition key, which effectively pushes scalability planning onto the users Kinesis. The choice of partition key can have far-reaching implications into how incoming data is split across the shards in a stream and the resulting load on the consumers. Testing ahead of time will allow you to confirm whether your partitioning scheme is production-ready, and work out a better one if not.

Until you’ve seen a system under stress, you don’t understand it

Load testing your Kinesis streams can help you get a more intuitive feel for the limitations of Kinesis and their implications to your particular application. Seeing the details of how the latency of the system can be affected by the amount of incoming data, the choice of partition key and number of shards is only possible by generating load on the system, and that’s why we built the Kinesis engine for Artillery.

Head over to the plugin’s Github page for installation and usage instructions.

Happy load testing!