Running tests from AWS with Artillery Pro ⚡

Overview

Artillery Pro’s run-test command allows for a test script to be run from an AWS ECS or Fargate cluster. This command lets you scale up and run your tests in distributed mode across multiple workers, and from a number of geographical regions.

Requirements

Use classic ECS or Fargate

Artillery Pro can use classic ECS clusters (with container instances managed by your team), or Fargate (to run containers without needing to manage any servers). By default, run-test will assume that a classic ECS cluster is used. To use Fargate, invoke run-test with the --launch-type ecs:fargate flag, and provide one or more public subnets to launch containers in.

For more information about ECS and Fargate, please see AWS ECS product page.

Should we use classic ECS or Fargate?

Fargate makes for an ideal starting point since there is no extra infrastructure to set up or manage. Your team can go from running tests from their local machines with artillery run to running distributed cloud tests from AWS with artillery run-test in a matter of minutes.

Generally, the decision on whether to use classic ECS or Fargate depends on the infrastructure that’s already in place & any standards & processes already used by your team or the wider organization.

Considerations may include:

  • Whether there are classic ECS clusters already in place, with capacity to be allocated towards running load tests
  • The maximum size of the tests you need to run. Fargate has a default limit of 50 tasks per region, which you may need to request AWS to increase for your account if your team needs to run tests with 50+ workers (or two concurrent tests which use 25+ workers each).
  • Workload simulated by your Artillery tests may require a custom CPU/memory configuration. Fargate defines a pre-defined set of configurations, and if your tests required more CPU without the corresponding increase in memory requirements, classic ECS may be a more economical choice.
  • Classic ECS clusters comprised of compute-optimized instance types may offer better & more consistent performance, especially for tests which are especially CPU and/or network intensive.
  • Classic ECS typically offers much faster worker startup times, especially for larger tests than Fargate.

We would typically recommend that you use Fargate by default for the flexibility and ease-of-use it provides (serverless containers with no need to set up any VMs), unless your team has a classic ECS cluster already in place, or if one or more of the considerations outlined above applies.

Running tests with run-test

Run a Hello World test

All following examples assume that an ECS cluster my-ecs-cluster has been set up in us-east-1 and that AWS credentials have been set up appropriately to allow the user running the artillery CLI to access the cluster as described in the installation guide for Artillery Pro).

Create a hello-world.yml adjusting the value of target to point to an URL under your control.

config:
  target: "https://example.net"
  phases:
    - duration: 20
      arrivalRate: 1
scenarios:
  - flow:
    - get:
        url: "/"

If you are using an ECS cluster with container instances, run:

artillery run-test --cluster my-ecs-cluster --region us-east-1 --count 1 hello-world.yml

If you want to use Fargate, add a --launch-type ecs:fargate flag:

artillery run-test  --cluster my-ecs-cluster --region us-east-1 --count 1 --launch-type ecs:fargate hello-world.yml

Using custom JS code and plugins

Your test scripts may include custom JS via the config.processor attribute in your test script, or plugins via the config.plugins section. For example:

config:
  target: "https://example.net"
  phases:
    - duration: 20
      arrivalRate: 1
  processor: "./functions.js"
  plugins:
    some-plugin: {}
scenarios:
  - flow:
    - get:
        url: "/"

Artillery Pro will automatically resolve any dependencies to external npm packages in your custom JS code as well as any plugins used and make those available when the test script is being run on ECS.

Explicitly bundling files with the test

run-test will automatically detect any custom JS modules (as well as their npm dependencies) and CSV files used with config.payload and bundle them into the test package that gets sent to the workers. Sometimes you may want to include other files which cannot be automatically detected (e.g. a file which is read with fs.readFile in a custom function). You can use config.includeFiles to tell run-test to include those files.

config:
  target: "https://example.net"
  includeFiles:
    - foo.json
    - bar.xml

Running a named test

run-test may be used with test scripts located on the local disk or with tests created and stored in your AWS account with the `create-test` command.

For example:

# Package up and upload hello-world.yml and all of its dependencies to S3 in your AWS account:
artillery create-test --name hello-world /path/to/hello-world.yml

# Run the named test:
artillery run-test --cluster my-ecs-cluster --region us-east-1 --count 1 hello-world

Using named tests with run-test is more efficient since the test script and its dependencies don’t need to be uploaded before the test runs. We recommend that test scripts that have been debugged and are unlikely to be modified very frequently are pre-uploaded and named with create-test.

Customizing runtime environment

It’s possible to customize the runtime environment of the Artillery script being run on ECS via the --launch-config flag. The following customizations are supported:

  • cpu - customize the number of CPU units reserved for each Artillery container; the default value is 1024
  • memory - customize the amount of memory presented to the Artillery container; the default value is 2048 on Fargate, and 1024 on classic ECS.
  • environment - customize the environment variables set in the Artillery worker container.
  • ulimits - customize ulimits such as the number of available file descriptors (nofile).

Example: Increase CPU and memory for Artillery workers

The following examples shows how CPU and memory allocation can be doubled from their default values when running on Fargate:

artillery run-test --cluster my-cluster --region us-east-1 --launch-type ecs:fargate \
  --launch-config '{"cpu": 2048, "memory": 4096}' \
  hello-world.yml

CPU and memory on Fargate

When using Fargate, cpu and memory values cannot be increased independently of each other and need to match according to this table:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html

Example: Increase the number of file descriptors

If you’re seeing EMFILE errors, Artillery workers may be running out of file descriptors. This can happen in test runs that open a lot of TCP connections. The default limit set by Artillery is 8192 file descriptors per works, but that limit can be increased. For example, to double the default limit and set it to 16384 for a test run, use:

artillery run-test --cluster my-ecs-cluster --region us-east-1 \
  --launch-config '{"ulimits":[{"name":"nofile","softLimit":"16384","hardLimit":"16384"}]}' \
  hello-world.yml

Example: Setting environment variables

To set two environment variables named VAR1 and VAR2 for Artillery workers running on ECS/Fargate, run:

artillery run-test --cluster my-ecs-cluster --region us-east-1 \
    --launch-config '{"environment": [{"name":"VAR1", "value":"hello"},{"name":"VAR2","value":"world"}]}' hello-world.yml

Support for setting environment variables with encryption using AWS Parameter Store will be available in an upcoming Artillery Pro release.