What you’ll learn

How to use Artillery Pro to run load tests on AWS.

The difference between using ECS or Fargate.

How to customize the runtime environment on AWS.

Overview

Artillery Pro’s run-test command allows for a test script to be run from an AWS Elastic Container Service (ECS) or Fargate cluster. This command lets you scale up and run your tests in distributed mode across multiple workers and from several geographical regions.

Requirements

Artillery Pro needs to be installed before running tests from AWS.

Before running tests on AWS, you’ll also need AWS credentials to access an ECS or Fargate cluster, along with the required IAM permissions set up on the local machine.

Use classic ECS or Fargate

Artillery Pro can use classic ECS clusters (with container instances managed by your team) or Fargate (to run containers without managing any servers).

By default, run-test will execute in a classic ECS cluster. To use Fargate, invoke run-test with the --launch-type ecs:fargate flag, and provide one or more public subnets to launch containers in.

For more information about ECS and Fargate, please see Amazon Elastic Container Service documentation page.

Should you use ECS or Fargate?

Fargate makes for an ideal starting point since there is no extra infrastructure to set up or manage. Your team can go from running tests from their local machines with artillery run to running distributed cloud tests from AWS with artillery run-test in a matter of minutes.

Generally, the decision to use classic ECS or Fargate depends on the infrastructure that’s already in place and any processes already used by your team or the broader organization.

Some areas to consider when deciding to use ECS or Fargate:

Whether there are classic ECS clusters already in place, with capacity to allocate towards running load tests.
The workload simulated by your Artillery tests may require a custom CPU/memory configuration. Fargate defines a pre-defined set of configurations, and if your tests required more CPU without the corresponding increase in memory requirements, classic ECS might be a more economical choice.
Classic ECS clusters comprised of Compute Optimized instance types may offer better and more consistent performance, especially for tests that are CPU or network intensive.
Classic ECS typically offers much faster worker startup times than Fargate, especially for more extensive tests.

We typically recommend that you use Fargate by default for the flexibility and ease of use it provides, unless your team has a classic ECS cluster already in place or you have a specific need where ECS is a better or more economical choice.

Running tests with Artillery Pro

All following examples assume that an ECS cluster called my-ecs-cluster has been set up in the us-east-1 region, and that AWS security credentials have been set up properly to allow the user running the `artillery` CLI to access the cluster as described in the installation guide for Artillery Pro).

ECS

If you are using an ECS cluster with container instances, run the artillery run-test command with your cluster name (--cluster), region (--region) and the number of workers (`–count):

artillery run-test \
  --cluster my-ecs-cluster \
  --region us-east-1 \
  --count 1 \
  my-script.yaml

Fargate

If you want to use Fargate instead, add the --launch-type ecs:fargate flag:

artillery run-test \
  --cluster my-ecs-cluster \
  --region us-east-1 \
  --count 1 \
  --launch-type ecs:fargate \
  my-script.yaml

Using custom JS code and plugins

If your test script includes custom JS via the config.processor attribute, Artillery Pro will automatically resolve any dependencies to external npm packages in your JS code. It will also resolve any plugins used and make those available when the test script runs on ECS.

Explicitly bundling files with the test

run-test will automatically detect any custom JS modules (including their npm dependencies) and CSV files used with the config.payload setting and bundle them into the test package that gets sent to the workers. You may want to include other files that Artillery Pro cannot automatically detect, such as a file that is read with fs.readFile in a custom function. You can use the config.includeFiles setting to tell run-test to include those files.

config:
  target: "https://example.net"
  includeFiles:
    - foo.json
    - bar.xml

Running a named test

run-test may be used with test scripts located on the local disk or with tests created and stored in your AWS account with the create-test command.

First, you need to upload your test script and package all its dependencies to S3 on your AWS account using artillery create-test:

artillery create-test \
  --name my-load-test \
  my-script.yaml

Once uploaded to your AWS account, you can run the named test using artillery run-test by referencing the name of the created test in the previous step:

artillery run-test \
  --cluster my-ecs-cluster \
  --region us-east-1 \
  --count 1 \
  my-load-test

Using named tests with run-test is convenient and efficient since the test script and its dependencies won’t need to be packaged and uploaded before the test runs. We recommend that test scripts that have been debugged and are unlikely to be modified very frequently are pre-uploaded and named with create-test.

Customizing runtime environment

It’s possible to customize the runtime environment of the Artillery script being run on ECS via the --launch-config flag. The following customizations are supported:

cpu - The number of CPU units reserved for each Artillery container. The default value is 1024.
memory - The amount of memory presented to the Artillery container. The default value is 2048 on Fargate and 1024 on classic ECS.
environment - The environment variables to set in the Artillery worker container.
ulimits - Customize ulimits such as the number of available file descriptors (nofile).

Examples

Run a local test script on a Fargate cluster called my-ecs-cluster in the US East (N. Virginia) region using 10 workers and modifying the reserved CPU and memory units:

artillery run-test \
  --cluster my-ecs-cluster \
  --region us-east-1 \
  --count 10
  --launch-type ecs:fargate \
  --launch-config '{"cpu": 2048, "memory": 4096}' \
  my-script.yaml

CPU and memory on Fargate
When using Fargate, cpu and memory values cannot be increased independently of each other. You must specify supported values as defined in the AWS documentation.

Create a named test called my-load-test:

artillery create-test \
  --name my-load-test \
  my-script.yaml

Run a named test named my-load-test on a classic ECS called my-ecs-cluster in the US West (Oregon) region using 20 workers:

artillery run-test \
  --region us-west-2 \
  --cluster my-ecs-cluster \
  --count 20 \
  my-load-test

Increase the number of file descriptors per worker:

artillery run-test \
  --cluster my-ecs-cluster \
  --region us-east-1 \
  --count 10 \
  --launch-config '{"ulimits":[{"name":"nofile","softLimit":"16384","hardLimit":"16384"}]}' \
  my-script.yaml

The default limit of file descriptors set by Artillery is 8192 per worker. If your test runs open lots of TCP connections, your workers may run out of file descriptors, which surface as EMFILE errors. Increasing the number of file descriptors as shown above can help resolve the issue.

Run a test script setting two environment variables named VAR1 and VAR2 for Artillery workers running on ECS/Fargate:

artillery run-test \
  --cluster my-ecs-cluster \
  --region ap-northeast-1 \
  --count 5
  --launch-config '{"environment": [{"name":"VAR1", "value":"hello"},{"name":"VAR2","value":"world"}]}' \
  my-script.yaml

You can also set encrypted secret values to use in your tests with the set-secret command. These secrets will get stored securely in AWS Parameter Store and are accessible via the $processEnvironment variable.