Running Tests With Artillery Pro

Artillery Pro extends the artillery CLI with a run-cluster command which works similarly to the run command, but instead of executing the test from the local machine it runs it from an AWS ECS cluster you specify.

To run tests with run-cluster, Artillery Pro backend components need to be installed in your AWS account first. Please see the installation guide for details.

run-cluster reference

The run-cluster command shares several flags with run such as --environment, --target, --insecure, --config, --overrides and --output. It also introduces several flags specifically for configuring how a test should be executed in a cloud environment.


  Usage: run-cluster [options] <test>

  Run a test from a cloud cluster. <test> may point to a local file or specify a named test created with the create-test command

  Options:

    -h, --help                  output usage information
    --count <number>            Number of load generators to launch
    --cluster <name>            Name of the ECS cluster to run the test on
    -r, --region <region>       The AWS region to run in (us-east-1, us-east-2, us-west-2, eu-west-1)
    -e, --environment <name>    Use a named test environment
    -t, --target <url>          Set target URL
    -k, --insecure              Allow insecure TLS connections
    --config <path>             Use test config from a separate location
    --overrides <JSON>          Object describing parts of the test script to override
    -o, --output <path>         Write a JSON log to file at provided path
    --launch-config <JSON>      Object describing launch configuration for ECS workers
    --fargate                   Use Fargate to run the test
    --public-subnet-ids <list>  Comma-separated list of public subnet IDs to use for tasks when running tests from Fargate
    --security-group-ids <id>   Optional: comma-separated security group IDs to use for tasks when running tests from Fargate

Classic ECS or Fargate?

Artillery Pro can use classic ECS clusters (with container instances managed by your team), or Fargate (to run containers without needing to manage any servers). By default, run-cluster will assume that a classic ECS cluster is used. To use Fargate, invoke run-cluster with the --fargate flag, and provide one or more public subnets to launch containers in.

For more information about ECS and Fargate, please see AWS ECS product page.

Should we use classic ECS or Fargate?

The decision on whether to use classic ECS or Fargate depends on infrastructure that's already in place & any standards already used by your team.

We would typically recommend that you use Fargate by default for the flexibility and ease-of-use it provides (serverless containers with no need to set up any VMs), unless your team has a classic ECS cluster already in place which can be used to run load tests from.

Run a Hello World test

All following examples assume that an ECS cluster my-ecs-cluster has been set up in us-east-1 and that AWS credentials have been set up appropriately to allow the user running the artillery CLI to access the cluster as described in the installation guide for Artillery Pro.

Create a hello-world.yml adjusting the value of target to point to an URL under your control.

config:
  target: "https://example.net"
  phases:
    - duration: 20
      arrivalRate: 1
scenarios:
  - flow:
    - get:
        url: "/"

If you are using an ECS cluster with container instances, run:

artillery run-cluster --cluster my-ecs-cluster --region us-east-1 --count 1 hello-world.yml

If you want to use Fargate, run something like:

artillery run-cluster --region eu-west-1 --count 1 --fargate --cluster a9 --public-subnet-ids subnet-id1,subnet-id2,subnet-id3 hello-world.yml

Replace subnet-id1 etc in the example above with IDs of public subnets in a VPC in the AWS account you are using

Using Custom JS code and plugins

Your test scripts may include custom JS via the config.processor attribute in your test script, or plugins via the config.plugins section. For example:

config:
  target: "https://example.net"
  phases:
    - duration: 20
      arrivalRate: 1
  processor: "./functions.js"
  plugins:
    some-plugin: {}
scenarios:
  - flow:
    - get:
        url: "/"

Artillery Pro will automatically resolve any dependencies to external npm packages in your custom JS code as well as any plugins used and make those available when the test script is being run on ECS.

Currently only publicly available npm packages and plugins are resolved and installed automatically. Private npm packages will be supported in an upcoming release.

Running A Named Test

run-cluster may be used with test scripts located on the local disk or with tests created and stored in your AWS account with the create-test command.

For example:

# Package up and upload hello-world.yml and all of its dependencies to S3 in your AWS account:
artillery create-test --name hello-world /path/to/hello-world.yml

# Run the named test:
artillery run-cluster --cluster my-ecs-cluster --region us-east-1 --count 1 hello-world

Using named tests with run-cluster is more efficient since the test script and its dependencies don't need to be uploaded before the test runs. We recommend that test scripts that have been debugged and are unlikely to be modified very frequently are pre-uploaded and named with create-test.

Customizing Runtime Environment

It's possible to customize the runtime environment of the Artillery script being run on ECS via the --launch-config flag. The following customizations are supported:

Example: Using artillery-plugin-datadog

artillery-plugin-datadog is a community plugin which can be used to send metrics generated by your test to Datadog. The plugin expects that the DATADOG_API_KEY environment variable is set, which can be done with:

artillery run-cluster --cluster my-ecs-cluster --region us-east-1 --launch-config '{"environment": [{"name":"DATADOG_API_KEY", "value":"123datadogkey"}]}' hello-world.yml

Support for setting environment variables with encryption using AWS Parameter Store will be available in an upcoming Artillery Pro release.