Overview
Artillery Pro’s run-test
command allows for a test script to be run from an AWS ECS or Fargate cluster. This command lets you scale up and run your tests in distributed mode across multiple workers, and from a number of geographical regions.
Requirements
- Artillery Pro needs to be installed before running tests from AWS.
- AWS credentials which allow access to an ECS/Fargate cluster, and have the required IAM permissions to use
artillery
need to be set up on the local machine
Use classic ECS or Fargate
Artillery Pro can use classic ECS clusters (with container instances managed by your team), or Fargate (to run containers without needing to manage any servers). By default, run-test
will assume that a classic ECS cluster is used. To use Fargate, invoke run-test
with the --fargate
flag, and provide one or more public subnets to launch containers in.
For more information about ECS and Fargate, please see AWS ECS product page.
Should we use classic ECS or Fargate?
Fargate makes for an ideal starting point since there is no extra infrastructure to set up or manage. Your team can go from running tests from their local machines with artillery run
to running distributed cloud tests from AWS with artillery run-test
in a matter of minutes.
Generally, the decision on whether to use classic ECS or Fargate depends on the infrastructure that’s already in place & any standards & processes already used by your team or the wider organization.
Considerations may include:
- Whether there are classic ECS clusters already in place, with capacity to be allocated towards running load tests
- The maximum size of the tests you need to run. Fargate has a default limit of 50 tasks per region, which you may need to request AWS to increase for your account if your team needs to run tests with 50+ workers (or two concurrent tests which use 25+ workers each).
- Workload simulated by your Artillery tests may require a custom CPU/memory configuration. Fargate defines a pre-defined set of configurations, and if your tests required more CPU without the corresponding increase in memory requirements, classic ECS may be a more economical choice.
- Classic ECS clusters comprised of compute-optimized instance types may offer better & more consistent performance, especially for tests which are especially CPU and/or network intensive.
- Classic ECS typically offers much faster worker startup times, especially for larger tests than Fargate.
We would typically recommend that you use Fargate by default for the flexibility and ease-of-use it provides (serverless containers with no need to set up any VMs), unless your team has a classic ECS cluster already in place, or if one or more of the considerations outlined above applies.
Running tests with run-test
Run a Hello World test
All following examples assume that an ECS cluster my-ecs-cluster
has been set up in us-east-1
and that AWS credentials have been set up appropriately to allow the user running the artillery
CLI to access the cluster as described in the installation guide for Artillery Pro).
Create a hello-world.yml
adjusting the value of target
to point to an URL under your control.
config:
target: "https://example.net"
phases:
- duration: 20
arrivalRate: 1
scenarios:
- flow:
- get:
url: "/"
If you are using an ECS cluster with container instances, run:
artillery run-test --cluster my-ecs-cluster --region us-east-1 --count 1 hello-world.yml
If you want to use Fargate, add a --fargate
flag:
artillery run-test --cluster my-ecs-cluster --region us-east-1 --count 1 --fargate hello-world.yml
Using custom JS code and plugins
Your test scripts may include custom JS via the config.processor
attribute in your test script, or plugins via the config.plugins
section. For example:
config:
target: "https://example.net"
phases:
- duration: 20
arrivalRate: 1
processor: "./functions.js"
plugins:
some-plugin: {}
scenarios:
- flow:
- get:
url: "/"
Artillery Pro will automatically resolve any dependencies to external npm packages in your custom JS code as well as any plugins used and make those available when the test script is being run on ECS.
Explicitly bundling files with the test
run-test
will automatically detect any custom JS modules (as well as their npm dependencies) and CSV files used with config.payload
and bundle them into the test package that gets sent to the workers. Sometimes you may want to include other files which cannot be automatically detected (e.g. a file which is read with fs.readFile
in a custom function). You can use config.includeFiles
to tell run-test
to include those files.
config:
target: "https://example.net"
includeFiles:
- foo.json
- bar.xml
Running a named test
run-test
may be used with test scripts located on the local disk or with tests created and stored in your AWS account with the create-test
command.
For example:
# Package up and upload hello-world.yml and all of its dependencies to S3 in your AWS account:
artillery create-test --name hello-world /path/to/hello-world.yml
# Run the named test:
artillery run-test --cluster my-ecs-cluster --region us-east-1 --count 1 hello-world
Using named tests with run-test
is more efficient since the test script and its dependencies don’t need to be uploaded before the test runs. We recommend that test scripts that have been debugged and are unlikely to be modified very frequently are pre-uploaded and named with create-test
.
Customizing runtime environment
It’s possible to customize the runtime environment of the Artillery script being run on ECS via the --launch-config
flag. The following customizations are supported:
cpu
- customize the number of CPU units reserved for each Artillery container; the default value is 1024memory
- customize the amount of memory presented to the Artillery container; the default value is 2048 on Fargate, and 1024 on classic ECS.environment
- customize the environment variables set in the Artillery worker container.ulimits
- customize ulimits such as the number of available file descriptors (nofile
).
Example: Increase the number of file descriptors
If you’re seeing EMFILE
errors, Artillery workers may be running out of file descriptors. This can happen in test runs that open a lot of TCP connections. The default limit set by Artillery is 8192 file descriptors per works, but that limit can be increased. For example, to double the default limit and set it to 16384 for a test run, use:
artillery run-test --cluster my-ecs-cluster --region us-east-1 \
--launch-config '{"ulimits":[{"name":"nofile","softLimit":"16384","hardLimit":"16384"}]}' \
hello-world.yml
Example: Setting environment variables
To set two environment variables named VAR1
and VAR2
for Artillery workers running on ECS/Fargate, run:
artillery run-test --cluster my-ecs-cluster --region us-east-1 \
--launch-config '{"environment": [{"name":"VAR1", "value":"hello"},{"name":"VAR2","value":"world"}]}' hello-world.yml
Support for setting environment variables with encryption using AWS Parameter Store will be available in an upcoming Artillery Pro release.