What you’ll learn
- How to run Artillery tests from the command line
- How to run tests from a single machine with
artillery run
- How to run large-scale distributed tests from AWS with
artillery run-test
⚡- How to accomplish other tasks with the Artillery CLI, such as generating HTML reports
Overview
Requirements
This guide assumes you’ve already read our Installing Artillery guide, and installed Artillery on your machine.
Using the CLI
Listing all commands
Run artillery --help
to see all of the available commands. To see more information about a specific command, such as the full list of flags it supports, use the --help
flag with that command, e.g. to list all flags supported by the run
command:
artillery run --help
Getting version info
Run artillery -V
to print the version of Artillery (and Artillery Pro if installed), as well as information about the environment, such as Node.js version and the OS.
$ artillery -V
___ __ _ ____ _
_____/ | _____/ /_(_) / /__ _______ __ (_)___ _____
/____/ /| | / ___/ __/ / / / _ \/ ___/ / / / / / __ \/____/
/____/ ___ |/ / / /_/ / / / __/ / / /_/ / / / /_/ /____/
/_/ |_/_/ \__/_/_/_/\___/_/ \__, (_)_/\____/
/____/
------------ Version Info ------------
Artillery: 1.6.0
Artillery Pro: 2.2.0
Node.js: v12.16.0
OS: darwin/x64
--------------------------------------
Core Commands
run
This command runs a test script through to completion. The basic way to run a test is with:
artillery run my-script.yaml
Options
Option | Description |
---|---|
--output , -o |
Write a JSON report to a file |
--environment , -e |
Run the test using the specified environment |
--config , -c |
Load config section from another file |
--overrides |
Override values in the test script dynamically |
--target , -t |
Set or override target URL for the test |
--payload , p |
Set path to an external payload file |
--variables , -v |
Set scenario variables dynamically |
--insecure , -k |
Turn off TLS verification. Should not be used in production |
--quiet , -q |
Run in “quiet” mode |
Create a JSON report: artillery run --output <path>
artillery run --output report.json my-script.yaml
You can tell Artillery to write a JSON report of a test run into a file. This JSON report can be used to generate an HTML report with the report
command, or to run queries on with a tool like jq
.
Test different deployment environments: artillery run --environment <name>
A service is usually deployed in more than one place. For example, your team may use PR/preview environments, which create a separate deployment of a service for each PR or feature branch. There’s often one or more staging
deployment and at least one production
deployment. Artillery lets you describe those environments in your test script and switch between to run tests.
Environment-specific configuration is set in config.environments
in the test script, for example:
config:
target: "https://service-foo.acmecorp.digital" # default target
phases:
- arrivalRate: 50
duration: 600
environments:
local-dev:
target: "http://localhost:8080"
phases:
- arrivalRate: 10
duration: 60
preprod:
target: "https://service-foo.preprod.acmecorp.digital"
scenarios:
# Scenario definitions would go here.
The script defines two environments: local-dev
and preprod
. Environment-specific definitions such as target
or phases
will override the top-level ones defined in config
.
You may then run the same test against the service running locally oon localhost:8080
(whilst working on a fix to some feature for example) with:
artillery run --environment local-dev my-script.yaml
And the same script could be used in a post-deployment CI job to run tests against the preprod
version of the service on https://service-foo.preprod.acmecorp.digital
with:
artillery run --environment preprod my-script.yaml
Extract and reuse configuration: artillery run --config <path>
While Artillery test scripts allow for any number of scenarios to be defined in the same test script, it is often useful to keep individual scenarios in their own files. Consider the following example layout of a repository containing Artillery test scripts:
acmecorp-backend-tests/
- services/
- service-foo/
- config.yaml
- scenarios/
- edit-records.yaml
- search-records.yaml
where edit-records.yaml
contains a scenario that exercises record creation/deletion/editing API endpoints, and search-records.yaml
contains a scenario that exercises search/read API endpoints. You can reuse the same config.yaml
file, which would contain common configuration with:
# Run edit-records scenario:
artillery run --config config.yaml scenarios/edit-records.yaml
# Run search-records scenario:
artillery run --config config.yaml scenarios/search-records.yaml
Override parts of the test script on the fly: artillery run --overrides <json>
This flag can be used to override parts of the test script from the command line, instead of editing the file containing the test script.
For example, we could dynamically override phase definition with:
artillery run \
--overrides '{"config": {"phases": [{"duration": 10, "arrivalRate": 1}]}}' \
my-script.yaml
(Note: the value passed to --overrides
must be valid JSON).
Set scenario variables on the fly: artillery run --variables <json>
The following command makes two variables - color
and size
- available in scenario definitions:
artillery run -v '{"color": ["red", "yellow", "blue"], "size":[120, 150, 200]}' my-script.yaml
The variables may be used as normal in scenario definitions, for example:
scenario:
- name: Create record
flow:
- post:
url: "/items"
json:
itemColor: "{{ color }}"
itemSize: "{{ itemSize }}"
As with other variables, each virtual user will get one of the values available, e.g. red
& 120
for one user, yellow
& 150
for another user, and so on.
quick
Use this command to run a quick HTTP test without needing to write any scenarios.
For example, we can run a quick test which opens 20 connections and sends 100 GET requests to /items
as follows:
artillery quick \
--count 20 \
--num 100 \
https://service-foo.preprod.acmecorp.digital/items
The quick
command supports a variety of other options:
Usage: quick [options] <target>
Run a quick test without writing a test script
Options:
-h, --help output usage information
-r, --rate <number> New arrivals per second
-c, --count <number> Fixed number of arrivals
-d, --duration <seconds> Duration of the arrival phase
-n, --num <number> Number of requests each new arrival will send
-t, --content-type <string> Set content-type (defaults to application/json],
-p, --payload <path> Set payload file (CSV)
-o, --output <path> Set file to write stats to (will output to stdout by default)
-k, --insecure Allow insecure TLS connections, e.g. with a self-signed cert
-q, --quiet Do not print anything to stdout
report
Use this command to produce a self-contained HTML report for a test run.
First, run a test and create a JSON report:
artillery run --output report.json my-script.yaml
You can then create an HTML report with:
artillery report --output report.html report.json
Pro Commands ⚡
Overview
Once Artillery Pro is installed, several new commands become available in the artillery
CLI, which are documented here.
run-test
This command is similar to the Core run
command, but instead of running a test from a single machine, the run-test
command runs it from an AWS ECS or Fargate cluster. This command lets you scale up and run your tests in distributed mode across multiple workers, and from a number of geographical regions.
For example, if you have a test script in my-script.yaml
which you would run with Artillery Core as follows:
artillery run my-script.yaml
run-test
makes it easy to scale up and run the same script from the cloud. For example, you would scale up to 20
workers (instead of 1
) running in us-east-1
on Fargate with:
artillery run-test \
--region us-east-1 \
--cluster mycluster \
--count 20 \
--fargate \
my-script.yaml
See Running tests from AWS for a detailed guide on running tests from AWS.
Shared flags
run-test
shares several flags with run
, which allows for a seamless transition from running tests locally to running them in the cloud. The common flags are:
--output
,-o
--environment
,-e
--config
--overrides
--target
,-t
--insecure
,-k
These flags work in the exact same way in run-test
as in run
.
run-test
-specific flags
run-test
has several other flags specifically for configuring how a test should be executed from a cloud environment.
Option | Description |
---|---|
--cluster <name> |
The name of the cluster to run the test from |
--fargate |
Tells Artillery Pro to use Fargate to run the test instead of classic ECS |
--count <number> |
Set the number of workers for running the test |
--region , -r |
The region where the test will run from. Artillery Pro supports 13 different regions |
--launch-config <json> |
A JSON object describing launch configuration for ECS tasks |
--subnet-ids <list> |
[Fargate-only] Comma-separated list of subnet IDs to place tasks into |
--security-group-ids <list> |
[Fargate-only] Comma-separated list of security group IDs to attach to tasks |
See Running tests from AWS for a detailed guide on running tests from AWS.
create-test
Usage: create-test [options] <script>
Package up a test script and name it to speed up running tests with run-test
Options:
-h, --help output usage information
-n, --name <name> The name for the test (to be used in run-cluster)
--config <path> Use common test configuration for this named test
The create-test
command resolves a test script’s dependencies (such as external CSV files, custom JS code and its dependencies etc), and uploads everything needed to run the script to a location is S3. The name given to the test may then be used with the run-test
command to reduce the amount of time it takes to set up a test environment before the test script is run.
The --config
section corresponds to the --config
section in artillery run-test
command.
For example, a named test can be created from a test script in my-script.yaml
as follows:
artillery create-test --name my-named-test my-script.yaml
If the script makes use of CSV files for variable data, plugins, or custom code (which may make use of external npm
dependencies), all of those references will be resolved automatically.
delete-test
This command can be used to delete a named test created with create-test
. For example:
artillery delete-test my-named-test
list-tests
This command lists all named tests created with create-test
.
deploy
Usage: deploy|setup-pro [options]
Set up Artillery Pro for an AWS account
Options:
-h, --help output usage information
-r, --region <region> AWS region to deploy the backend into
--export-terraform-template Export a Terraform template which creates an Artillery Pro CloudFormation stack
--upgrade Upgrade an existing Artillery Pro installation in an AWS account
--license-key-file <path> Path to file containing a license key
--license-key-text <text> Contents of a license key
This command is used to deploy the Artillery Pro backend. Please see Installing Artillery Pro ⚡ for a detailed guide.
set-config-value
This commands can be used to set configuration values which will be available to workers running on ECS/Fargate in a secure way.
artillery set-config-value --name NPM_TOKEN --value 01234567 --region us-east-1
The following config values may be set:
Option | Description |
---|---|
NPM_TOKEN |
Auth token to use with npm |
NPM_REGISTRY |
URL of private npm registry to use |
ARTIFACTORY_AUTH |
Auth token to use with Artifactory |
ARTIFACTORY_EMAIL |
Email to use with Artifactory |
NPMRC |
Customize npmrc with arbitrary config |
See Private npm
packages ⚡ for more information on using private packages in your tests.
subscription
Use this command to show the status of your team’s Artillery Pro subscription, or to request for a license key to be sent to the email address associated with the subscription.
Usage: subscription [options]
Show subscription status of Artillery Pro
Options:
-h, --help output usage information
--email <email> Request license key to be sent to an email address