What you’ll learn
- The format of Artillery test scripts
- Configuration options in Artillery test scripts
Overview
An Artillery test script is composed of two main sections: config
and scenarios
.
The config
section usually defines the target (the hostname or IP address of the system under test), the load progression, and protocol-specific settings, such as HTTP response timeouts or Socket.io transport options. It may also be used to load and configure plugins, and custom JS code.
The scenarios
section contains definitions for one or more scenarios for the virtual users (VUs) that Artillery will create.
The config
Section
target
- the URI of the application under test. For an HTTP application it’s the base URL for all requests, e.g.http://myapp.staging.local
. For a WebSocket server it would be the hostname (and optionally the port) of the server, e.g.ws://127.0.0.1
.environments
- specify a list of environments, and associated target URLs; see belowphases
- specify the duration of the test and frequency of requests; see belowpayload
- used for importing variables from a CSV file; see belowvariables
- set variables inline rather than loading them from an external CSV filedefaults
- set default headers that will apply to all HTTP requestsplugins
- configure pluginsprocessor
- load custom JS codetls
- configure how Artillery handles self-signed certificates. See HTTP Referenceensure
- set success conditions for latency or error rates; useful for CI/CD
Using dynamic values in config
Values can be set dynamically via environment variables which are available under $processEnvironment
template variable.
For example, to set a default HTTP header for all requests via the SERVICE_API_KEY
environment variable, your test definition would look like this:
config:
target: https://my.microservice.internal
phases:
- duration: 600
arrivalRate: 10
defaults:
headers:
x-api-key: "{{ $processEnvironment.SERVICE_API_KEY }}"
scenarios:
- flow:
- get:
url: "/"
Which would allow for the API key to be left out of the source code and provided on-the-fly with:
export SERVICE_API_KEY="012345-my-api-key"
artillery run my-test.yaml
Load Phases
A load phase defines how many new virtual users (VUs) will be generated in a time period. For example, a typical performance test will have a gentle warm up phase, followed by a ramp up phase which is then followed by a maximum load for a duration of time.
config.phases
is an array of phase definitions that Artillery goes through sequentially. Four kinds of phases are supported:
- A phase with a duration and a constant arrival rate of a number of new VUs per second.
- A linear ramp up phase where the number of new arrivals increases linearly over the duration of the phase.
- A phase which generates a fixed count of new arrivals over a period of time.
- A pause phase which generates no new VUs for a duration of time.
Total number of VUs can be capped with the maxVusers
option.
This is important so we’re going to mention it again. The duration of an arrival phase determines only how long virtual users will be generated for. It is not the same as the duration of a test run. How long a given test will run for depends on several factors, such as complexity and length of user scenarios, server response time, and network latency.
Examples
Create 50 virtual users every second for 5 minutes:
Constant arrival rate
config:
target: "https://staging.example.com"
phases:
- duration: 300
arrivalRate: 50
Constant arrival rate with no more than 50 concurrent VUs
config:
target: "https://staging.example.com"
phases:
- duration: 300
arrivalRate: 10
maxVusers: 50
A ramp up followed by an constant arrival rate
Ramp up arrival rate from 10 to 50 over 2 minutes, followed by 10 minutes at 50 arrivals per second.
config:
target: "https://staging.example.com"
phases:
- duration: 120
arrivalRate: 10
rampTo: 50
name: "Warm up the application"
- duration: 600
arrivalRate: 50
name: "Sustained max load"
(The name
attributes are optional. They are useful to help identify load phases in the reports generated by Artillery.)
Fixed count of arrivals
Create 20 virtual users in 60 seconds (approximately one every 3 seconds):
config:
target: "https://staging.example.com"
phases:
- duration: 60
arrivalCount: 20
A do-nothing pause
phase
config:
target: "https://staging.example.com"
phases:
- pause: 60
How do ramps work?
Think of a rampTo
as a shortcut for manually writing out a sequence of arrival phases, e.g. the following ramp:
phases:
- duration: 100
arrivalRate: 0
rampTo: 50
is equivalent to:
phases:
-
arrivalRate: 0
duration: 1.96
-
arrivalRate: 1
duration: 1.96
-
arrivalRate: 2
duration: 1.96
-
... etc ...
-
arrivalRate: 50
duration: 2
Environments
Typically we’d want to re-use a load testing script across multiple enviroments (e.g. dev
, staging
, and production
) with minor tweaks. That’s what config.environments
is for: a number of named environments can be defined with environment-specific configuration.
For example, a typical use-case is to define multiple targets with different load phase definitions for each of those:
config:
target: "http://wontresolve.local:3003"
phases:
- duration: 10
arrivalRate: 1
environments:
production:
target: "http://wontresolve.prod:44321"
phases:
- duration: 120
arrivalRate: 10
staging:
target: "http://127.0.0.1:3003"
phases:
- duration: 1200
arrivalRate: 20
scenarios:
- ...
Choose an environment on the command line with the -e
flag; e.g. artillery run -e staging my-script.yml
.
The $environment
variable
The name of the current environment (if set) is available in the $environment
variable.
Example - printing the name of the current environment from a scenario:
config:
#
# config here
#
scenarios:
- flow:
- log: "Current environment is set to: {{ $environment }}"
Payload Files
It can often be useful to be able to inject data from external files into your test scenarios. For example, you might have a list of usernames and passwords that you want to use to test the auth endpoint in your API.
Payload files are in the CSV format and Artillery allows you to map each of the rows to a variable name that can be used in scenario definitions. For example:
config:
payload:
# path is relative to the location of the test script
path: "users.csv"
fields:
- "username"
- "password"
scenarios:
- flow:
- post:
url: "/auth"
json:
username: "{{ username }}"
password: "{{ password }}"
We tell Artillery to load users.csv
file and make variables username
and password
available in scenarios containing values from one of the rows in the CSV file
To import multiple CSV files "payload"
may also be an an array:
payload:
-
path: "./pets.csv"
fields:
- "species"
- "name"
-
path: "./urls.csv"
fields:
- "url"
It’s also possible to load a different CSV file depending on the environment you set with the --environment
flag. For example, you may want to load a different set of usernames/passwords to use with an authentication endpoint when running the same test in different environments:
payload:
- path: "{{ $environment }}-logins.csv"
fields:
- "username"
- "password"
Options
order
(default:random
) – control how rows are picked from the CSV file for each new virtual user. Set tosequence
to iterate through the rows in a sequence (looping around and starting from the beginning after the last row has been reached).skipHeader
(default:false
) – set totrue
to make Artillery skip the first row in the file (i.e. the header row)delimiter
(default:,
) – if a delimiter other than comma is used, set this option to the delimiter charactercast
(default:true
) – by default Artillery will convert fields to native types (e.g. numbers or booleans). To keep those as string, set this option tofalse
.skipEmptyLines
– by default Artillery skips empty lines in the file. Set tofalse
to include empty lines.
Example
config:
payload:
path: "users.csv"
fields:
- "username"
- "password"
order: sequence
skipHeader: true
scenarios:
- # ... the rest of the script
Inline variables
Variables can defined in the config.variables
section of a script and used in subsequent request templates.
Variables defined in this block are only available in scenario definitions, i.e. they cannot be used to template any values in the config
section of your scripts. If you need to dynamically override values in the config
section, use environment variables in conjunction with $processEnvironment
.
config:
target: "http://app01.local.dev"
phases:
-
duration: 300
arrivalRate: 25
variables:
postcode:
- "SE1"
- "EC1"
- "E8"
- "WH9"
id:
- "8731"
- "9965"
- "2806"
The variables can then be used in templates as normal. For example: and
.
Setting success conditions with ensure
When running Artillery in CI/CD pipelines, it can be useful to have Artillery exit with a non-zero code when a condition is not met.
Latency
To check that the aggregate p95
latency is 200ms or less, and make Artillery exit with a non-zero exit code if it’s over 200, add the following configuration to your script:
config:
ensure:
p95: 200
min
, max
, median
, p95
, and p99
may be set.
Error rate
The error rate is defined as the ratio of virtual users that didn’t complete their scenarios successfully to the total number of virtual users created during the test.
To make Artillery exit with a non-zero if the total error rate exceeded 1%:
config:
ensure:
maxErrorRate: 1
The scenarios
section
The scenarios
section is where one or more virtual user scenarios are defined.
A scenario is a sequence of steps that will be run sequentially which represents a typical sequence of requests or messages sent by a user of an application.
A scenario definiton is an object which must contain a flow
attribute and may contain a number of other attributes.
flow
- a “flow” is an array of operations that a virtual user performs, e.g. GET and POST requests for an HTTP-based application.name
- allows to assign a descriptive name to a scenario, e.g."search for a product and get its details"
weight
- allows for the probability of a scenario being picked by a new virtual user to be “weighed” relative to other scenarios
Some Artillery engines will also support other scenario attributes, e.g. the HTTP engine allows for scenario-level hooks to be defined.
Scenario weights
Weights allow you to specify that some scenarios should be picked more often than others. If you have three scenarios with weights 1
, 2
, and 5
, the scenario with the weight of 2
is twice as likely to be picked as the one with the weight of 1
, and 2.5 times less likely than the one with weight of 5
. Or in terms of probabilities:
- scenario 1: 1/8 = 12.5% probability of being picked
- scenario 2: 2/8 = 25% probability
- scenario 3: 5/8 = 62.5% probability
Weights are optional, and if not specified are set to 1
(i.e. each scenario is equally likely to be picked).