What you’ll learn

The format of Artillery test scripts

Configuration options in Artillery test scripts

Overview

An Artillery test script is a YAML file composed of two main sections: config and scenarios.

The `config` Section

The config section usually defines the target (the hostname or IP address of the system under test), the load progression, and protocol-specific settings, such as HTTP response timeouts or Socket.io transport options. It may also be used to load and configure plugins and custom JS code.

You can define the following attributes in the config section:

target - The URI of the application under test. For an HTTP application, it’s the base URL for all requests (e.g. http://myapp.staging.local). For a WebSocket server, it’s the hostname (and optionally the port) of the server (e.g. ws://127.0.0.1).
environments - Specify environment-specific configuration (read more).
phases - Specify the duration of the test and frequency of requests (read more).
payload - Used for importing variables from a CSV file (read more).
variables - Set variables inline rather than loading them from an external CSV file.
defaults - Set default headers that will apply to all HTTP requests.
plugins - Configure plugins to add functionality and extend Artillery’s built-in features (read more)
processor - Load custom JS code to be called at certain points during the execution of a scenario (read more).
tls - Configure how Artillery handles self-signed certificates (read more).
ensure - Set success conditions for latency or error rates. Useful for running performance tests in a CI/CD pipeline.

Using environment values in `config`

Values can be set dynamically via environment variables which are available under $processEnvironment template variable. This functionality helps set different configuration values without modifying the test definition and keeping secrets out of your source code.

For example, to set a default HTTP header for all requests via the SERVICE_API_KEY environment variable, your test definition would look like this:

config:
  target: https://my.microservice.internal
  phases:
    - duration: 600
      arrivalRate: 10
  defaults:
    headers:
      x-api-key: "{{ $processEnvironment.SERVICE_API_KEY }}"
scenarios:
  - flow:
      - get:
          url: "/"

You can keep the API key out of the source code and provide it on the fly when executing the test script:

export SERVICE_API_KEY="012345-my-api-key"
artillery run my-test.yaml

Load Phases

A load phase defines how Artillery generates new virtual users (VUs) in a specified time period. For example, a typical performance test will have a gentle warm-up phase, followed by a ramp-up phase, and finalizing with a maximum load for a duration of time.

config.phases is an array of phase definitions that Artillery goes through sequentially. Four kinds of phases are supported:

A phase with a duration and a constant arrival rate of a number of new VUs per second.
A linear ramp-up phase where the number of new arrivals increases linearly throughout the phase.
A phase that generates a fixed count of new arrivals over a period of time.
A pause phase which generates no new VUs for a duration of time.

You can cap the total number of VUs with the maxVusers option for any phase.

The duration of an arrival phase determines only how long virtual users will be generated for. It is not the same as the duration of a test run. How long a given test will run for depends on several factors, such as complexity and length of user scenarios, server response time, and network latency.

Load phase examples

Constant arrival rate

The following example generates 50 virtual users every second for 5 minutes:

config:
  target: "https://staging.example.com"
  phases:
    - duration: 300
      arrivalRate: 50

The following example generates 10 virtual users every second for 5 minutes, with no more than 50 concurrent virtual users at any given time:

config:
  target: "https://staging.example.com"
  phases:
    - duration: 300
      arrivalRate: 10
      maxVusers: 50

Ramp up rate

The following example ramps up the arrival rate of virtual users from 10 to 50 over 2 minutes:

config:
  target: "https://staging.example.com"
  phases:
    - duration: 120
      arrivalRate: 10
      rampTo: 50

Fixed number of arrivals per second

The following example creates 20 virtual users in 60 seconds (one virtual user approximately every 3 seconds):

config:
  target: "https://staging.example.com"
  phases:
    - duration: 60
      arrivalCount: 20

A do-nothing `pause` phase

The following example does not send any virtual users for 60 seconds:

config:
  target: "https://staging.example.com"
  phases:
    - pause: 60

How do ramps work?

Think of the rampTo setting as a shortcut for manually writing out a sequence of arrival phases. For example, let’s say you have the following load phase defined:

phases:
  - duration: 100
    arrivalRate: 0
    rampTo: 50

The above load phase is equivalent to the following:

phases:
  -
    arrivalRate: 0
    duration: 1.96
  -
    arrivalRate: 1
    duration: 1.96
  -
    arrivalRate: 2
    duration: 1.96
  -
    #  ... etc ...
  -
    arrivalRate: 50
    duration: 2

Environments

Typically, you may want to reuse a load testing script across multiple environments with minor tweaks. For instance, you may want to run the same performance tests in your development, staging, and production servers. However, for each environment, you need to set a different target and modify the load phases.

Instead of duplicating your test definition files for each environment, you can use the config.environments setting. It allows you to specify the number of named environments that you can define with environment-specific configuration.

A typical use-case is to define multiple targets with different load phase definitions for each of those systems:

config:
  target: "http://wontresolve.local:3003"
  phases:
    - duration: 10
      arrivalRate: 1
  environments:
    production:
      target: "http://wontresolve.prod:44321"
      phases:
        - duration: 120
          arrivalRate: 10
    staging:
      target: "http://127.0.0.1:3003"
      phases:
        - duration: 1200
          arrivalRate: 20

When running your performance test, you can specify the environment on the command line using the -e flag. For example, to execute the example test script defined above with the staging configuration:

artillery run -e staging my-script.yml

The `$environment` variable

When running your tests in a specific environment, you can access the name of the current environment using the $environment variable.

For example, you can print the name of the current environment from a scenario during test execution:

config:
  target: "http://wontresolve.local:3003"
  phases:
    - duration: 10
      arrivalRate: 1
  environments:
    production:
      target: "http://wontresolve.prod:44321"
      phases:
        - duration: 120
          arrivalRate: 10
    staging:
      target: "http://127.0.0.1:3003"
      phases:
        - duration: 1200
          arrivalRate: 20
scenarios:
  - flow:
    - log: "Current environment is set to: {{ $environment }}"

If you run the test with artillery run -e staging my-script.yml, Artillery will print “Current environment is set to: staging”.

Payload files

It can often be helpful to be able to inject data from external files into your test scenarios. For example, you might have a list of usernames and passwords that you want to use to test authentication in your API.

Payload files are in the CSV format, and Artillery allows you to map each of the rows to a variable name that you can use in scenario definitions. For example, you may have a file named users.csv with the following contents:

testuser1,password1
testuser2,password2
testuser3,password3

To access this information in a test definition, you can load the data from the CSV file using config.payload setting:

config:
  payload:
    # path is relative to the location of the test script
    path: "users.csv"
    fields:
      - "username"
      - "password"
scenarios:
  - flow:
      - post:
          url: "/auth"
          json:
            username: "{{ username }}"
            password: "{{ password }}"

In this example, we tell Artillery to load users.csv file with the path setting and make the variables username and password available in scenarios containing values from one of the rows in the CSV file.

It’s also possible to import multiple CSV files in a test definition by setting payload as an array:

payload:
  -
    path: "pets.csv"
    fields:
      - "species"
      - "name"
  -
    path: "urls.csv"
    fields:
      - "url"

You can also dynamically load different CSV files depending on the environment you set with the -e flag by using the $environment variable when specifying the path:

payload:
  - path: "{{ $environment }}-logins.csv"
    fields:
      - "username"
      - "password"

An example for dynamically loading a payload file is to load a different set of usernames and passwords to use with an authentication endpoint when running the same test in different environments.

Payload file options

order (default: random) - Control how rows are selected from the CSV file for each new virtual user. Set to sequence to iterate through the rows in a sequence (looping around and starting from the beginning after reaching the last row).
skipHeader (default: false) - Set to true to make Artillery skip the first row in the file (typically the header row).
delimiter (default: ,) - If the payload file uses a delimiter other than a comma, set this option to the delimiter character.
cast (default: true) - By default, Artillery will convert fields to native types (e.g. numbers or booleans). To keep those fields as strings, set this option to false.
skipEmptyLines (default: true) - By default, Artillery skips empty lines in the payload. Set to false to include empty lines.

Example

The following example loads a payload file called users.csv, skips the first row, and selects each subsequent row sequentially:

config:
    payload:
      path: "users.csv"
      fields:
        - "username"
        - "password"
      order: sequence
      skipHeader: true
  scenarios:
    - # ... the rest of the script

Inline variables

Variables can be defined in the config.variables setting and used in subsequent request templates.

Variables work similarly to loading fields from a payload file. You can define multiple values for a variable and access them randomly in your scenarios. For instance, the following example defined two variables, {{ id }} and {{ postcode }}, with multiple values:

config:
  target: "http://app01.local.dev"
  phases:
    -
      duration: 300
      arrivalRate: 25
  variables:
    postcode:
      - "SE1"
      - "EC1"
      - "E8"
      - "WH9"
    id:
      - "8731"
      - "9965"
      - "2806"

Variables defined in this block are only available in scenario definitions. They cannot be used to template any values in the config section of your scripts. If you need to dynamically override values in the config section, use environment variables in conjunction with $processEnvironment.

Setting success conditions with `ensure`

When running Artillery in CI/CD pipelines, it can be useful to have Artillery exit with a non-zero code when a condition is not met. Artillery allows you to validate if the aggregate response time latency or error rate meets specific requirements.

Latency

You can check that the aggregate response time latency is under a specific threshold. For example, to check that the aggregate p95 latency of a performance test is 200 milliseconds or less, add the following configuration to your script:

config:
  ensure:
    p95: 200

In this test definition, Artillery will exit with a non-zero exit code if the aggregate p95 is over 200 milliseconds.

You can validate the aggregate latency for min, max, median, p95, and p99.

Error rate

You can also verify that the error rate of your performance test doesn’t exceed a defined percentage. The error rate is the ratio of virtual users that didn’t complete their scenarios successfully to the total number of virtual users created during the test. For instance, if your performance test generates 1000 virtual users and 50 didn’t complete their scenarios successfully, the error rate for the performance test is 5%.

The following example will make Artillery exit with a non-zero exit code if the total error rate exceeded 1%:

config:
  ensure:
    maxErrorRate: 1

The `scenarios` section

The scenarios section contains definitions for one or more scenarios for the virtual users (VUs) that Artillery will create. Each scenario is a series of steps representing a typical sequence of requests or messages sent by a user of an application.

A scenario definition is an object which requires a flow attribute and may contain additional optional attributes:

flow (required) - An array of operations that a virtual user performs. For example, you can execute GET and POST requests for an HTTP-based application or emit events for a Socket.IO test.
name (optional) - Assign a descriptive name to a scenario, which can be helpful in reporting.
weight (optional) - Allows for the probability of a scenario being picked by a new virtual user to be “weighed” relative to other scenarios.

Each Artillery engine used during testing supports additional scenario attributes. Read the documentation to learn what you can do in a scenario for each Artillery engine:

The `before` and `after` sections

The before and after are optional top level sections that can be used to run an arbitrary scenario once per test definition, before or after the scenarios section has run. Any variable captured during the before or after execution will be available to all virtual users. These sections can be useful to set up or tear down test data.

The following example calls an authentication endpoint and captures an auth token before the virtual users arrive. After the scenarios have run, the after section invalidates the token:

config:
  target: "http://app01.local.dev"
  phases:
    - duration: 300
      arrivalRate: 25

before:
  flow:
    - log: "Get auth token"
    - post:
        url: "/auth"
        json:
          username: "myUsername"
          password: "myPassword"
        capture:
          - json: $.id_token
            as: token
scenarios:
  - flow:
    - get:
        url: "/data"
        headers:
          authorization: "Bearer {{ token }}"
after:
  flow:
    - log: "Invalidate token"
    - post:
        url: "/logout"
        json:
          token: "{{ token }}"

Scenario weights

Weights allow you to specify that some scenarios should be picked more often than others. If you have three scenarios with weights 1, 2, and 5, the scenario with the weight of 2 is twice as likely to be picked as the one with a weight of 1, and 2.5 times less likely than the one with a weight of 5. Or in terms of probabilities:

scenario 1: 1/8 = 12.5% probability of being picked
scenario 2: 2/8 = 25% probability of being picked
scenario 3: 5/8 = 62.5% probability of being picked

Scenario weights are optional and set to 1 by default, meaning each scenario has the same probability of getting picked.

What you’ll learn