Test Script Reference

What you’ll learn

  • The format of Artillery test scripts
  • Configuration options in Artillery test scripts

Overview

An Artillery test script is composed of two main sections: config and scenarios.

The config section usually defines the target (the hostname or IP address of the system under test), the load progression, and protocol-specific settings, such as HTTP response timeouts or Socket.io transport options. It may also be used to load and configure plugins, and custom JS code.

The scenarios section contains definitions for one or more scenarios for the virtual users (VUs) that Artillery will create.

The config Section

  • target - the URI of the application under test. For an HTTP application it’s the base URL for all requests, e.g. http://myapp.staging.local. For a WebSocket server it would be the hostname (and optionally the port) of the server, e.g. ws://127.0.0.1.
  • environments - specify a list of environments, and associated target URLs; see below
  • phases - specify the duration of the test and frequency of requests; see below
  • payload - used for importing variables from a CSV file; see below
  • variables - set variables inline rather than loading them from an external CSV file
  • defaults - set default headers that will apply to all HTTP requests
  • plugins - configure plugins
  • processor - load custom JS code
  • tls - configure how Artillery handles self-signed certificates. See HTTP Reference
  • ensure - set success conditions for latency or error rates; useful for CI/CD

Using dynamic values in config

Values can be set dynamically via environment variables which are available under $processEnvironment template variable.

For example, to set a default HTTP header for all requests via the SERVICE_API_KEY environment variable, your test definition would look like this:

config:
  target: https://my.microservice.internal
  phases:
    - duration: 600
      arrivalRate: 10
  defaults:
    headers:
      x-api-key: "{{ $processEnvironment.SERVICE_API_KEY }}"
scenarios:
  - flow:
      - get:
          url: "/"

Which would allow for the API key to be left out of the source code and provided on-the-fly with:

export SERVICE_API_KEY="012345-my-api-key"
artillery run my-test.yaml

Load Phases

A load phase defines how many new virtual users (VUs) will be generated in a time period. For example, a typical performance test will have a gentle warm up phase, followed by a ramp up phase which is then followed by a maximum load for a duration of time.

config.phases is an array of phase definitions that Artillery goes through sequentially. Four kinds of phases are supported:

  • A phase with a duration and a constant arrival rate of a number of new VUs per second.
  • A linear ramp up phase where the number of new arrivals increases linearly over the duration of the phase.
  • A phase which generates a fixed count of new arrivals over a period of time.
  • A pause phase which generates no new VUs for a duration of time.

Total number of VUs can be capped with the maxVusers option.

This is important so we're going to mention it again. The duration of an arrival phase determines only how long virtual users will be generated for. It is not the same as the duration of a test run. How long a given test will run for depends on several factors, such as complexity and length of user scenarios, server response time, and network latency.

Examples

Create 50 virtual users every second for 5 minutes:

Constant arrival rate

config:
  target: "https://staging.example.com"
  phases:
    - duration: 300
      arrivalRate: 50

Constant arrival rate with no more than 50 concurrent VUs

config:
  target: "https://staging.example.com"
  phases:
    - duration: 300
      arrivalRate: 10
      maxVusers: 50

A ramp up followed by an constant arrival rate

Ramp up arrival rate from 10 to 50 over 2 minutes, followed by 10 minutes at 50 arrivals per second.

config:
  target: "https://staging.example.com"
  phases:
    - duration: 120
      arrivalRate: 10
      rampTo: 50
      name: "Warm up the application"
    - duration: 600
      arrivalRate: 50
      name: "Sustained max load"

(The name attributes are optional. They are useful to help identify load phases in the reports generated by Artillery.)

Fixed count of arrivals

Create 20 virtual users in 60 seconds (approximately one every 3 seconds):

config:
  target: "https://staging.example.com"
  phases:
    - duration: 60
      arrivalCount: 20

A do-nothing pause phase

config:
  target: "https://staging.example.com"
  phases:
    - pause: 60

How do ramps work?

Think of a rampTo as a shortcut for manually writing out a sequence of arrival phases, e.g. the following ramp:

phases:
  - duration: 100
    arrivalRate: 0
    rampTo: 50

is equivalent to:

phases:
  -
    arrivalRate: 0
    duration: 1.96
  -
    arrivalRate: 1
    duration: 1.96
  -
    arrivalRate: 2
    duration: 1.96
  -
     ... etc ...
  -
    arrivalRate: 50
    duration: 2

Environments

Typically we’d want to re-use a load testing script across multiple enviroments (e.g. dev, staging, and production) with minor tweaks. That’s what config.environments is for: a number of named environments can be defined with environment-specific configuration.

For example, a typical use-case is to define multiple targets with different load phase definitions for each of those:

config:
  target: "http://wontresolve.local:3003"
  phases:
    - duration: 10
      arrivalRate: 1
  environments:
    production:
      target: "http://wontresolve.prod:44321"
      phases:
        - duration: 120
          arrivalRate: 10
    staging:
      target: "http://127.0.0.1:3003"
      phases:
        - duration: 1200
          arrivalRate: 20
scenarios:
  - ...

Choose an environment on the command line with the -e flag; e.g. artillery run -e staging my-script.yml.

The $environment variable

The name of the current environment (if set) is available in the $environment variable.

Example - printing the name of the current environment from a scenario:

config:
  #
  # config here
  #
scenarios:
  - flow:
    - log: "Current environment is set to: {{ $environment }}"

Payload Files

It can often be useful to be able to inject data from external files into your test scenarios. For example, you might have a list of usernames and passwords that you want to use to test the auth endpoint in your API.

Payload files are in the CSV format and Artillery allows you to map each of the rows to a variable name that can be used in scenario definitions. For example:

config:
  payload:
    # path is relative to the location of the test script
    path: "users.csv"
    fields:
      - "username"
      - "password"
scenarios:
  - flow:
      - post:
          url: "/auth"
          json:
            username: "{{ username }}"
            password: "{{ password }}"

We tell Artillery to load users.csv file and make variables username and password available in scenarios containing values from one of the rows in the CSV file

To import multiple CSV files "payload" may also be an an array:

payload:
  -
    path: "./pets.csv"
    fields:
      - "species"
      - "name"
  -
    path: "./urls.csv"
    fields:
      - "url"

It’s also possible to load a different CSV file depending on the environment you set with the --environment flag. For example, you may want to load a different set of usernames/passwords to use with an authentication endpoint when running the same test in different environments:

payload:
  - path: "{{ $environment }}-logins.csv"
    fields:
      - "username"
      - "password"

Options

  • order (default: random) – control how rows are picked from the CSV file for each new virtual user. Set to sequence to iterate through the rows in a sequence (looping around and starting from the beginning after the last row has been reached).
  • skipHeader (default: false) – set to true to make Artillery skip the first row in the file (i.e. the header row)
  • delimiter (default: ,) – if a delimiter other than comma is used, set this option to the delimiter character
  • cast (default: true) – by default Artillery will convert fields to native types (e.g. numbers or booleans). To keep those as string, set this option to false.
  • skipEmptyLines – by default Artillery skips empty lines in the file. Set to false to include empty lines.

Example

config:
    payload:
      path: "users.csv"
      fields:
        - "username"
        - "password"
      order: sequence
      skipHeader: true
  scenarios:
    - # ... the rest of the script

Inline variables

Variables can defined in the config.variables section of a script and used in subsequent request templates.

Variables defined in this block are only available in scenario definitions, i.e. they cannot be used to template any values in the `config` section of your scripts. If you need to dynamically override values in the `config` section, use environment variables in conjunction with $processEnvironment.

config:
  target: "http://app01.local.dev"
  phases:
    -
      duration: 300
      arrivalRate: 25
  variables:
    postcode:
      - "SE1"
      - "EC1"
      - "E8"
      - "WH9"
    id:
      - "8731"
      - "9965"
      - "2806"

The variables can then be used in templates as normal. For example: {{ id }} and {{ postcode }}.

Setting success conditions with ensure

When running Artillery in CI/CD pipelines, it can be useful to have Artillery exit with a non-zero code when a condition is not met.

Latency

To check that the aggregate p95 latency is 200ms or less, and make Artillery exit with a non-zero exit code if it’s over 200, add the following configuration to your script:

config:
  ensure:
    p95: 200

min, max, median, p95, and p99 may be set.

Error rate

The error rate is defined as the ratio of virtual users that didn’t complete their scenarios successfully to the total number of virtual users created during the test.

To make Artillery exit with a non-zero exit code if the total error rate exceeded 1%:

config:
  ensure:
    maxErrorRate: 1

The scenarios section

The scenarios section is where one or more virtual user scenarios are defined.

A scenario is a sequence of steps that will be run sequentially which represents a typical sequence of requests or messages sent by a user of an application.

A scenario definiton is an object which must contain a flow attribute and may contain a number of other attributes.

  • flow - a “flow” is an array of operations that a virtual user performs, e.g. GET and POST requests for an HTTP-based application.
  • name - allows to assign a descriptive name to a scenario, e.g. "search for a product and get its details"
  • weight - allows for the probability of a scenario being picked by a new virtual user to be “weighed” relative to other scenarios

Some Artillery engines will also support other scenario attributes, e.g. the HTTP engine allows for scenario-level hooks to be defined.

Scenario weights

Weights allow you to specify that some scenarios should be picked more often than others. If you have three scenarios with weights 1, 2, and 5, the scenario with the weight of 2 is twice as likely to be picked as the one with the weight of 1, and 2.5 times less likely than the one with weight of 5. Or in terms of probabilities:

  • scenario 1: 1/8 = 12.5% probability of being picked
  • scenario 2: 2/8 = 25% probability
  • scenario 3: 5/8 = 62.5% probability

Weights are optional, and if not specified are set to 1 (i.e. each scenario is equally likely to be picked).

Next