run - run an Artillery test

artillery run [options] <script>

The run command runs a test script. The basic way to run a test is with:

artillery run my-script.yaml

Options

--output - create a JSON report

You can tell Artillery to write a JSON report of a test run into a file. The JSON report can be used to generate an HTML report with the report command, or to run queries on with a tool like jq (opens in a new tab).

artillery run --output report.json my-script.yaml

Test different deployment environments

A service is usually deployed in more than one place. For example, your team may use pull request or preview environments, which create a separate deployment of a service for each pull request or feature branch. Most teams often have one or more staging deployment and at least one production deployment. Artillery lets you describe those environments in your test script and switch between to run tests.

Environment-specific configuration is set in the config.environments settings in the test script:

config:
  target: 'https://service-foo.acmecorp.digital' # default target
  phases:
    - arrivalRate: 50
      duration: 600
  environments:
    local-dev:
      target: 'http://localhost:8080'
      phases:
        - arrivalRate: 10
          duration: 60
    preprod:
      target: 'https://service-foo.preprod.acmecorp.digital'
scenarios:
  # Scenario definitions would go here.

The above example script defines two environments: local-dev and preprod. Environment-specific definitions such as target or phases will override the top-level ones defined in config.

You may then run the same test against the service running locally on http://localhost:8080 with:

artillery run --environment local-dev my-script.yaml

The same script could be used in a post-deployment CI job to run tests against the preprod version of the service on https://service-foo.preprod.acmecorp.digital with:

artillery run --environment preprod my-script.yaml

Extract and reuse configuration

While Artillery test scripts allow for any number of scenarios to be defined in the same test script, it's useful to keep individual scenarios in their own files for reusability and to prevent duplicating the configuration settings.

Consider the following example layout of a repository containing two Artillery test scripts, edit-records.yaml and search-records.yaml:

acmecorp-backend-tests/
  └── services/
      └── service-foo/
          ├── config.yaml
          └── scenarios/
              ├── edit-records.yaml
              └── search-records.yaml

For both tests, you can define the same config settings to run each test script individually. However, if you need to modify the test configuration (like changing the target URL, for instance), you would need to make the same change on all scripts, which has the risk of forgetting to update all test scripts where needed.

To keep your test script settings identical, you can reuse the same config settings by defining a separate YAML file. In this example, the config.yaml contains common configuration used for both test scripts:

config:
  target: 'https://service-foo.acmecorp.digital'
  phases:
    - arrivalRate: 50
      duration: 600

With a shared configuration, each test script can only contain a scenarios section to define the scenarios for executing the test. You can use the shared configuration with both test scripts using the --config flag:

# Run edit-records scenarios with the shared configuration:
artillery run --config config.yaml scenarios/edit-records.yaml
 
# Run search-records scenarios with the shared configuration:
artillery run --config config.yaml scenarios/search-records.yaml

When using --config, you can merge different configuration settings between the shared configuration file and the test scripts. However, the settings from the shared configuration will take precedence over the same settings inside the test scenario.

Override parts of the test script on the fly

If you want to override parts of a test script from the command line, you can use the --overrides flag. The flag accepts a JSON string with parts of the test script you wish to override.

For example, you have the following test script, containing a phase generating 50 virtual users per second for 10 minutes:

config:
  target: 'https://service-foo.acmecorp.digital'
  phases:
    - arrivalRate: 50
      duration: 600
scenarios:
  # Scenario definitions would go here.

You can use the --overrides flag to override the configuration and run a phase generating 1 virtual user per second for 10 seconds:

artillery run \
    --overrides '{"config": { "phases": [{ "duration": 10, "arrivalRate": 1 }] } }' \
    my-script.yaml

If you're running Artillery on Windows using the Command Prompt, you'll need to wrap the --overrides flag values using double-quotes and escape the inner quotes of the JSON string using backslashes:

artillery run \
    --overrides "{\"config\": { \"phases\": [{ \"duration\": 10, \"arrivalRate\": 1 }] } }" \
    my-script.yaml

If you're running Artillery on Windows using PowerShell, you'll need to wrap the --overrides flag values using single-quotes and escape the inner quotes of the JSON string using backslashes:

artillery run \
    --overrides '{\"config\": { \"phases\": [{ \"duration\": 10, \"arrivalRate\": 1 }] } }' \
    my-script.yaml

Set scenario variables on the fly

To define variables on the fly to use in your test scenarios, the --variables flag allows you to set a JSON string containing one or more variable names with multiple values.

You can also load environment variables from a file with the --env-file flag. You can access those values in the script via the $env variable

The following example makes two variables - color and size - available in scenario definitions:

artillery run \
    --variables '{ "color": ["red", "yellow", "blue"], "size": [120, 150, 200] }' \
    my-script.yaml

The variables may be used as normal in scenario definitions:

scenario:
  - name: Create record
    flow:
      - post:
          url: '/items'
          json:
            itemColor: '{{ color }}'
            itemSize: '{{ size }}'

Each virtual user will get one of the available values for each variable.

If you're running Artillery on Windows using the Command Prompt, you'll need to wrap the --variables flag values using double-quotes and escape the inner quotes of the JSON string using backslashes:

artillery run \
    --variables "{ \"color\": [\"red\", \"yellow\", \"blue\"], \"size\": [120, 150, 200] }" \
    my-script.yaml

If you're running Artillery on Windows using PowerShell, you'll need to wrap the --variables flag values using single-quotes and escape the inner quotes of the JSON string using backslashes:

artillery run \
    --variables '{ \"color\": [\"red\", \"yellow\", \"blue\"], \"size\": [120, 150, 200] }' \
    my-script.yaml

Override the target URL

The --target flag lets you override the specified target URL of a test script:

artillery run --target https://service-bar.acmecorp.digital my-script.yaml

Turn off TLS verification

By default, Artillery will reject SSL certificates that it's unable to validate. You can disable certificate validation with the --insecure flag:

artillery run --insecure my-script.yaml

Suppress output when running tests

When running a test, Artillery prints a report on the console every 10 seconds for the number of scenarios executed during that period. At the end of the performance test, it prints a complete summary.

You can suppress this output if you don't need it (like when running tests on a continuous integration environment) using the --quiet flag:

artillery run --quiet my-script.yaml