What you’ll learn

How to run Artillery tests from the command line

How to run tests from a single machine with artillery run

How to run large-scale distributed tests from AWS with artillery run-test ⚡

How to accomplish other tasks with the Artillery CLI, such as generating HTML reports

Overview

Requirements

This guide assumes you’ve already read our Installing Artillery guide, and installed Artillery on your machine.

Using the CLI

Listing all commands

Run artillery --help to see all of the available commands. To see more information about a specific command, such as the full list of flags it supports, use the --help flag with that command, e.g. to list all flags supported by the run command:

artillery run --help

Getting version info

Run artillery -V to print the version of Artillery (and Artillery Pro if installed), as well as information about the environment, such as Node.js version and the OS.

$ artillery -V

        ___         __  _ ____                  _
  _____/   |  _____/ /_(_) / /__  _______  __  (_)___  _____
 /____/ /| | / ___/ __/ / / / _ \/ ___/ / / / / / __ \/____/
/____/ ___ |/ /  / /_/ / / /  __/ /  / /_/ / / / /_/ /____/
    /_/  |_/_/   \__/_/_/_/\___/_/   \__, (_)_/\____/
                                    /____/

------------ Version Info ------------
Artillery: 1.6.0
Artillery Pro: 2.2.0
Node.js: v12.16.0
OS: darwin/x64
--------------------------------------

Core Commands

`run`

This command runs a test script through to completion. The basic way to run a test is with:

artillery run my-script.yaml

Options

Option	Description
`--output`, `-o`	Write a JSON report to a file
`--environment`, `-e`	Run the test using the specified environment
`--config`, `-c`	Load `config` section from another file
`--overrides`	Override values in the test script dynamically
`--target`, `-t`	Set or override target URL for the test
`--payload`, `p`	Set path to an external payload file
`--variables`, `-v`	Set scenario variables dynamically
`--insecure`, `-k`	Turn off TLS verification. Should not be used in production
`--quiet`, `-q`	Run in “quiet” mode

Create a JSON report: `artillery run --output <path>`

artillery run --output report.json my-script.yaml

You can tell Artillery to write a JSON report of a test run into a file. This JSON report can be used to generate an HTML report with the report command, or to run queries on with a tool like `jq`.

Test different deployment environments: `artillery run --environment <name>`

A service is usually deployed in more than one place. For example, your team may use PR/preview environments, which create a separate deployment of a service for each PR or feature branch. There’s often one or more staging deployment and at least one production deployment. Artillery lets you describe those environments in your test script and switch between to run tests.

Environment-specific configuration is set in config.environments in the test script, for example:

config:
    target: "https://service-foo.acmecorp.digital" # default target
    phases:
      - arrivalRate: 50
        duration: 600
    environments:
        local-dev:
            target: "http://localhost:8080"
            phases:
            - arrivalRate: 10
              duration: 60
       preprod:
           target: "https://service-foo.preprod.acmecorp.digital"
scenarios:
    # Scenario definitions would go here.

The script defines two environments: local-dev and preprod. Environment-specific definitions such as target or phases will override the top-level ones defined in config.

You may then run the same test against the service running locally oon localhost:8080 (whilst working on a fix to some feature for example) with:

artillery run --environment local-dev my-script.yaml

And the same script could be used in a post-deployment CI job to run tests against the preprod version of the service on https://service-foo.preprod.acmecorp.digital with:

artillery run --environment preprod my-script.yaml

Extract and reuse configuration: `artillery run --config <path>`

While Artillery test scripts allow for any number of scenarios to be defined in the same test script, it is often useful to keep individual scenarios in their own files. Consider the following example layout of a repository containing Artillery test scripts:

acmecorp-backend-tests/
  - services/
    - service-foo/
      - config.yaml
      - scenarios/
        - edit-records.yaml
        - search-records.yaml

where edit-records.yaml contains a scenario that exercises record creation/deletion/editing API endpoints, and search-records.yaml contains a scenario that exercises search/read API endpoints. You can reuse the same config.yaml file, which would contain common configuration with:

# Run edit-records scenario:
artillery run --config config.yaml scenarios/edit-records.yaml

# Run search-records scenario:
artillery run --config config.yaml scenarios/search-records.yaml

Override parts of the test script on the fly: `artillery run --overrides <json>`

This flag can be used to override parts of the test script from the command line, instead of editing the file containing the test script.

For example, we could dynamically override phase definition with:

artillery run \
    --overrides '{"config": {"phases": [{"duration": 10, "arrivalRate": 1}]}}' \
    my-script.yaml

(Note: the value passed to --overrides must be valid JSON).

Set scenario variables on the fly: `artillery run --variables <json>`

The following command makes two variables - color and size - available in scenario definitions:

artillery run -v '{"color": ["red", "yellow", "blue"], "size":[120, 150, 200]}' my-script.yaml

The variables may be used as normal in scenario definitions, for example:

scenario:
    - name: Create record
      flow:
          - post:
              url: "/items"
              json:
                  itemColor: "{{ color }}"
                  itemSize: "{{ itemSize }}"

As with other variables, each virtual user will get one of the values available, e.g. red & 120 for one user, yellow & 150 for another user, and so on.

quick

Use this command to run a quick HTTP test without needing to write any scenarios.

For example, we can run a quick test which opens 20 connections and sends 100 GET requests to /items as follows:

artillery quick \
    --count 20 \
    --num 100 \
    https://service-foo.preprod.acmecorp.digital/items

The quick command supports a variety of other options:


Usage: quick [options] <target>

Run a quick test without writing a test script

Options:

  -h, --help                   output usage information
  -r, --rate <number>          New arrivals per second
  -c, --count <number>         Fixed number of arrivals
  -d, --duration <seconds>     Duration of the arrival phase
  -n, --num <number>           Number of requests each new arrival will send
  -t, --content-type <string>  Set content-type (defaults to application/json],
  -p, --payload <path>         Set payload file (CSV)
  -o, --output <path>          Set file to write stats to (will output to stdout by default)
  -k, --insecure               Allow insecure TLS connections, e.g. with a self-signed cert
  -q, --quiet                  Do not print anything to stdout

report

Use this command to produce a self-contained HTML report for a test run.

First, run a test and create a JSON report:

artillery run --output report.json my-script.yaml

You can then create an HTML report with:

artillery report --output report.html report.json

Pro Commands ⚡

Overview

Once Artillery Pro is installed, several new commands become available in the artillery CLI, which are documented here.

run-test

This command is similar to the Core run command, but instead of running a test from a single machine, the run-test command runs it from an AWS ECS or Fargate cluster. This command lets you scale up and run your tests in distributed mode across multiple workers, and from a number of geographical regions.

For example, if you have a test script in my-script.yaml which you would run with Artillery Core as follows:

artillery run my-script.yaml

run-test makes it easy to scale up and run the same script from the cloud. For example, you would scale up to 20 workers (instead of 1) running in us-east-1 on Fargate with:

artillery run-test \
    --region us-east-1 \
    --cluster mycluster \
    --count 20 \
    --launch-type ecs:fargate \
    my-script.yaml

See Running tests from AWS for a detailed guide on running tests from AWS.

Shared flags

run-test shares several flags with run, which allows for a seamless transition from running tests locally to running them in the cloud. The common flags are:

--output, -o
--environment, -e
--config
--overrides
--target, -t
--insecure, -k

These flags work in the exact same way in run-test as in run.

`run-test`-specific flags

run-test has several other flags specifically for configuring how a test should be executed from a cloud environment.

Option	Description
`--cluster <name>`	The name of the cluster to run the test from
`--launch-type`	Set the platform to run tests from. Possible values: `ecs:fargate` and `ecs:ec2` (default)
`--count <number>`	Set the number of workers for running the test
`--secret <name>`	Make a secret created with `set-secret` command available in the test as an environment variable
`--region`, `-r`	The region where the test will run from. Artillery Pro supports 13 different regions
`--launch-config <json>`	A JSON object describing launch configuration for ECS tasks
`--subnet-ids <list>`	[Fargate-only] Comma-separated list of subnet IDs to place tasks into
`--security-group-ids <list>`	[Fargate-only] Comma-separated list of security group IDs to attach to tasks

See Running tests from AWS for a detailed guide on running tests from AWS.

create-test


Usage: create-test [options] <script>

Package up a test script and name it to speed up running tests with run-test

Options:

  -h, --help         output usage information
  -n, --name <name>  The name for the test (to be used in run-cluster)
  --config <path>    Use common test configuration for this named test

The create-test command resolves a test script’s dependencies (such as external CSV files, custom JS code and its dependencies etc), and uploads everything needed to run the script to a location is S3. The name given to the test may then be used with the run-test command to reduce the amount of time it takes to set up a test environment before the test script is run.

The --config section corresponds to the --config section in artillery run-test command.

For example, a named test can be created from a test script in my-script.yaml as follows:

artillery create-test --name my-named-test my-script.yaml

If the script makes use of CSV files for variable data, plugins, or custom code (which may make use of external npm dependencies), all of those references will be resolved automatically.

delete-test

This command can be used to delete a named test created with create-test. For example:

artillery delete-test my-named-test

list-tests

This command lists all named tests created with create-test.

deploy


Usage: deploy|setup-pro [options]

Set up Artillery Pro for an AWS account

Options:

  -h, --help                   output usage information
  -r, --region <region>        AWS region to deploy the backend into
  --export-terraform-template  Export a Terraform template which creates an Artillery Pro CloudFormation stack
  --upgrade                    Upgrade an existing Artillery Pro installation in an AWS account
  --license-key-file <path>    Path to file containing a license key
  --license-key-text <text>    Contents of a license key

This command is used to deploy the Artillery Pro backend. Please see Installing Artillery Pro ⚡ for a detailed guide.

set-secret

Create an encrypted secret value which can be used in test runs.

artillery create-secret --name API_KEY --value 012345 --region us-east-1

The secret may then be made available in a test run:

artillery run-test --secret API_KEY --launch-type ecs:fargate my-test.yml

The value is available as an API_KEY environment variable accessible via $processEnvironment variable, e.g.:

- post:
    url: "/pets"
    headers:
      x-api-key: "{{ $processEnvironment.API_KEY }}"
    json:
      name: Tiki
      species: pony

How are the secrets stored?

The secrets are encrypted and stored securely in AWS Parameter Store in your AWS account (under /artilleryio/ scope).

set-config-value

This commands can be used to set configuration values which will be available to workers running on ECS/Fargate in a secure way.

artillery set-config-value --name NPM_TOKEN --value 01234567 --region us-east-1

The following config values may be set:

Option	Description
`NPM_TOKEN`	Auth token to use with `npm`
`NPM_REGISTRY`	URL of private `npm` registry to use
`ARTIFACTORY_AUTH`	Auth token to use with Artifactory
`ARTIFACTORY_EMAIL`	Email to use with Artifactory
`NPMRC`	Customize `npmrc` with arbitrary config

See Private `npm` packages ⚡ for more information on using private packages in your tests.

subscription

Use this command to show the status of your team’s Artillery Pro subscription, or to request for a license key to be sent to the email address associated with the subscription.


Usage: subscription [options]

Show subscription status of Artillery Pro

Options:

  -h, --help       output usage information
  --email <email>  Request license key to be sent to an email address

What you’ll learn

Overview

Requirements

Using the CLI

Listing all commands

Getting version info

Core Commands

run

Options

Create a JSON report: artillery run --output <path>

Test different deployment environments: artillery run --environment <name>

Extract and reuse configuration: artillery run --config <path>

Override parts of the test script on the fly: artillery run --overrides <json>

Set scenario variables on the fly: artillery run --variables <json>

quick

report

Pro Commands ⚡

Overview

run-test

Shared flags

run-test-specific flags

create-test

delete-test

list-tests

deploy

set-secret

set-config-value

subscription

`run`

Create a JSON report: `artillery run --output <path>`

Test different deployment environments: `artillery run --environment <name>`

Extract and reuse configuration: `artillery run --config <path>`

Override parts of the test script on the fly: `artillery run --overrides <json>`

Set scenario variables on the fly: `artillery run --variables <json>`

`run-test`-specific flags