What you’ll learn

How to run Artillery tests from the command line

How to run tests from a single machine with artillery run

How to run large-scale distributed tests from AWS with artillery run-test

How to accomplish other tasks with the Artillery CLI, such as generating HTML reports

Overview

Requirements

This guide assumes you’ve already read our Installing Artillery guide, and installed Artillery on your machine.

Using the CLI

Listing all commands

Run artillery --help to see all of the available commands.

To see more information about a specific command, such as the full list of flags it supports, use the --help flag with that command. For instance, to list all flags supported by the run command:

artillery run --help

Getting version info

Run artillery -V to print the version of Artillery (and Artillery Pro if installed), as well as information about the environment, such as Node.js version and the OS.

$ artillery -V

        ___         __  _ ____                  _
  _____/   |  _____/ /_(_) / /__  _______  __  (_)___  _____
 /____/ /| | / ___/ __/ / / / _ \/ ___/ / / / / / __ \/____/
/____/ ___ |/ /  / /_/ / / /  __/ /  / /_/ / / / /_/ /____/
    /_/  |_/_/   \__/_/_/_/\___/_/   \__, (_)_/\____/
                                    /____/

------------ Version Info ------------
Artillery: 1.6.0
Artillery Pro: 2.2.0
Node.js: v12.16.0
OS: darwin/x64
--------------------------------------

Core Commands

`run`

artillery run [options] <script>

The run command executes a test script through to completion. The basic way to run a test is with:

artillery run my-script.yaml

Options

Option	Description
`--output`, `-o`	Write a JSON report to a file
`--environment`, `-e`	Run the test using the specified environment
`--config`, `-c`	Load `config` section from another file
`--overrides`	Override values in the test script dynamically
`--target`, `-t`	Set or override target URL for the test
`--variables`, `-v`	Set scenario variables dynamically
`--insecure`, `-k`	Turn off TLS verification. Should not be used in production
`--quiet`, `-q`	Run in “quiet” mode

Create a JSON report

You can tell Artillery to write a JSON report of a test run into a file. The JSON report can be used to generate an HTML report with the report command, or to run queries on with a tool like `jq`.

artillery run --output report.json my-script.yaml

You can also use the online Artillery Report Viewer to visualize the test run.

Test different deployment environments

A service is usually deployed in more than one place. For example, your team may use pull request or preview environments, which create a separate deployment of a service for each pull request or feature branch. Most teams often have one or more staging deployment and at least one production deployment. Artillery lets you describe those environments in your test script and switch between to run tests.

Environment-specific configuration is set in the config.environments settings in the test script:

config:
  target: "https://service-foo.acmecorp.digital" # default target
  phases:
    - arrivalRate: 50
      duration: 600
  environments:
    local-dev:
      target: "http://localhost:8080"
      phases:
      - arrivalRate: 10
        duration: 60
    preprod:
      target: "https://service-foo.preprod.acmecorp.digital"
scenarios:
    # Scenario definitions would go here.

The above example script defines two environments: local-dev and preprod. Environment-specific definitions such as target or phases will override the top-level ones defined in config.

You may then run the same test against the service running locally on http://localhost:8080 with:

artillery run --environment local-dev my-script.yaml

The same script could be used in a post-deployment CI job to run tests against the preprod version of the service on https://service-foo.preprod.acmecorp.digital with:

artillery run --environment preprod my-script.yaml

Extract and reuse configuration

While Artillery test scripts allow for any number of scenarios to be defined in the same test script, it’s useful to keep individual scenarios in their own files for reusability and to prevent duplicating the configuration settings.

Consider the following example layout of a repository containing two Artillery test scripts, edit-records.yaml and search-records.yaml:

acmecorp-backend-tests/
  └── services/
      └── service-foo/
          ├── config.yaml
          └── scenarios/
              ├── edit-records.yaml
              └── search-records.yaml

For both tests, you can define the same config settings to run each test script individually. However, if you need to modify the test configuration (like changing the target URL, for instance), you would need to make the same change on all scripts, which has the risk of forgetting to update all test scripts where needed.

To keep your test script settings identical, you can reuse the same config settings by defining a separate YAML file. In this example, the config.yaml contains common configuration used for both test scripts:

config:
  target: "https://service-foo.acmecorp.digital"
  phases:
    - arrivalRate: 50
      duration: 600

With a shared configuration, each test script can only contain a scenarios section to define the scenarios for executing the test. You can use the shared configuration with both test scripts using the --config flag:

# Run edit-records scenarios with the shared configuration:
artillery run --config config.yaml scenarios/edit-records.yaml

# Run search-records scenarios with the shared configuration:
artillery run --config config.yaml scenarios/search-records.yaml

When using --config, you can merge different configuration settings between the shared configuration file and the test scripts. However, the settings from the shared configuration will take precedence over the same settings inside the test scenario.

Override parts of the test script on the fly

If you want to override parts of a test script from the command line, you can use the --overrides flag. The flag accepts a JSON string with parts of the test script you wish to override.

For example, you have the following test script, containing a phase generating 50 virtual users per second for 10 minutes:

config:
  target: "https://service-foo.acmecorp.digital"
  phases:
    - arrivalRate: 50
      duration: 600
scenarios:
  # Scenario definitions would go here.

You can use the --overrides flag to override the configuration and run a phase generating 1 virtual user per second for 10 seconds:

artillery run \
    --overrides '{"config": { "phases": [{ "duration": 10, "arrivalRate": 1 }] } }' \
    my-script.yaml

You can also use the --overrides flag to add new settings or scenarios to a test script:

artillery run \
    --overrides '{"config": { "processor": "./my-functions.js" } }' \
    my-script.yaml

Set scenario variables on the fly

To define variables on the fly to use in your test scenarios, the --variables flag allows you to set a JSON string containing one or more variable names with multiple values.

The following example makes two variables - color and size - available in scenario definitions:

artillery run \
    --variables '{ "color": ["red", "yellow", "blue"], "size": [120, 150, 200] }' \
    my-script.yaml

The variables may be used as normal in scenario definitions:

scenario:
  - name: Create record
    flow:
      - post:
          url: "/items"
          json:
            itemColor: "{{ color }}"
            itemSize: "{{ size }}"

As with setting variables in the configuration, each virtual user will get one of the available values for each variable.

Override the target URL

The --target flag lets you override the specified target URL of a test script:

artillery run --target https://service-bar.acmecorp.digital my-script.yaml

Turn off TLS verification

By default, Artillery will reject SSL certificates that it’s unable to validate. You can disable certificate validation with the --insecure flag:

artillery run --insecure my-script.yaml

This flag is useful for testing a service in a development or staging environment that doesn’t have a SSL certificate that can be validated, like a self-signed certificate.

You should never use the `--insecure` flag on a production environment. Ignoring certificate errors in a production system can lead to potential security vulnerabilities, like man-in-the-middle attacks.

Suppress output when running tests

When running a test, Artillery prints a report on the console every 10 seconds for the number of scenarios executed during that period. At the end of the performance test, it prints a complete summary.

You can suppress this output if you don’t need it (like when running tests on a continuous integration environment) using the --quiet flag:

artillery run --quiet my-script.yaml

quick

artillery quick [options] <target>

Use this command to run a quick HTTP test without needing to write any scenarios or need a test script.

The `quick` command only works to test HTTP services. You cannot run quick tests for other services, such as Socket.IO or WebSockets.

Options

Option	Description
`--rate`, `-r`	Specify the number of new arrivals per second
`--count`, `-c`	Specify a fixed number of arrivals
`--duration`, `-d`	Specify the duration of the arrival phase
`--num`, `-n`	Specify the number of GET requests each new arrival will send
`--content-type`, `-t`	Set the Content-Type for the request (defaults to `application/json`)
`--output`, `-o`	Write a JSON report to a file
`--insecure`, `-k`	Allow insecure TLS connections
`--quiet`, `-q`	Run in “quiet” mode

Examples

Run a quick test which generates 20 virtual users, each sending 100 GET requests to the specified target:

artillery quick \
    --count 20 \
    --num 100 \
    https://service-foo.preprod.acmecorp.digital/items

Run a quick test which generates 10 virtual users per second, each sending 50 GET requests to the specified target:

artillery quick \
    --rate 10 \
    --num 50 \
    https://service-foo.preprod.acmecorp.digital/items

report

artillery report [options] <file>

The report command converts a JSON report generated by Artillery into a self-contained HTML report.

Options

Option	Description
`--output`, `-o`	Set the filename to write the HTML report

Generating an HTML report

First, run a test and create a JSON report with the --output flag.

artillery run --output report.json my-script.yaml

You can then use the generated JSON report to create an HTML report:

artillery report --output report.html report.json

If you don’t need to save the HTML report into a file, you can omit the --output flag to open the report in your default browser.

Pro Commands

Once Artillery Pro is installed, several new commands become available in the artillery CLI.

run-test

artillery run-test [options] <test>

This command is similar to the Core run command, but instead of running a test from a single machine, the run-test command runs it from an Amazon Elastic Container Service or Amazon Fargate cluster. This command lets you scale up and run your tests in distributed mode across multiple workers, and from a number of geographical regions.

For example, if you have a test script called my-script.yaml, you can run it in a single, local instance with Artillery Core using the run command:

artillery run my-script.yaml

The run-test command makes it easy to scale up and run the same script from the cloud. For example, you can scale up to 20 workers (instead of single worker) running in the us-east-1 region on Fargate with the following Artillery Pro command:

artillery run-test \
    --region us-east-1 \
    --cluster mycluster \
    --count 20 \
    --launch-type ecs:fargate \
    my-script.yaml

See Running tests from AWS for a detailed guide on running Artillery tests from AWS.

Options

The run-test command shares several flags with run, which allows for a seamless transition from running tests locally to running them in the cloud.

Option	Description
`--output`, `-o`	Write a JSON report to a file
`--environment`, `-e`	Run the test using the specified environment
`--config`, `-c`	Load `config` section from another file
`--overrides`	Override values in the test script dynamically
`--target`, `-t`	Set or override target URL for the test
`--insecure`, `-k`	Turn off TLS verification. Should not be used in production

These flags work in the exact same way in run-test as in run.

The run-test command has several other flags specifically for configuring how a test should be executed from a cloud environment.

Option	Description
`--cluster`	The name of the cluster to run the test from
`--launch-type`	Set the platform to run tests from. Possible values: `ecs:fargate` and `ecs:ec2` (default)
`--count`	Set the number of workers for running the test
`--secret`	Make a secret created with `set-secret` command available in the test as an environment variable
`--region`, `-r`	The region where the test will run from. Artillery Pro supports 13 different regions
`--launch-config`	A JSON object describing launch configuration for ECS tasks
`--subnet-ids`	[Fargate-only] Comma-separated list of subnet IDs to place tasks into
`--security-group-ids`	[Fargate-only] Comma-separated list of security group IDs to attach to tasks

Examples

Run a test by launching 10 workers on Amazon Fargate in the US West (Oregon) region:

artillery run-test \
    --region us-west-2 \
    --cluster mycluster \
    --count 10 \
    --launch-type ecs:fargate \
    my-script.yaml

Run a test by launching 20 workers on Amazon ECS in the Europe (Ireland) region:

artillery run-test \
    --region eu-west-1 \
    --cluster mycluster \
    --count 20 \
    --launch-type ecs:ec2 \
    my-script.yaml

get-tests

artillery get-tests

The get-tests command list all running tests.

Options

Option	Description
`--json`	Format output as JSON

stop-test

The stop-test command stops a running test gracefully, by stopping worker tasks and performing other clean up tasks as necessary.

artillery stop-test <test-id>

The run-test command prints test IDs at the beginning of each test run. IDs of currently running tests may also be seen by running artillery get-tests.

Example

Stop a running test with id = e84a482c-cf34-404c-a95d-22446b92a2c1.

artillery stop-test e84a482c-cf34-404c-a95d-22446b92a2c1

create-test

artillery create-test [options] <script>

The create-test command resolves a test script’s dependencies (such as external CSV files, custom JS code and its dependencies), and uploads everything needed to run the script to a location in Amazon S3. The name given to the test may then be used with the run-test command to reduce the amount of time it takes to set up a test environment before the running the test script.

Options

Option	Description
`--name`, `-n`	The name for the test (to be used in `run-test`)
`--config`	Use common test configuration for this named test

Examples

Create a named test from a test script in my-script.yaml called my-named-test:

artillery create-test --name my-named-test my-script.yaml

If the script makes use of CSV files for variable data, plugins, or custom code (which may make use of external npm dependencies), all of those references will be resolved automatically.

After creating the test, you can run the test using run-test:

artillery run-test \
    --region ap-northeast-1 \
    --cluster my-named-test \
    --count 25 \
    --launch-type ecs:fargate \
    my-script.yaml

delete-test

artillery delete-test <name>

This command can be used to delete a named test created with create-test.

Examples

Delete an existing named test called my-named-test:

artillery delete-test my-named-test

list-tests

artillery list-tests [options]

This command lists all named tests created with create-test.

Options

Option	Description
`--json`	Format output as JSON

deploy

artillery deploy [options]

The deploy command sets up the Artillery Pro backend components in your AWS account. A CloudFormation stack is created (or updated) with the resources needed to run Artillery Pro tests. See Installing Artillery Pro for a detailed guide on deploying the backend.

Options

Option	Description
`--region`, `-r`	The AWS region to deploy the backend into
`--upgrade`	Upgrade an existing Artillery Pro installation in an AWS account
`--license-key-file`	Specify the path to file containing an Artillery Pro license key
`--license-key-text`	Specify the contents of an Artillery Pro license key
`--set <key=value>`	Customize names of resources in the CloudFormation stack

Upgrading an existing installation

To upgrade an existing installation of Artillery Pro, install the latest version of the CLI, and run the deploy command in the region in which the backend is already deployed, e.g.:

artillery deploy --upgrade --region us-east-1

Customizing CloudFormation resource names

You can customize the names of some resources created by the Artillery Pro CloudFormation stack (to follow naming standards in your organization for example). The --set flag currently supports the following options:

Key	Description
`cli-user-role-name`	The name of IAM role for CLI users
`cli-user-policy-name`	The name of IAM managed policy assumed by CLI user role

Example: set a custom name for the IAM role for CLI users who run tests

This examples presumes an existing installation in us-east-1 that we’re wishing to upgrade, and set the name of the IAM role generated for Artillery Pro CLI users to tools-sre-artillery.

artillery deploy --set cli-user-role-name=tools-sre-artillery --region us-east-1

set-secret

artillery set-secret [options]

The set-secret command creates an encrypted secret value which can be used in test runs. You can then access these values in your test scenarios via the $processEnvironment variable.

How are the secrets stored?

The secrets are encrypted and stored securely in AWS Parameter Store in your AWS account (under /artilleryio/ scope).

Artillery Pro will support additional secret storage backends in future.

Options

Option	Description
`--name`, `-n`	Name of the secret
`--value`, `-v`	Value of the secret
`--backend`	Specify the secret storage backend (currently only `ssm`)
`--region`, `-r`	AWS region where the secret should be created

Examples

Create a secret called API_KEY with a value of 012345 in the US East (N. Virginia) region:

artillery create-secret --name API_KEY --value 012345 --region us-east-1

You can then make this secret available in a test run using the --secret flag in the run-test command:

artillery run-test --secret API_KEY --launch-type ecs:fargate my-test.yml

The value for the secret will be available as an API_KEY environment variable accessible via $processEnvironment variable:

- post:
    url: "/pets"
    headers:
      x-api-key: "{{ $processEnvironment.API_KEY }}"
    json:
      name: Tiki
      species: pony

set-config-value

artillery set-config-value [options]

This command can be used to set specific configuration values which will be available to workers running on ECS/Fargate in a secure way. These configuration values can be used for installing private packages from npm and Artifactory, and for customizing your npm configuration.

See Private `npm` packages for more information on using private packages in your tests.

Options

Option	Description
`--name`, `-n`	Name of the configuration parameter (see available parameters below)
`--value`, `-v`	Value of the configuration parameter
`--region`, `-r`	AWS region where Artillery Pro is deployed

Configuration parameters

You can set values for the following configuration parameters:

Parameter	Description
`NPM_TOKEN`	Auth token to use with `npm`
`NPM_REGISTRY`	URL of private `npm` registry to use
`ARTIFACTORY_AUTH`	Auth token to use with Artifactory
`ARTIFACTORY_EMAIL`	Email to use with Artifactory
`NPMRC`	Customize `.npmrc` with arbitrary config

Examples

Set the npm authentication token for an Artillery Pro deployment in the US East (N. Virginia) region:

artillery set-config-value \
    --name NPM_TOKEN \
    --value 01234567 \
    --region us-east-1

Set the loglevel setting for the .npmrc configuration to debug for the workers in an Artillery Pro deployment in the Europe (Ireland) region:

artillery set-config-value \
    --name NPMRC \
    --value "loglevel=debug" \
    --region eu-west-1

subscription

artillery subscription [options]

The subscription command shows the status of your team’s Artillery Pro subscription. You can also use this command to request an Artillery Pro license key to be sent to an email address associated with the subscription.

Options

Option	Description
`--email`	Request license key to be sent to an email address