Core Concepts

What you’ll learn

  • Key high-level concepts for using Artillery
  • Configuration and scenario sections in Artillery test scripts
  • What load phases and virtual users are
  • What an Artillery test definition looks like and how to run it

High-level view

Artillery runs performance tests against backend services. You write test definitions, and Artillery runs them for you.

A test definition is written as a YAML file (with the option for near-infinite customization with Node.js code, which may use any public or private npm packages). The artillery CLI is then used to run that test definition.

Test definitions

A test definition has two main parts: configuration and one or more scenarios.

  1. Configuration is where you define the endpoint(s) on which the system under test can be accessed, define load phases for the test run, load data from external CSV files, configure any plugins etc.
  2. Scenarios describe the actions which a virtual user created by Artillery will take. For example, if you’re testing a RESTful ecommerce backend, a scenario may be composed of:
    1. making a request to the search API endpoint to search for a product
    2. making a request to add one of the products returned in the results to basket
    3. loading the basket
    4. initiating the checkout flow

Load phases and virtual users

These two concepts are fundamental to how Artillery works, and for being able to use Artillery to its full potential.

A load phase tells Artillery how many virtual users to create over a period of time. Load phases is how you model load on a service for the duration of a test run. A load test will usually have multiple load phases, such as a warm up phase, followed by a gentle ramp up phase, which is then followed by one or more heavy load phases. Let’s look at an example:

phases:
  - duration: 300
    arrivalRate: 10
    name: Warm up phase

This code describes a load phase which lasts 300 seconds (or 5 minutes), where 10 new virtual users will be created every second.

A “virtual user” is exactly what the name suggests - a simulation of a real user (or a client) using the service. Each virtual user is completely independent of other virtual users - just like in the real world. For example, when testing an HTTP-based service, each virtual user will open & maintain its own TCP connection(s), will have its own set of cookies if sent by the server etc.

Each virtual user will pick and run one of the scenarios in the test definition and run it to completion. A test run ends when all virtual users finish running their scenarios. This is an important distinction to understand: the duration of a test is not the combined duration of all of its load phases, but the amount of time it takes for all virtual users created in those load phases to finish their scenarios.

Example - How long will a test run for?

Let’s look at an example. Imagine we define a test with a single load phase which lasts five seconds and creates one virtual user every second (so a total of fives virtual users will be created). The test definition contains a single scenario, which sends a few HTTP requests one after another.

How long will that test take to run?

The answer is “it depends”, with a couple of factors that need to be considered:

  1. How quickly the target service can respond to each request, which will vary for different kinds of requests (e.g. a GET request that returns a cached value vs a POST that creates a new record in a database)
  2. How quickly the service will actually respond to requests. The response time from the service will likely change as more load is placed on the it, i.e. it will probably get slower at some point
  3. Network latency and performance which can add some overhead to how long a request takes to be completed, before the virtual user moves on to the next one
  4. The number of steps in the scenario

This can be visualized with the following (oversimplified) diagram:

Artillery vusers

The horizontal axis is time, starting at 0. Each blue line represents a single virtual user running their scenario from start to finish. Five users are created in the first five seconds of the test.

Even though each scenario definition is the same, the amount of time different virtual users will take to complete their scenario will vary, due to the factors described above, with the total running time of the test in this made-up imaginary example of 30+ seconds.

Putting it all together

An Artillery test definition for testing an ecommerce backend could look like this:

config:
  target: "https://shopping.service.staging"
  phases:
    - duration: 60
      arrivalRate: 5
      name: Warm up
    - duration: 120
      arrivalRate: 5
      rampTo: 50
      name: Ramp up load
    - duration: 600
      arrivalRate: 50
      name: Sustained load
  payload:
    # Load search keywords from an external CSV file and make them available
    # to virtual user scenarios as variable "keywords":
    path: "keywords.csv"
    fields:
      - "keywords"
scenarios:
  # We define one scenario:
  - name: "Search and buy"
    flow:
      - post:
          url: "/search"
          body: "kw={{ keywords }}"
          # The endpoint responds with JSON, which we parse and extract a field from
          # to use in the next request:
          capture:
            json: "$.results[0].id"
            as: "id"
      # Get the details of the product:
      - get:
          url: "/product/{{ id }}/details"
      # Pause for 3 seconds:
      - think: 3
      # Add product to cart:
      - post:
          url: "/cart"
          json:
            productId: "{{ id }}"

If you saved the test definition in a file called search-and-add-to-cart.yml, you could then run it with:

artillery run search-and-add-to-cart.yml