Artillery
8KSign in
How to

Using Artillery for Your Functional Testing

Hassy VeldstraHassy Veldstra

When performing a load test using Artillery, you can set a few conditions in your test script to ensure that the aggregate results meet different requirements using the ensure configuration setting. For instance, you can set a condition to verify that the aggregate median latency of your service is under 200 milliseconds, or that less than your scenarios have less than a 1% error rate. These checks are helpful in CI/CD environments since they can help confirm your service’s performance before deploying new updates or validate if your production services are meeting some of your SLOs.

However, you can use your existing Artillery test scripts to go beyond load testing and checking aggregate results of your HTTP services. With the artillery-plugin-expect plugin, you can easily extend Artillery’s built-in functionality to include functional testing. This plugin adds support for setting expectations and assertions using your current test scenarios, giving you a quick and straightforward solution to check the functionality of your services using your existing testing toolkit.

An example Artillery load test

Let’s take the following Artillery load testing script for an HTTP service as an example. The test script sends a consistent number of virtual users (50 VUs per second for 10 minutes), going through a single scenario. The scenario has three operations: the virtual user will log in to set a cookie with their authentication details, access a secure endpoint, and log out. The script also has a few checks to verify some aggregate results.

config:
  target: 'http://lab.artillery.io'
  phases:
    - duration: 600
      arrivalRate: 50
  ensure:
    p99: 300
    maxErrorRate: 1

scenarios:
  - flow:
      - post:
          url: '/login'
          json:
            username: 'testuser'
            password: 'testpassword'
      - get:
          url: '/account'
      - delete:
          url: '/logout'

As part of the load test, we want to ensure that 99% of the scenarios should complete in under 300 milliseconds, with fewer than a 1% error rate. If the p99 latency is over 300 milliseconds, or more than 1% of the virtual users fail to complete their scenario entirely, the test script returns a non-zero exit code.

While this test is beneficial to confirm our service works well under load, we’d also like to validate that the HTTP service’s endpoints are working as expected. For instance, we want to check that the endpoints return the correct status code, content type, and properties. However, thanks to the artillery-plugin-expect plugin, you don’t need a separate testing tool to add this support. Let’s see how to set up this plugin and begin setting up these expectations in the same test script.

Using the artillery-plugin-expect plugin

Artillery plugins are created and distributed as standard npm packages and can be installed using the npm or Yarn package managers. Installing the latest version of the artillery-plugin-expect plugin depends on how you’ve set up Artillery in your test environment.

If Artillery is installed globally, you’ll need to install the plugin globally:

# Using npm.
npm install artillery-plugin-expect@latest --global

# Using Yarn (assuming Artillery is installed in /usr/local).
yarn global add artillery-plugin-expect@latest --prefix /usr/local

If you’re using Artillery as a local development dependency of a project (set up in a project’s package.json file), the plugin must be installed as a development dependency:

# Using npm.
npm install artillery-plugin-expect@latest --save-dev

# Using Yarn.
yarn add artillery-plugin-expect@latest --dev

Once the plugin is installed in the test environment, we’ll need to enable the plugin in the Artillery test script. Enabling plugins in Artillery is done with the config.plugins setting in the test script. In the case of artillery-plugin-expect, the configuration is the following:

config:
  target: 'http://lab.artillery.io'
  phases:
    - duration: 600
      arrivalRate: 50
  ensure:
    p99: 300
    maxErrorRate: 1
  plugins:
    expect: {}

When configuring a plugin, Artillery will attempt to load an npm package using the naming convention of artillery-plugin-<plugin-name>, where <plugin-name> is the name used under config.plugins in the test script. Along with specifying the plugin’s name, we’ll also need to supply any configuration required for the plugin to work. In the case of artillery-plugin-expect, we don’t need to provide any additional configuration, so we’ll set an empty map.

To verify that the plugin is set up correctly, we can run the test script as we usually do. The load test will work the same as before, without any warnings messages if Artillery found the plugin. If the plugin is not set up correctly, the test will still execute, but we’ll see the following warning message appear at the start of the test run:

WARNING: Plugin expect specified but module artillery-plugin-expect could not be found (MODULE_NOT_FOUND)

Usually, this warning shows up because Artillery and the plugin are installed differently — for instance, Artillery is installed globally while the plugin is installed as a local dependency. If the test run shows any plugin warnings, double-check where both Artillery and artillery-plugin-expect are installed, along with the configuration in the test script.

Once we’ve confirmed the plugin is working, we can begin adding expectations and assertions to our test script. To start, we’ll define the following expectations that we want to validate for our functional tests:

  • POST /login - check that the service responds with a 200 OK status code.
  • GET /account - check that the service responds with a 200 OK status code, returns a JSON Content-Type header, and the body has a property named user.
  • DELETE /logout - check that the service responds with a 204 No Content status code.

Let’s add these expectations and assertions to the scenario operations in the Artillery test script:

config:
  target: 'http://lab.artillery.io'
  phases:
    - duration: 600
      arrivalRate: 50
  ensure:
    p99: 300
    maxErrorRate: 1
  plugins:
    expect: {}

scenarios:
  - flow:
      - post:
          url: '/login'
          json:
            username: 'testuser'
            password: 'testpassword'
          expect:
            - statusCode: 200
      - get:
          url: '/account'
          expect:
            - statusCode: 200
            - contentType: 'json'
            - hasProperty: 'user'
      - delete:
          url: '/logout'
          expect:
            - statusCode: 204

Each request in the test scenarios we have now defines what we want to assert for each one. The plugin processes the expect keyword during the test run, checking each of the expectations set below, like statusCode, contentType, and hasProperty. These expectations and assertions cover what we wanted to check for our functional test.

When we run the test script using Artillery, the artillery-plugin-expect plugin will now print out a report on the state of the assertions for each request it makes:

* POST /login
  ok statusCode 200
* GET /account
  ok statusCode 200
  ok contentType json
  ok hasProperty user
* DELETE /logout
  ok statusCode 204

With just a few additions to an existing Artillery test script, you can include functional testing alongside your load tests. It’s a simple yet powerful way to test an HTTP service without introducing new tools or libraries into your projects.

Separating load tests and functional tests in the same test script

With the existing test script, we’re also validating the assertions for every virtual user. This approach for functional testing is not the best one to take since there’s a good chance an HTTP service will have occasional errors while under heavy load. Ideally, we would run through our scenarios only once for functional tests.

Thankfully, we don’t have to create two separate test scripts to split up load and functional tests. Artillery allows us to set different configurations for distinct purposes using config.environments. This setting can let us reuse our load testing script while changing some configuration under different contexts. In our case, we want to have two environments:

  • A load testing environment, where we’ll send a large number of virtual users to the HTTP service for an extended amount of time and verify a few aggregate results after the test run.
  • A functional testing environment, where we’ll run our scenario with a single virtual user to validate expectations for each request once.

The main difference between each environment is the load phase and what we want to verify during or after a test run. We can use the config.environments setting to set these up separately, which we can then use to run the test script according to the testing type:

config:
  target: 'http://lab.artillery.io'
  environments:
    load:
      phases:
        - duration: 600
          arrivalRate: 50
      ensure:
        p99: 300
        maxErrorRate: 1
    functional:
      phases:
        - duration: 1
          arrivalCount: 1
      plugins:
        expect: {}

scenarios:
  - flow:
      - post:
          url: '/login'
          json:
            username: 'testuser'
            password: 'testpassword'
          expect:
            - statusCode: 200
      - get:
          url: '/account'
          expect:
            - statusCode: 200
            - contentType: 'json'
            - hasProperty: 'user'
      - delete:
          url: '/logout'
          expect:
            - statusCode: 204

Here, we’re defining two environments called load and functional. Each environment has different configuration settings suitable for the type of test we want to run under each. The rest of the configuration in the test script remains the same since it’ll target the same HTTP service and run the same scenarios. Although we won’t use the artillery-plugin-expect plugin during load testing, we can still keep it in the test script — it simply won’t run the expectations or assertions in that environment.

To run a specific environment, we’ll use the --environment flag in the command line when executing our tests, specifying the environment name we used in the test script:

# Runs load tests.
artillery run --environment load api-test.yml

# Runs functional tests.
artillery run --environment functional api-test.yml

When using the --environment load flag, Artillery will run the load test, sending 50 virtual users per second for 10 minutes and check the aggregate results at the end of the test run. With the --environment functional flag, Artillery will load the artillery-plugin-expect plugin and run through the scenario with a single virtual user, going through the expectations and verifying its assertions for each request only once.

Specifying different environments in an Artillery test script using the config.environments setting helps avoid duplication for your testing. It also allows us to reuse a single test script to handle various kinds of testing from one place.

Debugging expectations during functional testing

The artillery-plugin-expect plugin has a few methods to help you debug your functional tests if they’re not passing. When a functional test fails, the plugin will print out additional details about the expected and actual results for the expectations, like the request parameters, the response headers, and the body.

For example, if our POST /login request fails with a 500 Internal Server Error during a functional test, the plugin will print out the following details:

* POST /login
  not ok statusCode 500
  expected: 200
       got: 500
  Request params:
    http://lab.artillery.io/login
    {
      "username": "testuser",
      "password": "testpassword"
    }
  Headers:
   content-type : application/json; charset=utf-8
   content-length : 64
   etag : W/"27-HSrEpTCl9tImXS685QUvpoLGKhI"
   date : Mon, 02 Aug 2021 01:01:06 GMT
   connection : keep-alive
   keep-alive : timeout=5
  User variables:
     target : http://lab.artillery.io
     $environment : functional
     $uuid : 52f0cab7-32c8-4c24-a906-4d3703a37763

To get additional details about what the artillery-plugin-expect plugin is doing for a request, we can set the DEBUG=plugin:expect environment variable when executing the functional test:

DEBUG=plugin:expect artillery run --environment functional api-test.yml

Setting this environment variable will print out extra debugging information before each request to help you see the different types of checks that occur with the plugin. For example, the plugin’s debugging information for the GET /account request in our functional test prints out the following output:

plugin:expect Checking expectations +1s
plugin:expect checker: statusCode +0ms
plugin:expect check statusCode +0ms
plugin:expect checker: contentType +0ms
plugin:expect check contentType +0ms
plugin:expect expectation: { contentType: 'json' } +0ms
plugin:expect body: object +1ms
plugin:expect checker: hasProperty +1ms
plugin:expect check hasProperty +0ms

Although the plugin automatically prints out more details about the actual request and response with a failed assertion, using the DEBUG=plugin:expect environment variable is an additional way to help verify that we’re setting the correct expectation types and values in your test script.

Upcoming functional testing improvements for Artillery

Artillery 2.0 — currently a development release — includes the --solo flag that we can use to execute our tests with a single VU, regardless of the phase definition in the test script. This flag is ideal for functional testing since we wouldn’t have to define two distinct phases for each environment. Using this flag, we can simplify the test script configuration for each environment:

config:
  target: 'http://lab.artillery.io'
  phases:
    - duration: 600
      arrivalRate: 50
  environments:
    load:
      ensure:
        p99: 300
        maxErrorRate: 1
    functional:
      plugins:
        expect: {}

These configuration changes give us some extra flexibility for running our Artillery tests for different purposes:

# Install the development release of Artillery.
npm install -g artillery@dev

# Runs load tests without verifying aggregate results.
artillery run api-test.yml

# Runs load tests and verifies aggregate results after the test run.
artillery run --environment load api-test.yml

# Runs functional tests with a single VU.
artillery run --environment functional --solo api-test.yml

Help meet SLOs and validate your service’s functionality with a single Artillery test script

In this article, we covered a handful of the expectations provided by the plugin, like verifying a request’s status code or part of the response body. The plugin has additional expectations you can use to validate response headers, compare a captured value from a previous response, and more. You can learn more on the plugin’s documentation page.

Thanks to the artillery-plugin-expect plugin, you can use Artillery to easily cover both your performance and functional testing needs for your HTTP services with the same toolkit. You can reuse the same scenarios for both types of testing to help your applications meet SLOs and are functioning according to specifications.

You can try a live example of the artillery-plugin-expect plugin to see it in action using Artillery’s SuperREPL tool.