expect - Expectations & Assertions

Overview

The expect plugin adds support for checks and assertions on HTTP requests to enable functional testing in Artillery tests.

Features

• Use expectations and assertions in your HTTP scenarios
• Use the same artillery command to run functional / acceptance tests on your APIs
• See details of any failed assertions (request headers & body, response etc)
• Run post-deployment smoke tests from CI/CD pipelines

Usage

Enable the plugin

To enable the plugin, add it to the config section of a test script.

config:
  target: 'http://lab.artillery.io'
  plugins:
    expect: {}

Add expectations to HTTP requests

scenarios:
  - name: Get a movie
    flow:
      - get:
          url: '/movies'
          capture:
            - json: '$[5].title'
              as: title
          expect:
            - statusCode: 200
            - contentType: json
            - equals:
                - 'From Dusk Till Dawn'
                - '{{ title }}'

Try it live! (opens in a new tab)

Run the test

Run your script that uses expectations with:

artillery run --quiet my-script.yaml

(The --quiet option is to stop Artillery from printing its default reports to the console.)

Run load + functional tests

This plugin allows for the same scenario to be re-used for either load testing or functional testing of an API, with the help of config.environments definition in your test script.

(The only real difference between the two, of course, is how many virtual users you run -- only one for functional tests, and lots of virtual users for a load test.)

config:
  target: 'https://my.api.internal'
  environments:
    #
    # This is our load testing profile, where we create a lot of virtual users.
    # Note that we don't load the plugin here, so that we don't see the output
    # from the plugin.
    #
    load:
      phases:
        - duration: 600
          arrivalRate: 10
    #
    # This is our functional testing profile, with a single virtual user, and
    # the plugin enabled.
    #
    functional:
      phases:
        - duration: 1
          arrivalCount: 1
      plugins:
        expect: {}
scenarios:
  # Your scenario definitions go here.

You could run the script above in load testing mode with:

artillery run --environment load script.yaml

and in functional testing mode with:

artillery run --environment functional script.yaml

Configuration

You can set your plugin configuration in your config. For example:

config:
  target: "http://lab.artillery.io"
  plugins:
    expect:
      reportFailuresAsErrors: true
      outputFormat: prettyError

Here are the configuration options available to you:

Metrics

The plugin tracks successful and failed expectations with custom metrics (opens in a new tab).

• expect.ok - count of all expectations that have been successful
• expect.failed - count of all expectations that failed
• expect.ok.{type} - count of successful expectations of {type}, for example: expect.ok.statusCode for statusCode expectations
• expect.failed.{type} - count of failed expectations of {type}, for example: expect.failed.statusCode for statusCode expectations

Expectations

statusCode

Check that the status code of the response.

expect:
  - statusCode: 201

The value may also be a list to indicate that any of the codes in the list is OK:

expect:
  - statusCode:
      - 200
      - 301

notStatusCode

This check succeeds if the status code is anything but the list of codes specified.

For example, this expectation succeeds when the response status code is anything but 403 or 500:

expect:
  - notStatusCode:
      - 403
      - 500

contentType

Check the value of Content-Type (opens in a new tab) header.

hasProperty and notHasProperty

When the response is JSON, check that the response object has a property (or does not in case of notHasProperty). Same behavior as lodash#has (opens in a new tab).

expect:
  - hasProperty: 'data[0].id'

equals

Check that two or more values are the same. NOTE only primitive values (e.g. booleans, strings and numbers) are currently supported.

- get:
    url: '/pets/f037ed9a'
    capture:
      - json: '$.species'
        as: species
    expect:
      - equals:
          - '{{ species }}'
          - 'dog'

hasHeader

Check that the response contains a header.

- get:
    url: '/pets/f037ed9a'
    expect:
      - hasHeader: 'object-version'

headerEquals

Check that the response contains a header and its value matches some string.

- get:
    url: '/pets/f037ed9a'
    expect:
      - headerEquals:
          - 'object-version'
          - 'v3'

Multiple headers with the same name can also be matched. For example, if we expect a request to return two Set-Cookie headers with some known values we can check that they match like this:

- get:
    url: "/protected/url"
    expect:
      - headerEquals:
        # Expect two Set-Cookie headers returned by this request,
        # in a particular order:
        - set-cookie
        -
          - platform=astmh;version=1
          - status=draft;version=1

matchesRegexp

Check that response body matches a regular expression. The regular expression provided must be a string which is a valid argument to the RegExp constructor (opens in a new tab).

- get:
    url: '/pets/f037ed9a'
    expect:
      - matchesRegexp: .+ # body is not empty

cdnHit

This expectation checks for presence of a cache hit/miss header from a CDN. The following CDNs are supported:

• Cloudflare
• CloudFront
• Fastly
• Vercel

The check will succeed if:

• A CDN cache hit/miss header is present in the response (one of cf-cache-status, x-cache, x-vercel-cache)
• The value of the header is either hit or stale

- get:
    url: '/pets/f037ed9a'
    expect:
      cdnHit: true

jmespath

Evaluate JMESPath expressions as an expectation. If the expression evaluates to a truthy value, the expectation will pass. For exaample:

- get:
    url: '/foo/1234'
    expect:
      jmespath: 'title != null'

Metrics reported by the plugin

In addition to the default metrics reported by Artillery and your chosen engine(s), this plugin reports the following additional metrics:

Debugging

If you're having issues with the plugin, you can print out helpful debugging messages using the DEBUG environment variable.

Print expectations and values

Set DEBUG=plugin:expect when running your tests to view when using an expectation and any values to check.

DEBUG=plugin:expect artillery run my-script.yaml