When performing a load test using Artillery, you can set a few conditions in your test script to ensure that the aggregate results meet different requirements using the
ensure configuration setting. For instance, you can set a condition to verify that the aggregate median latency of your service is under 200 milliseconds, or that less than your scenarios have less than a 1% error rate. These checks are helpful in CI/CD environments since they can help confirm your service’s performance before deploying new updates or validate if your production services are meeting some of your SLOs.
However, you can use your existing Artillery test scripts to go beyond load testing and checking aggregate results of your HTTP services. With the
artillery-plugin-expect plugin, you can easily extend Artillery’s built-in functionality to include functional testing. This plugin adds support for setting expectations and assertions using your current test scenarios, giving you a quick and straightforward solution to check the functionality of your services using your existing testing toolkit.
Let’s take the following Artillery load testing script for an HTTP service as an example. The test script sends a consistent number of virtual users (50 VUs per second for 10 minutes), going through a single scenario. The scenario has three operations: the virtual user will log in to set a cookie with their authentication details, access a secure endpoint, and log out. The script also has a few checks to verify some aggregate results.
As part of the load test, we want to ensure that 99% of the scenarios should complete in under 300 milliseconds, with fewer than a 1% error rate. If the
p99 latency is over 300 milliseconds, or more than 1% of the virtual users fail to complete their scenario entirely, the test script returns a non-zero exit code.
While this test is beneficial to confirm our service works well under load, we’d also like to validate that the HTTP service’s endpoints are working as expected. For instance, we want to check that the endpoints return the correct status code, content type, and properties. However, thanks to the
artillery-plugin-expect plugin, you don’t need a separate testing tool to add this support. Let’s see how to set up this plugin and begin setting up these expectations in the same test script.
Artillery plugins are created and distributed as standard npm packages and can be installed using the npm or Yarn package managers. Installing the latest version of the
artillery-plugin-expect plugin depends on how you’ve set up Artillery in your test environment.
If Artillery is installed globally, you’ll need to install the plugin globally:
# Using npm.
If you’re using Artillery as a local development dependency of a project (set up in a project’s
package.json file), the plugin must be installed as a development dependency:
# Using npm.
Once the plugin is installed in the test environment, we’ll need to enable the plugin in the Artillery test script. Enabling plugins in Artillery is done with the
config.plugins setting in the test script. In the case of
artillery-plugin-expect, the configuration is the following:
When configuring a plugin, Artillery will attempt to load an npm package using the naming convention of
<plugin-name> is the name used under
config.plugins in the test script. Along with specifying the plugin’s name, we’ll also need to supply any configuration required for the plugin to work. In the case of
artillery-plugin-expect, we don’t need to provide any additional configuration, so we’ll set an empty map.
To verify that the plugin is set up correctly, we can run the test script as we usually do. The load test will work the same as before, without any warnings messages if Artillery found the plugin. If the plugin is not set up correctly, the test will still execute, but we’ll see the following warning message appear at the start of the test run:
WARNING: Plugin expect specified but module artillery-plugin-expect could not be found (MODULE_NOT_FOUND)
Usually, this warning shows up because Artillery and the plugin are installed differently — for instance, Artillery is installed globally while the plugin is installed as a local dependency. If the test run shows any plugin warnings, double-check where both Artillery and
artillery-plugin-expect are installed, along with the configuration in the test script.
Once we’ve confirmed the plugin is working, we can begin adding expectations and assertions to our test script. To start, we’ll define the following expectations that we want to validate for our functional tests:
POST /login- check that the service responds with a
200 OKstatus code.
GET /account- check that the service responds with a
200 OKstatus code, returns a JSON
Content-Typeheader, and the body has a property named
DELETE /logout- check that the service responds with a
204 No Contentstatus code.
Let’s add these expectations and assertions to the scenario operations in the Artillery test script:
Each request in the test scenarios we have now defines what we want to assert for each one. The plugin processes the
expect keyword during the test run, checking each of the expectations set below, like
hasProperty. These expectations and assertions cover what we wanted to check for our functional test.
When we run the test script using Artillery, the
artillery-plugin-expect plugin will now print out a report on the state of the assertions for each request it makes:
* POST /login
With just a few additions to an existing Artillery test script, you can include functional testing alongside your load tests. It’s a simple yet powerful way to test an HTTP service without introducing new tools or libraries into your projects.
With the existing test script, we’re also validating the assertions for every virtual user. This approach for functional testing is not the best one to take since there’s a good chance an HTTP service will have occasional errors while under heavy load. Ideally, we would run through our scenarios only once for functional tests.
Thankfully, we don’t have to create two separate test scripts to split up load and functional tests. Artillery allows us to set different configurations for distinct purposes using
config.environments. This setting can let us reuse our load testing script while changing some configuration under different contexts. In our case, we want to have two environments:
- A load testing environment, where we’ll send a large number of virtual users to the HTTP service for an extended amount of time and verify a few aggregate results after the test run.
- A functional testing environment, where we’ll run our scenario with a single virtual user to validate expectations for each request once.
The main difference between each environment is the load phase and what we want to verify during or after a test run. We can use the
config.environments setting to set these up separately, which we can then use to run the test script according to the testing type:
Here, we’re defining two environments called
functional. Each environment has different configuration settings suitable for the type of test we want to run under each. The rest of the configuration in the test script remains the same since it’ll target the same HTTP service and run the same scenarios. Although we won’t use the
artillery-plugin-expect plugin during load testing, we can still keep it in the test script — it simply won’t run the expectations or assertions in that environment.
To run a specific environment, we’ll use the
--environment flag in the command line when executing our tests, specifying the environment name we used in the test script:
# Runs load tests.
When using the
--environment load flag, Artillery will run the load test, sending 50 virtual users per second for 10 minutes and check the aggregate results at the end of the test run. With the
--environment functional flag, Artillery will load the
artillery-plugin-expect plugin and run through the scenario with a single virtual user, going through the expectations and verifying its assertions for each request only once.
Specifying different environments in an Artillery test script using the
config.environments setting helps avoid duplication for your testing. It also allows us to reuse a single test script to handle various kinds of testing from one place.
artillery-plugin-expect plugin has a few methods to help you debug your functional tests if they’re not passing. When a functional test fails, the plugin will print out additional details about the expected and actual results for the expectations, like the request parameters, the response headers, and the body.
For example, if our
POST /login request fails with a
500 Internal Server Error during a functional test, the plugin will print out the following details:
* POST /login
To get additional details about what the
artillery-plugin-expect plugin is doing for a request, we can set the
DEBUG=plugin:expect environment variable when executing the functional test:
DEBUG=plugin:expect artillery run --environment functional api-test.yml
Setting this environment variable will print out extra debugging information before each request to help you see the different types of checks that occur with the plugin. For example, the plugin’s debugging information for the
GET /account request in our functional test prints out the following output:
plugin:expect Checking expectations +1s
Although the plugin automatically prints out more details about the actual request and response with a failed assertion, using the
DEBUG=plugin:expect environment variable is an additional way to help verify that we’re setting the correct expectation types and values in your test script.
Artillery 2.0 — currently a development release — includes the
--solo flag that we can use to execute our tests with a single VU, regardless of the phase definition in the test script. This flag is ideal for functional testing since we wouldn’t have to define two distinct phases for each environment. Using this flag, we can simplify the test script configuration for each environment:
These configuration changes give us some extra flexibility for running our Artillery tests for different purposes:
# Install the development release of Artillery.
In this article, we covered a handful of the expectations provided by the plugin, like verifying a request’s status code or part of the response body. The plugin has additional expectations you can use to validate response headers, compare a captured value from a previous response, and more. You can learn more on the plugin’s documentation page.
Thanks to the
artillery-plugin-expect plugin, you can use Artillery to easily cover both your performance and functional testing needs for your HTTP services with the same toolkit. You can reuse the same scenarios for both types of testing to help your applications meet SLOs and are functioning according to specifications.
You can try a live example of the
artillery-plugin-expect plugin to see it in action using Artillery’s SuperREPL tool.