Extension APIs


What you’ll learn

  • Different ways of extending and customizing Artillery
  • How to track custom metrics from your tests
  • How to customize VU behavior with custom JS code
  • How Artillery plugins work

Artillery is designed to be extensible and hackable to let users customize it if needed. Artillery is powered by Node.js, allowing you to leverage thousands of high-quality npm modules to do pretty much anything you may need to do.

Extending VU behavior

Virtual user behavior can be customized via custom functions written in JS and attached to scenarios via extension points also known as “hooks”. Different engines provide different extension points. Please refer to documentation of individual engines for details on what hooks they support, and what the APIs look like:

Tracking custom metrics

You can track custom metrics through JS code. The following metric types are supported:

  1. counters - A counter to keep track of certain events, like the number of times a VU completed a specific transaction.
  2. histograms - A distribution (max, min, median, p95, p99) of some measurements, like API response times or content length.

Custom metrics will get included in Artillery’s output alongside standard metrics.

Custom metrics API

Custom metrics are sent through an EventEmitter object, available as an argument in hook functions and as a constructor parameter for plugins and engines.

  • events.emit('counter', 'my_counter', 1) - Increments the value of a counter called my_counter by 1.
  • events.emit('histogram, 'my_histogram', 42) - Records a measurement of 42 in a histogram called my_histogram.

While you can use any string in the custom metrics API, a standard convention is to use snake_case for naming your metrics. When sending metrics to external monitoring systems (such as when using the artillery-publish-metrics plugin), some of these services will automatically convert spaces in metric names to underscores. If your metric name uses spaces, it may make your metrics difficult to search and categorize on these services.

Example

Let’s say we have an HTTP API with a route that creates a new pet object. We want to count how many pets we’ve created during the load test. Pets are also computationally expensive to create on the server, so we’d like to track the response time for the requests through the Server-Timing response header.

The test script contains the following:

config:
target: "https://pet-generator.test"
processor: "./metrics.js"
phases:
- arrivalRate: 25
duration: 300

scenarios:
- afterResponse: "trackPets"
flow:
- post:
url: "/pets"
json:
species: "pony"
name: "Tiki"

The test script contains one scenario with an afterResponse hook. The hook executes the trackPets function located inside the metrics.js file:

module.exports = { trackPets };

function trackPets(req, res, context, events, done) {
// After every response, increment the 'Pets created' counter by 1.
events.emit("counter", "pets_created", 1);

// Parse the 'server-timing' header and look for the 'pets' metric,
// and add it to 'Pet creation latency' histogram.
const latency = parseServerTimingLatency(res.headers["server-timing"], "pets");
events.emit("histogram", "pet_created_latency", latency);

return done();
}

function parseServerTimingLatency(header, timingMetricName) {
const serverTimings = header.split(",");

for (let timing of serverTimings) {
const timingDetails = timing.split(";");
if (timingDetails[0] === timingMetricName) {
return parseFloat(timingDetails[1].split("=")[1]);
}
}
}

The trackPets function collects two metrics after every response returned to a virtual user:

  • A counter called pets_created, incremented by 1.
  • A histogram called pet_created_latency, taken from the server-timing header.

When running the test script, Artillery will collect the custom metrics and display the metrics as part of the test results:

All virtual users finished
Summary report @ 17:04:07(+0900) 2021-05-19
Scenarios launched: 7500
Scenarios completed: 7500
Requests completed: 7500
Mean response/sec: 24.96
Response time (msec):
min: 117
max: 532
median: 264
p95: 489
p99: 501
Scenario counts:
0: 7500 (100%)
Codes:
200: 7500
pet_created_latency:
min: 100.1
max: 500.6
median: 252.1
p95: 477.2
p99: 497
pets_created: 7500

Plugins

Plugins provide a way to package up some functionality in an easy-to-use interface. Plugins in Artillery have full access to the test script and Artillery’s internals, and it’s possible to get very creative with what they can do.

Naming conventions

Artillery plugins are distributed as npm packages and follow the artillery-plugin-$NAME convention, e.g. artillery-plugin-publish-metrics.

Plugin lifecycle

  1. Plugins are enabled with the config.plugins section of a test script, which makes Artillery try to load the plugin:
config:
target: "http://svc-foo.prod.acme-corp.internal"
plugins:
my-custom-plugin:
option1: some value
option2: some other value

This example will make Artillery try to find and load the artillery-plugin-my-custom-plugin package.

  1. The plugin package is expected to export a Plugin class. The constructor receives two arguments:

    1. script - The full test script definition, which your plugin code can modify. For example, a plugin may attach additional hook functions to scenario definitions, like the metrics-by-endpoint plugin.
    2. events - An EventEmitter which receives events from Artillery and can also send custom metrics, like the publish-metrics plugin.
  2. A plugin may define a cleanup function, which takes a done callback. Artillery will call this function before exiting when a test finishes, providing an opportunity for any cleanup that a plugin needs to do. For example, you can use the function to flush buffered metrics to an external monitoring system.

Plugin events

Plugins can subscribe to the following events:

  1. phaseStarted - A new arrival phase has started.
  2. phaseCompleted - An arrival phase has finished.
  3. stats - Provides metrics for the previous reporting period.
  4. done - All VUs finished running their scenarios.

Example

The artillery-plugin-hello-world package contains an example plugin to learn about Artillery’s plugin interface and can serve as a starting point to create your custom plugins.

Engines

An engine packages up a way to communicate with a particular type of server in Artillery, like HTTP, Socket.IO, or WebSockets.

Engines may work on levels 4 or 7 of the OSI model or a higher-level protocol level like an SQL engine or a AWS Kinesis engine.

If you’re looking to write a custom engine for Artillery, the AWS Kinesis engine is a good starting point.

Reporters

Custom reporters to send metrics and reports to external systems can get implemented as a plugin.

Please see the official publish-metrics plugin for a complete example of a custom reporter.