July 22nd, 2021How to

Extend Artillery by Creating Your Own Plugins

Dennis Martinez

We’ve designed Artillery to be hackable and easy to extend. Plugins hook into Artillery’s internal APIs to allow you to include additional functionality like generating different kinds of data for your tests, integrating with third-party systems, or sending metrics to another platform.

Since Artillery uses Node.js, you can use tens of thousands of Node.js packages available on npm to take advantage of existing libraries you can integrate right away. You can find npm packages for almost anything, making it much easier to create and extend Artillery without much hassle.

Let’s see how easy it is to create a new plugin for Artillery. Imagine that we want to send our test run metrics to an AWS Lambda function to store in a DynamoDB database or send an event to an SNS topic. In this article, we’ll go through the process of creating a new plugin that extends Artillery’s functionality to do that. The plugin we’ll create will invoke a Lambda function after a test run using the AWS SDK for JavaScript library.

Setting up an NPM module

Plugins in Artillery are standard npm packages, and they can be either public or private. By default, Artillery will look in the npm package path for the plugin with artillery-plugin- prefixed to the name describing your plugin. In this example, we’ll name our plugin artillery-plugin-lambda since we will invoke Lambda functions in our code.

Creating the npm package is a straightforward process using the npm init command inside of a directory where you want to build your plugin. For our example, we’ll create a directory called artillery-plugin-lambda. When executing the npm init command without any arguments inside this directory, it will ask you a few questions for setting up the project, like the name of the package, a description, and other details. Most of the questions are optional, so you don’t have to answer all of them at this time. If you plan to publish the plugin later, it’s a good practice to add as much detail as possible to help other users find it.

> npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help init` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
package name: (artillery-plugin-lambda)
version: (1.0.0) 0.0.1
description: Send Artillery metrics to an AWS Lambda function
entry point: (index.js)
test command:
git repository:
keywords:
author: Dennis Martinez
license: (ISC) MIT
About to write to /tmp/artillery-plugin-lambda/package.json:

{
  "name": "artillery-plugin-lambda",
  "version": "0.0.1",
  "description": "Send Artillery metrics to an AWS Lambda function",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Dennis Martinez",
  "license": "MIT"
}

Is this OK? (yes)

npm init will generate a package.json file for you based on your answers. This file serves as a starting point for your plugin. Later in the article, this file will get updated to include our dependencies like the AWS SDK.

Building the skeleton of the plugin

The package for a plugin is expected to export a Plugin class so Artillery can load it properly. The constructor for the class is a function that receives two arguments:

script - The full test script definition. The plugin code can modify the test script to extend the current load test definition, like attaching hooks to existing scenarios.
events - An EventEmitter which you can use to subscribe to events sent from Artillery during a test run. You can also use the argument to send custom metrics.

In addition to subscribing to Artillery events, plugins can also define a cleanup function with a done callback. This function gets called before Artillery exits upon the completion of a test run. It provides an opportunity to perform any cleanup that your plugin needs before wrapping up. We’ll see how to use this function later in the article.

To start constructing our plugin, let’s create a new file called index.js in our plugin directory. This file will serve as the entry point for our npm package. Inside the file, we can set up the Plugin class and a function to define the constructor of the class:

'use strict';

module.exports.Plugin = ArtilleryLambdaPlugin;

function ArtilleryLambdaPlugin(script, events) {
  // Your plugin code goes here.
}

As mentioned at the beginning of this section, Artillery plugins need to export a Plugin class. Here, we’re using the ArtilleryLambdaPlugin function to load the plugin’s code. The ArtilleryLambdaPlugin function receives the script and events arguments, and from there, you can create the custom functionality for your plugin. To see how these work, we’ll first demonstrate how to use the events argument and later see how to use the script argument to access our test script inside the plugin.

The events argument allows us to subscribe to the following events:

phaseStarted - Emitted when a new load phase starts.
phaseCompleted - Emitted when a load phase completes.
stats - Emitted after each reporting period during a test run (every 10 seconds or at the end of a load phase).
done - Emitted when all VUs complete their scenarios, and there are no more load phases to run.

Given that events is a Node.js EventEmitter object, we can set up a listener function that gets called when Artillery emits the event using .on. For example, let’s set up a listener for each of the events emitted by an Artillery test run to see how plugins work by printing a few messages to the console:

'use strict';

module.exports.Plugin = ArtilleryLambdaPlugin;

function ArtilleryLambdaPlugin(script, events) {
  // Prints out "🧑 New load phase started".
  events.on('phaseStarted', function () {
    console.log('🧑 New load phase started');
  });

  // Prints out "✅ Load phase complete".
  events.on('phaseCompleted', function () {
    console.log('✅ Load phase complete');
  });

  // Prints out "🖥️ Reporting".
  events.on('stats', function () {
    console.log('🖥️ Reporting');
  });

  // Prints out "🪄 All VUs finished".
  events.on('done', function () {
    console.log('🪄 All VUs finished');
  });

  return this;
}

Let’s verify the plugin is working correctly by loading it into an Artillery test script and putting it into action.

Loading the plugin into an Artillery test script

For purposes of this example, let’s assume we have the following Artillery test script called api-test.yml, which has three load phases and a scenario that hits two endpoints for an HTTP service:

config:
  target: 'https://test-app.dev/api'
  phases:
    - duration: 60
      arrivalRate: 5
      name: 'Warm up'
    - duration: 120
      arrivalRate: 5
      rampTo: 50
      name: 'Ramp up load'
    - duration: 600
      arrivalRate: 50
      name: 'Sustained load'

scenarios:
  - flow:
      - get:
          url: '/products'
      - post:
          url: '/cart'
          json:
            productId: '123'

An existing test script can load an Artillery plugin by using the config.plugins setting. The setting will tell Artillery to load an npm package using the specified plugin name, prefixed with artillery-plugin- as mentioned at the beginning of this article. Our example plugin is named artillery-plugin-lambda, so we can configure our plugin using lambda:

config:
  target: 'https://test-app.dev/api'
  phases:
    - duration: 60
      arrivalRate: 5
      name: 'Warm up'
    - duration: 120
      arrivalRate: 5
      rampTo: 50
      name: 'Ramp up load'
    - duration: 600
      arrivalRate: 50
      name: 'Sustained load'
  plugins:
    lambda: {}

scenarios:
  - flow:
      - get:
          url: '/products'
      - post:
          url: '/cart'
          json:
            productId: '123'

Under the config.plugins setting, we’re loading our plugin using lambda. Since we’re not providing any configuration to use in the plugin at this time, we’re setting an empty mapping for it. Later on, we’ll show how to set up some configuration that the plugin code can access and use internally.

Since plugins are npm packages, Artillery will look for them as any other package. That means if you installed Artillery globally in your system (like using npm install -g artillery@latest, for instance), it would try to find the artillery-plugin-lambda plugin in your global npm packages for this example. If you installed Artillery in your project (without the -g flag), it would search for the plugin in your project’s installed dependencies.

If we attempt to run the example test script with artillery run api-test.yml, the load test will run. However, it will print the following warning message in the console: WARNING: Plugin lambda specified but module artillery-plugin-lambda could not be found (MODULE_NOT_FOUND). The plugin is not installed either locally or globally because we don’t have an npm package yet to set up during development.

We don’t have to create an npm package for our test, though. Artillery allows us to set up the path of our plugin by using the ARTILLERY_PLUGIN_PATH environment variable. It’s a convenient way to tell Artillery where to look for any defined plugins. For this example, let’s assume our plugin is in the /home/artillery/plugins/artillery-plugin-lambda directory. We can load the plugin when running the example test script with the following command:

ARTILLERY_PLUGIN_PATH="/home/artillery/plugins" \
  artillery run custom-metrics.yml

Artillery will look inside the /home/artillery/plugins directory for an npm package matching the name of the plugin it’s expecting. In our case, it’s expecting the artillery-plugin-lambda directory. It will find the directory and load the entry point, activating the plugin for the test run. If everything goes as expected, you will see the console messages set up in the plugin during the test run.

Allow users to configure the plugin

We’ve seen how to build the basics of an Artillery plugin, some of the events you can listen to, and how to load the plugin locally during development. Let’s begin setting up the main bits of functionality we want to extend in Artillery, which is invoking an AWS Lambda function when a test run completes.

Most Artillery plugins will have different settings to allow anyone using the plugin to add different configuration settings for their needs. For instance, you can set up access keys to third-party services that the plugin uses or define paths to generate various artifacts in the system running the test. For our example, we want to allow users of our plugin to let others set the name of the function they wish to invoke and the AWS region where the function exists. We’ll call these settings function and region.

For our example, we’ll have an AWS Lambda function called AggregateArtilleryResults located in the ap-northeast-1 region. We want our plugin’s configuration to look like this:

config:
  target: 'https://test-app.dev/api'
  phases:
    - duration: 60
      arrivalRate: 5
      name: 'Warm up'
    - duration: 120
      arrivalRate: 5
      rampTo: 50
      name: 'Ramp up load'
    - duration: 600
      arrivalRate: 50
      name: 'Sustained load'
  plugins:
    lambda:
      function: 'AggregateArtilleryResults'
      region: 'ap-northeast-1'

scenarios:
  - flow:
      - get:
          url: '/products'
      - post:
          url: '/cart'
          json:
            productId: '123'

We have to tell our plugin to read these values from the test script to use them later when setting up our invocation functionality. The Plugin class constructor receives a script argument, which contains the entirety of the test script as an object. We can access the configuration for our plugin through this argument.

In the index.js file for our plugin, let’s delete the example code we used before and set up our custom functionality. First, we’ll access the configuration from the test script and set it up in a variable for future use:

'use strict';

module.exports.Plugin = ArtilleryLambdaPlugin;

function ArtilleryLambdaPlugin(script, events) {
  const self = this;

  self.config = script.config.plugins['lambda'];

  return this;
}

In this code, we first set up a reference to this by assigning it to the self variable to manage the original context when handling events, as we’ll show later. Next, we’re reading our test script through the script argument. We want to access our plugin’s configuration, so we’ll access script.config.plugins['lambda'] to get the configuration settings and assign them to self.config. Based on our test script above, the value of self.config will be a JavaScript object containing our settings:

{
  "function": "AggregateArtilleryResults",
  "region": "ap-northeast-1"
}

Because we’ll need these settings to configure the AWS SDK and invoke the correct Lambda function, a good practice is to alert any users loading the plugin if either of these settings is missing. A quick way to do this is by throwing an exception with a helpful message telling users that some configuration is missing:

'use strict';

module.exports.Plugin = ArtilleryLambdaPlugin;

function ArtilleryLambdaPlugin(script, events) {
  const self = this;

  self.config = script.config.plugins['lambda'];

  if (!self.config.function) {
    throw new Error('function is missing');
  }

  if (!self.config.region) {
    throw new Error('region is missing');
  }

  return this;
}

With the added checks, if either the function or region setting is missing from the plugin configuration under lambda, the plugin will throw the exception and print out the respective message in the console at the beginning of the Artillery test run. Keep in mind that although your plugin won’t load after the error, the Artillery test script will continue running without using any of the plugin’s functionality.

Using the AWS SDK library after a test run

With the plugin configuration at hand, we can now set up the AWS SDK library to begin invoking the configured Lambda function.

As mentioned at the beginning of this article, you can use any Node.js package in your plugin. That means we can use the official AWS SDK for JavaScript created by Amazon. For this example, we’ll use version 3.x of the SDK. This version uses modularized packages to allow developers to load only the functionality they need. Since we only want to access AWS Lambda, we can set up the Lambda client of the SDK as a dependency using npm install:

npm install @aws-sdk/client-lambda

After installing the package, we can use it to set up the Lambda client in our plugin. We’ll need two functions from the SDK. The first function is LambdaClient, which initializes the client. The second function we need is the InvokeCommand to set up how we’ll invoke the function using the client. The SDK separates both client and commands, so we can import both functions and begin setting them up in the plugin:

'use strict';

const { LambdaClient, InvokeCommand } = require('@aws-sdk/client-lambda');

module.exports.Plugin = ArtilleryLambdaPlugin;

function ArtilleryLambdaPlugin(script, events) {
  const self = this;

  self.config = script.config.plugins['lambda'];

  if (!self.config.function) {
    throw new Error('function is missing');
  }

  if (!self.config.region) {
    throw new Error('region is missing');
  }

  self.lambdaClient = function () {
    return new LambdaClient({ region: self.config.region });
  };

  self.lambdaCommand = function (report) {
    return new InvokeCommand({
      FunctionName: self.config.function,
      Payload: new TextEncoder().encode(JSON.stringify(report)),
    });
  };

  return this;
}

The LambdaClient function is a class constructor where we can set different settings used for the SDK functionality. It accepts an object with various configuration settings for Lambda. In our case, we only care about selecting the region of the function, which we should already have from our test script configuration. We’ll set up a new LambdaClient instance and assign it to self.lambdaClient so we can use it to invoke the function.

The InvokeCommand function takes care of setting up the command used to invoke the Lambda function from the client instance. For our purposes, we’re configuring the command with two settings:

FunctionName - The name of the function to invoke, which we’ll get from the plugin’s configuration in the test script.
PayloadThe input to send to the Lambda function.

We’re assigning the InvokeCommand function to self.lambdaCommand, with an argument for our payload. The payload value we want to send to our Lambda function is our Artillery test run metrics when completed. The value of the payload needs to be a Uint8Array type, so we first have to encode it to the proper format using the TextEncoder class.

We have our AWS SDK setup wrapped up. All that’s left is to invoke the Lambda function after the Artillery test run. We can use the events argument to listen to the done event, which will contain the metrics we want to send to the Lambda function. However, we can’t invoke the Lambda function when listening to this event. Event handlers in Node.js are asynchronous, so the Artillery process will exit before giving us a chance to invoke the Lambda function.

To get around this issue, we can define the plugin’s cleanup function to invoke the Lambda function. The cleanup function can invoke the Lambda function asynchronously and wait for the response from AWS before the Artillery process exits. Our plugin will end up looking like this:

'use strict';

const { LambdaClient, InvokeCommand } = require('@aws-sdk/client-lambda');

module.exports.Plugin = ArtilleryLambdaPlugin;

function ArtilleryLambdaPlugin(script, events) {
  const self = this;

  self.config = script.config.plugins['lambda'];

  if (!self.config.function) {
    throw new Error('function is missing');
  }

  if (!self.config.region) {
    throw new Error('region is missing');
  }

  self.lambdaClient = function () {
    return new LambdaClient({ region: self.config.region });
  };

  self.lambdaCommand = function (report) {
    return new InvokeCommand({
      FunctionName: self.config.function,
      Payload: new TextEncoder().encode(JSON.stringify(report)),
    });
  };

  events.on('done', function (report) {
    self.report = report;
  });

  return this;
}

ArtilleryLambdaPlugin.prototype.cleanup = async function (done) {
  const client = this.lambdaClient();

  try {
    const result = await client.send(this.lambdaCommand(this.report));
    console.log(
      `Sent Artillery report to Lambda function: ${this.config.function} (${this.config.region})`,
    );
  } catch (err) {
    console.log('There was an error invoking the Lambda function');
  } finally {
    done(null);
  }
};

First, we’ll listen to the done event to capture the test run metrics and assign it to self.report. Next, we’ll create the cleanup function for the Plugin class. We set up an async function since the AWS SDK makes its requests asynchronously, and we want to wait for the Lambda invocation response. Inside the function, we can access all the class properties we set in the constructor through this, like our LambdaClient and InvokeCommand functions for the AWS SDK, the plugin configuration settings, and the Artillery test run metrics.

With this information, we’ll invoke the function and await the response. If there aren’t any errors from the AWS SDK, we’ll print a message indicating a successful invocation of the function. If the AWS SDK throws any errors, we’ll catch them and let the plugin user know. Regardless of a successful or failed attempt to invoke the Lambda function, we’ll call the done callback function to tell Artillery that our plugin finished doing what it needs to do.

Our plugin is now feature-complete! We can test the plugin by ensuring we set the AWS SDK credentials and a valid Lambda function specified in the plugin’s configuration. With everything set up correctly, we can use the plugin to run any Artillery test and have an AWS Lambda function receive the complete test run metrics.

Add debugging info to help isolate issues

To help aid you or users of the plugin in debugging issues during test runs, it’s a good practice to have a simple way to print out debugging messages to get a glimpse at the inner workings of your plugin. Using console.log can work during development, but you don’t want to litter the console for users of the plugin.

A simple way to add unobtrusive debugging to your plugins is with a small debugging package called debug. This package helps you set up debugging messages that only appear in your output when using the DEBUG environment variable with specific values. Since debug is an npm package, that means you can easily integrate it into your plugin. Let’s set up the package and add a few debugging statements to see how it works.

First, we’ll set up the debug package as a dependency with npm install:

npm install debug

Inside the index.js entrypoint for the plugin, we can import the debug function and set it up to print out debug messages relevant to our plugin:

const debug = require('debug')('plugin:lambda');

The plugin:lambda setting when importing the package is the name we want to specify for our module. It allows us to print the debugging messages with the DEBUG=plugin:lambda environment variable. This code exposes a function — assigned to debug — that we can use. Here’s how our plugin would look with a few debugging messages when invoking the Lambda function so that we can see the AWS SDK responses:

'use strict';

const { LambdaClient, InvokeCommand } = require('@aws-sdk/client-lambda');
const debug = require('debug')('plugin:lambda');

module.exports.Plugin = ArtilleryLambdaPlugin;

function ArtilleryLambdaPlugin(script, events) {
  const self = this;

  self.config = script.config.plugins['lambda'];

  if (!self.config.function) {
    throw new Error('function is missing');
  }

  if (!self.config.region) {
    throw new Error('region is missing');
  }

  self.lambdaClient = function () {
    return new LambdaClient({ region: self.config.region });
  };

  self.lambdaCommand = function (report) {
    return new InvokeCommand({
      FunctionName: self.config.function,
      Payload: new TextEncoder().encode(JSON.stringify(report)),
    });
  };

  events.on('done', function (report) {
    self.report = report;
  });

  return this;
}

ArtilleryLambdaPlugin.prototype.cleanup = async function (done) {
  const client = this.lambdaClient();

  try {
    const result = await client.send(this.lambdaCommand(this.report));
    console.log(
      `Sent Artillery report to Lambda function: ${this.config.function} (${this.config.region})`,
    );
    debug(result);
  } catch (err) {
    console.log('There was an error invoking the Lambda function');
    debug(err);
  } finally {
    debug('Wrapping up');
    done(null);
  }
};

The last thing left to do is publish the plugin on npm so you can share your work with the world!

Publish the plugin on npm

Publishing an npm package is a straightforward process — in just three steps, you’ll have your Artillery plugin published in just a few minutes:

First, you’ll need to sign up for a free npm account if you don’t have one already.
Next, set up a registry user in your local system using the npm adduser command. This command will prompt you to log in using your npm account credentials.
Finally, you can publish your package to npm using the npm publish command. This command picks up the information in the package.json file (generated by npm init) and makes your package available publicly.

If everything went well, you can visit npmjs.com and search for the plugin. It should appear available to everyone for immediate downloading. Now anyone can install your plugin and extend Artillery in a snap!

With your plugin available publicly, here are a few tips to help others get the most out of it:

When setting up your plugin, add as much information as possible to package.json, such as your name, the plugin’s homepage or code repository, and a brief but helpful description of how the plugin works. When others search for Artillery plugins on npmjs.com, this information will help them discover it.
Include a README file in your package with information about your plugin. The page for your plugin on npmjs.com will render the file to give more details on how it works. Adding installation notes, configuration settings, and debugging methods will considerably help others who want to use your plugin.
Do your best to follow semantic versioning to help your fellow developers depending on your plugin to keep track of any breaking changes. Make sure to increment your plugin’s version number and set up tags when updating the package.

Summary

This article shows just a small example of how you can extend Artillery’s functionality beyond what’s already included in the toolkit. Many developers have created convenient and practical plugins and made them available on npmjs.com. Here are a few plugins that can boost your load tests in different ways:

artillery-plugin-publish-metrics - Send metrics and events from your test runs to different services like Datadog and Mixpanel.
artillery-plugin-faker - Test the resiliency of your services by using randomized fake data for your scenarios.
artillery-plugin-metrics-by-endpoint - Expand your test run reports by breaking down latency and response codes for HTTP services on a per-endpoint basis.

If you can’t find a plugin to do what you need, Artillery makes it dead-simple to create a plugin that listens to different events during your load tests and gives you the chance to perform any action you want to add your testing. And thanks to the npm ecosystem, you can find npm packages for just about anything you need. The only limit is your creativity!