We’ve designed Artillery to be hackable and easy to extend. Plugins hook into Artillery’s internal APIs to allow you to include additional functionality like generating different kinds of data for your tests, integrating with third-party systems, or sending metrics to another platform.
Since Artillery uses Node.js, you can use tens of thousands of Node.js packages available on npm to take advantage of existing libraries you can integrate right away. You can find npm packages for almost anything, making it much easier to create and extend Artillery without much hassle.
Plugins in Artillery are standard npm packages, and they can be either public or private. By default, Artillery will look in the npm package path for the plugin with
artillery-plugin- prefixed to the name describing your plugin. In this example, we’ll name our plugin
artillery-plugin-lambda since we will invoke Lambda functions in our code.
Creating the npm package is a straightforward process using the
npm init command inside of a directory where you want to build your plugin. For our example, we’ll create a directory called
artillery-plugin-lambda. When executing the
npm init command without any arguments inside this directory, it will ask you a few questions for setting up the project, like the name of the package, a description, and other details. Most of the questions are optional, so you don’t have to answer all of them at this time. If you plan to publish the plugin later, it’s a good practice to add as much detail as possible to help other users find it.
> npm init
npm init will generate a
package.json file for you based on your answers. This file serves as a starting point for your plugin. Later in the article, this file will get updated to include our dependencies like the AWS SDK.
The package for a plugin is expected to export a
Plugin class so Artillery can load it properly. The constructor for the class is a function that receives two arguments:
script- The full test script definition. The plugin code can modify the test script to extend the current load test definition, like attaching hooks to existing scenarios.
EventEmitterwhich you can use to subscribe to events sent from Artillery during a test run. You can also use the argument to send custom metrics.
In addition to subscribing to Artillery events, plugins can also define a
cleanup function with a
done callback. This function gets called before Artillery exits upon the completion of a test run. It provides an opportunity to perform any cleanup that your plugin needs before wrapping up. We’ll see how to use this function later in the article.
To start constructing our plugin, let’s create a new file called
index.js in our plugin directory. This file will serve as the entry point for our npm package. Inside the file, we can set up the
Plugin class and a function to define the constructor of the class:
As mentioned at the beginning of this section, Artillery plugins need to export a
Plugin class. Here, we’re using the
ArtilleryLambdaPlugin function to load the plugin’s code. The
ArtilleryLambdaPlugin function receives the
events arguments, and from there, you can create the custom functionality for your plugin. To see how these work, we’ll first demonstrate how to use the
events argument and later see how to use the
script argument to access our test script inside the plugin.
events argument allows us to subscribe to the following events:
phaseStarted- Emitted when a new load phase starts.
phaseCompleted- Emitted when a load phase completes.
stats- Emitted after each reporting period during a test run (every 10 seconds or at the end of a load phase).
done- Emitted when all VUs complete their scenarios, and there are no more load phases to run.
events is a Node.js
EventEmitter object, we can set up a listener function that gets called when Artillery emits the event using
.on. For example, let’s set up a listener for each of the events emitted by an Artillery test run to see how plugins work by printing a few messages to the console:
Let’s verify the plugin is working correctly by loading it into an Artillery test script and putting it into action.
For purposes of this example, let’s assume we have the following Artillery test script called
api-test.yml, which has three load phases and a scenario that hits two endpoints for an HTTP service:
An existing test script can load an Artillery plugin by using the
config.plugins setting. The setting will tell Artillery to load an npm package using the specified plugin name, prefixed with
artillery-plugin- as mentioned at the beginning of this article. Our example plugin is named
artillery-plugin-lambda, so we can configure our plugin using
config.plugins setting, we’re loading our plugin using
lambda. Since we’re not providing any configuration to use in the plugin at this time, we’re setting an empty mapping for it. Later on, we’ll show how to set up some configuration that the plugin code can access and use internally.
Since plugins are npm packages, Artillery will look for them as any other package. That means if you installed Artillery globally in your system (like using
npm install -g artillery@latest, for instance), it would try to find the
artillery-plugin-lambda plugin in your global npm packages for this example. If you installed Artillery in your project (without the
-g flag), it would search for the plugin in your project’s installed dependencies.
If we attempt to run the example test script with
artillery run api-test.yml, the load test will run. However, it will print the following warning message in the console:
WARNING: Plugin lambda specified but module artillery-plugin-lambda could not be found (MODULE_NOT_FOUND). The plugin is not installed either locally or globally because we don’t have an npm package yet to set up during development.
We don’t have to create an npm package for our test, though. Artillery allows us to set up the path of our plugin by using the
ARTILLERY_PLUGIN_PATH environment variable. It’s a convenient way to tell Artillery where to look for any defined plugins. For this example, let’s assume our plugin is in the
/home/artillery/plugins/artillery-plugin-lambda directory. We can load the plugin when running the example test script with the following command:
Artillery will look inside the
/home/artillery/plugins directory for an npm package matching the name of the plugin it’s expecting. In our case, it’s expecting the
artillery-plugin-lambda directory. It will find the directory and load the entry point, activating the plugin for the test run. If everything goes as expected, you will see the console messages set up in the plugin during the test run.
We’ve seen how to build the basics of an Artillery plugin, some of the events you can listen to, and how to load the plugin locally during development. Let’s begin setting up the main bits of functionality we want to extend in Artillery, which is invoking an AWS Lambda function when a test run completes.
Most Artillery plugins will have different settings to allow anyone using the plugin to add different configuration settings for their needs. For instance, you can set up access keys to third-party services that the plugin uses or define paths to generate various artifacts in the system running the test. For our example, we want to allow users of our plugin to let others set the name of the function they wish to invoke and the AWS region where the function exists. We’ll call these settings
For our example, we’ll have an AWS Lambda function called
AggregateArtilleryResults located in the
ap-northeast-1 region. We want our plugin’s configuration to look like this:
We have to tell our plugin to read these values from the test script to use them later when setting up our invocation functionality. The
Plugin class constructor receives a
script argument, which contains the entirety of the test script as an object. We can access the configuration for our plugin through this argument.
index.js file for our plugin, let’s delete the example code we used before and set up our custom functionality. First, we’ll access the configuration from the test script and set it up in a variable for future use:
In this code, we first set up a reference to
this by assigning it to the
self variable to manage the original context when handling events, as we’ll show later. Next, we’re reading our test script through the
script argument. We want to access our plugin’s configuration, so we’ll access
script.config.plugins['lambda'] to get the configuration settings and assign them to
self.config. Based on our test script above, the value of
Because we’ll need these settings to configure the AWS SDK and invoke the correct Lambda function, a good practice is to alert any users loading the plugin if either of these settings is missing. A quick way to do this is by throwing an exception with a helpful message telling users that some configuration is missing:
With the added checks, if either the
region setting is missing from the plugin configuration under
lambda, the plugin will throw the exception and print out the respective message in the console at the beginning of the Artillery test run. Keep in mind that although your plugin won’t load after the error, the Artillery test script will continue running without using any of the plugin’s functionality.
With the plugin configuration at hand, we can now set up the AWS SDK library to begin invoking the configured Lambda function.
npm install @aws-sdk/client-lambda
After installing the package, we can use it to set up the Lambda client in our plugin. We’ll need two functions from the SDK. The first function is
LambdaClient, which initializes the client. The second function we need is the
InvokeCommand to set up how we’ll invoke the function using the client. The SDK separates both client and commands, so we can import both functions and begin setting them up in the plugin:
LambdaClient function is a class constructor where we can set different settings used for the SDK functionality. It accepts an object with various configuration settings for Lambda. In our case, we only care about selecting the region of the function, which we should already have from our test script configuration. We’ll set up a new
LambdaClient instance and assign it to
self.lambdaClient so we can use it to invoke the function.
InvokeCommand function takes care of setting up the command used to invoke the Lambda function from the client instance. For our purposes, we’re configuring the command with two settings:
FunctionName- The name of the function to invoke, which we’ll get from the plugin’s configuration in the test script.
PayloadThe input to send to the Lambda function.
We’re assigning the
InvokeCommand function to
self.lambdaCommand, with an argument for our payload. The payload value we want to send to our Lambda function is our Artillery test run metrics when completed. The value of the payload needs to be a
Uint8Array type, so we first have to encode it to the proper format using the
We have our AWS SDK setup wrapped up. All that’s left is to invoke the Lambda function after the Artillery test run. We can use the
events argument to listen to the
done event, which will contain the metrics we want to send to the Lambda function. However, we can’t invoke the Lambda function when listening to this event. Event handlers in Node.js are asynchronous, so the Artillery process will exit before giving us a chance to invoke the Lambda function.
To get around this issue, we can define the plugin’s
cleanup function to invoke the Lambda function. The
cleanup function can invoke the Lambda function asynchronously and wait for the response from AWS before the Artillery process exits. Our plugin will end up looking like this:
First, we’ll listen to the
done event to capture the test run metrics and assign it to
self.report. Next, we’ll create the
cleanup function for the
Plugin class. We set up an
async function since the AWS SDK makes its requests asynchronously, and we want to wait for the Lambda invocation response. Inside the function, we can access all the class properties we set in the constructor through
this, like our
InvokeCommand functions for the AWS SDK, the plugin configuration settings, and the Artillery test run metrics.
With this information, we’ll invoke the function and
await the response. If there aren’t any errors from the AWS SDK, we’ll print a message indicating a successful invocation of the function. If the AWS SDK throws any errors, we’ll catch them and let the plugin user know. Regardless of a successful or failed attempt to invoke the Lambda function, we’ll call the
done callback function to tell Artillery that our plugin finished doing what it needs to do.
Our plugin is now feature-complete! We can test the plugin by ensuring we set the AWS SDK credentials and a valid Lambda function specified in the plugin’s configuration. With everything set up correctly, we can use the plugin to run any Artillery test and have an AWS Lambda function receive the complete test run metrics.
To help aid you or users of the plugin in debugging issues during test runs, it’s a good practice to have a simple way to print out debugging messages to get a glimpse at the inner workings of your plugin. Using
console.log can work during development, but you don’t want to litter the console for users of the plugin.
A simple way to add unobtrusive debugging to your plugins is with a small debugging package called debug. This package helps you set up debugging messages that only appear in your output when using the
DEBUG environment variable with specific values. Since debug is an npm package, that means you can easily integrate it into your plugin. Let’s set up the package and add a few debugging statements to see how it works.
First, we’ll set up the debug package as a dependency with
npm install debug
index.js entrypoint for the plugin, we can import the
debug function and set it up to print out debug messages relevant to our plugin:
const debug = require('debug')('plugin:lambda');
plugin:lambda setting when importing the package is the name we want to specify for our module. It allows us to print the debugging messages with the
DEBUG=plugin:lambda environment variable. This code exposes a function — assigned to
debug — that we can use. Here’s how our plugin would look with a few debugging messages when invoking the Lambda function so that we can see the AWS SDK responses:
The last thing left to do is publish the plugin on npm so you can share your work with the world!
Publishing an npm package is a straightforward process — in just three steps, you’ll have your Artillery plugin published in just a few minutes:
- First, you’ll need to sign up for a free npm account if you don’t have one already.
- Next, set up a registry user in your local system using the
npm addusercommand. This command will prompt you to log in using your npm account credentials.
- Finally, you can publish your package to npm using the
npm publishcommand. This command picks up the information in the
package.jsonfile (generated by
npm init) and makes your package available publicly.
If everything went well, you can visit npmjs.com and search for the plugin. It should appear available to everyone for immediate downloading. Now anyone can install your plugin and extend Artillery in a snap!
With your plugin available publicly, here are a few tips to help others get the most out of it:
- When setting up your plugin, add as much information as possible to
package.json, such as your name, the plugin’s homepage or code repository, and a brief but helpful description of how the plugin works. When others search for Artillery plugins on npmjs.com, this information will help them discover it.
- Include a
READMEfile in your package with information about your plugin. The page for your plugin on npmjs.com will render the file to give more details on how it works. Adding installation notes, configuration settings, and debugging methods will considerably help others who want to use your plugin.
- Do your best to follow semantic versioning to help your fellow developers depending on your plugin to keep track of any breaking changes. Make sure to increment your plugin’s version number and set up tags when updating the package.
This article shows just a small example of how you can extend Artillery’s functionality beyond what’s already included in the toolkit. Many developers have created convenient and practical plugins and made them available on npmjs.com. Here are a few plugins that can boost your load tests in different ways:
artillery-plugin-publish-metrics- Send metrics and events from your test runs to different services like Datadog and Mixpanel.
artillery-plugin-faker- Test the resiliency of your services by using randomized fake data for your scenarios.
artillery-plugin-metrics-by-endpoint- Expand your test run reports by breaking down latency and response codes for HTTP services on a per-endpoint basis.
If you can’t find a plugin to do what you need, Artillery makes it dead-simple to create a plugin that listens to different events during your load tests and gives you the chance to perform any action you want to add your testing. And thanks to the npm ecosystem, you can find npm packages for just about anything you need. The only limit is your creativity!