- How to set up a Jenkins Pipeline to run your Artillery tests
- How to generate and view Artillery test reports from Jenkins
- How to schedule your Artillery tests to run at a specific time
- How to use Artillery Pro in a Jenkins pipeline
Integrating Artillery with Jenkins allows you to track your service’s performance while developing your applications using the popular open-source CI/CD tool. The following guide will show you how to load-test your services using a Jenkins Pipeline.
You can find the complete source code used in this guide in the Artillery CI/CD Examples GitHub repo.
In this guide, we’ll use the following Artillery test script to run a load test on a running Socket.IO service, sending 50 virtual users per second for 10 minutes and ensuring the aggregate maximum latency for the test is under 500 milliseconds:
This guide assumes you have an accessible Jenkins server running. The examples in this guide use Docker as a Jenkins agent for running Jenkins Pipelines, and build integration with GitHub to run the Artillery load test from a GitHub repository. Jenkins supports various build agents and integration with different source code management services, so you can easily modify the sample configuration files for your needs.
For this guide, the Artillery load test for the Socket.IO service will run after pushing new code to the
main branch of an existing GitHub repository with our Artillery test. We’ll also use the official Docker image to execute the load test without setting up Artillery as part of the Pipeline.
We’ll use a Jenkins Pipeline to run our Artillery load tests. You can create a Jenkins Pipeline through the Jenkins UI or the newer Blue Ocean interface. For this guide, we’ll initialize the Pipeline through the Jenkins interface and later create the Pipeline’s configuration inside of the code repository.
The Jenkins Pipeline setup offers multiple options to set up a build and how to trigger it. The initial setup also asks how to define the Pipeline. The Pipeline can be set up directly through the Jenkins UI. However, for better control and traceability, a good choice is to create a Pipeline script in the code repository for your test, which we’ll do in this example.
To have Jenkins look in our code repository, choose the “Pipeline script from SCM” Pipeline definition option and configure the repository based on your source code management service. Jenkins looks for a file called
Jenkinsfile in the root of the code repository. The
Jenkinsfile configuration supports two different ways to configure a Pipeline — Declarative and Scripted. Both formats work similarly, with some differences under the hood. For this guide, we’ll follow the Declarative Pipeline syntax for the
After saving the Jenkins Pipeline, create a new file called
Jenkinsfile in the root of your code repository with the following contents:
Note that when setting up the Artillery Docker image under
agent, we’re adding additional arguments that Jenkins will use to start the build:
-u root:root- By default, the Docker build agent will execute any actions inside the container using the
jenkinsuser. This user does not have permission to run Artillery inside the container, so adding this argument will allow us to execute any actions as the
rootuser inside the container.
-i- This argument sets the container to run in ‘interactive’ mode to allow Jenkins to execute any commands in the Pipeline stages.
--entrypoint=- Jenkins will use the Docker image’s defined
ENTRYPOINTwhen starting the container, which we don’t need for continuous integration purposes. This argument overrides the
ENTRYPOINTso we can execute commands in the container.
Commit this file to the
main branch of your repository. You can manually trigger a build by clicking on the Build Now option inside of your Pipeline project on Jenkins:
Artillery can output a JSON file with additional details from the load test and use it to generate a self-contained HTML report.
First, the job needs to create a directory to place the test reports. Next, you can generate a JSON report when executing the Artillery load test. You can then use the
report command to generate the HTML report from the JSON file. Finally, you’ll need to archive the files as artifacts to access the reports upon completion of the job.
The following configuration accomplishes these steps:
After successful execution of the load test, Jenkins will store the files from the
reports directory created as part of the build, which you can download from the build details page on Jenkins:
For more details on accessing the artifacts of a Jenkins build, read the “Recording tests and artifacts” section in the Jenkins documentation.
You can also set up Jenkins to run a build on a recurring schedule by setting up a trigger in the Pipeline definition, which is helpful if you want to execute your Artillery load tests at a specific time. For instance, you may wish to load-test your production applications outside of peak hours.
You can include the
triggers directive in the
Jenkinsfile configuration file with the
cron parameter using the POSIX cron syntax (with some minor differences). The
cron trigger uses the time zone set in your Jenkins server. The following configuration will automatically trigger the Jenkins build every day at midnight, based on the server’s time zone:
You can scale your load tests by using Artillery Pro to execute your tests. Artillery Pro allows you to run your Artillery tests from different geographic regions within your AWS infrastructure.
To use Artillery Pro with GitHub Actions, you must have previously deployed the Artillery Pro backend to your AWS account. You also must set up the following environment variables as secret text values in the Jenkins global credentials section:
AWS_ACCESS_KEY_ID- The access key ID of an IAM user with permissions to access the Artillery Pro backend resources. In this example, we’ll set up this credential with the ID
AWS_SECRET_ACCESS_KEY- The secret access key of the IAM user. In this example, we’ll set up this credential with the ID
The following configuration will execute your Artillery test script on an Artillery Pro backend deployed in the
us-east-1 region using the
AWS_ACCESS_KEY_ID (using the credential ID
AWS_SECRET_ACCESS_KEY (using the credential ID
jenkins-aws-secret-access-key) set as environment variables using the credentials specified in Jenkins:
For more options on executing your tests using Artillery Pro, check the Pro Commands section in the Command Line guide.