Integrating Artillery with GitLab CI/CD


What you’ll learn

  • How to set up GitLab CI/CD to run your Artillery tests
  • How to generate and view Artillery test reports from GitLab CI/CD
  • How to schedule your Artillery tests to run at a specific time
  • How to use Artillery Pro in GitLab CI/CD

Overview

Integrating Artillery with GitLab CI/CD allows you to track your service’s performance while developing your applications in your existing GitLab repositories. The following guide will show you how to load-test your services using GitLab’s built-in continuous integration and continuous delivery service.

You can find the complete source code used in this guide in the Artillery CI/CD Examples GitHub repo.

Artillery test script example

In this guide, we’ll use the following Artillery test script to run a load test on a running Socket.IO service, sending 50 virtual users per second for 10 minutes and ensuring the aggregate maximum latency for the test is under 500 milliseconds:

config:
target: "http://lab.artillery.io"
phases:
- duration: 600
arrivalRate: 50
ensure:
maxErrorRate: 1
max: 500

scenarios:
- name: "Emit an event"
engine: "socketio"
flow:
- emit:
channel: "echo"
data: "Hello from Artillery"
response:
channel: "echoResponse"
data: "Hello from Artillery"

You can run an example of this test script and see it in action.

Setting up GitLab CI/CD

GitLab CI/CD looks for a configuration file placed inside your code repository called .gitlab-ci.yml. The configuration files use YAML syntax to define the events that will trigger your workflow and the jobs and steps to execute.

For this guide, the Artillery load test for the Socket.IO service will run after pushing new code to the main branch of your existing GitLab repository. We’ll also use the official Docker image to execute the load test without setting up Artillery as part of the workflow.

Before creating the .gitlab-ci.yml configuration file, you’ll first need to ensure your GitLab project has runners available to execute your CI/CD jobs. Most projects hosted on GitLab’s SaaS offering have shared runners available. To verify the available CI/CD runners for your project, go to the Settings → CI/CD section on your project and expand the Runners section.

Verifying GitLab CI/CD runners for a project

If your project does not have any available runners (for instance, if your organization self-hosts GitLab), you’ll have to install GitLab Runner and register a runner before using GitLab CI/CD.

When your project has available runners, create the GitLab CI/CD configuration file in .gitlab-ci.yml with the following contents:

artillery:
image:
name: artilleryio/artillery:latest
entrypoint: [""]
script: /home/node/artillery/bin/artillery run tests/performance/socket-io.yml

Note that the Docker image has its ENTRYPOINT overridden in the configuration file. GitLab CI/CD runners start any Docker containers using the defined ENTRYPOINT, which we don’t need for continuous integration purposes.

Commit this file to the main branch of your repository. Once you push the update to GitLab, a GitLab CI/CD runner will execute the load test:

Successful Artillery load test on GitLab CI/CD

Generating and viewing Artillery test reports

Artillery can output a JSON file with additional details from the load test and use it to generate a self-contained HTML report.

First, the job needs to create a directory to place the test reports. Next, you can generate a JSON report when executing the Artillery load test. You can then use the report command to generate the HTML report from the JSON file. Finally, you’ll need to store the files as artifacts to access the reports upon completion of the job.

The following configuration accomplishes these steps:

artillery:
image:
name: artilleryio/artillery:latest
entrypoint: [""]
script: |
mkdir reports
/home/node/artillery/bin/artillery run --output reports/report.json tests/performance/socket-io.yml
/home/node/artillery/bin/artillery report --output reports/report.html reports/report.json
artifacts:
paths:
- reports

After successful execution of the load test, GitLab CI/CD will store a zip file of the directory created as part of the job, which you can download from the list of jobs for your project:

Downloading Artillery test reports from GitLab CI/CD job artifacts

For more details on accessing the artifacts of a GitLab CI/CD job, read the “Download job artifacts” section in GitLab’s documentation.

Scheduling Artillery load tests

You can also set up GitLab CI/CD to run a job on a recurring schedule by using the built-in Schedules section for your project, which is helpful if you want to execute your Artillery load tests at a specific time. For instance, you may wish to load-test your production applications outside of peak hours.

You can set a schedule for any available pipelines in your GitLab project by going to the CI/CD → Schedules section on your project and clicking on the New schedule button:

GitLab CI/CD Schedules section Setting up a new schedule on GitLab CI/CD

When setting up a new schedule, you can specify the interval, which timezone you want to use, set the target branch, and include variables for your job. For more information, read the “Pipeline schedules” section in GitLab’s documentation.

Geographically distributed load testing with Artillery Pro

You can scale your load tests by using Artillery Pro to execute your tests. Artillery Pro allows you to run your Artillery tests from different geographic regions within your AWS infrastructure.

To use Artillery Pro with GitHub Actions, you must have previously deployed the Artillery Pro backend to your AWS account. You also must set up the following environment variables to make them accessible to your GitLab CI/CD job:

  • AWS_ACCESS_KEY_ID - The access key ID of an IAM user with permissions to access the Artillery Pro backend resources.
  • AWS_SECRET_ACCESS_KEY - The secret access key of the IAM user.

Environment variables for GitLab CI/CD can be set for your project by going to the Settings → CI/CD section on your project and expanding the Variables section:

Setting up environment variables on GitLab CI/CD

Once the environment variables are set up for your project in GitLab, the container running Artillery Pro will have access to its values. You won’t need to specify these values in the configuration file.

The following configuration will execute your Artillery test script on an Artillery Pro backend deployed in the us-east-1 region (automatically using the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables set up for the project):

artillery:
image:
name: artilleryio/artillery:latest
entrypoint: [""]
script: |
npm install artillery-pro@latest
/home/node/artillery/bin/artillery artillery run-test --cluster artillery-pro-cluster --region us-east-1 --count 5

For more options on executing your tests using Artillery Pro, check the Pro Commands section in the Command Line guide.