Running Artillery on Jenkins

Overview

Integrating Artillery with Jenkins allows you to track your service's performance while developing your applications using the popular open-source CI/CD tool. The following guide will show you how to load-test your services using a Jenkins Pipeline.

You can find the complete source code used in this guide in the Artillery CI/CD Examples GitHub repo (opens in a new tab).

Artillery test script example

In this guide, we'll use the following Artillery test script to run a load test on a simple API, creating 50 virtual users per second for 10 minutes and ensuring the aggregate maximum latency for the test is under 500 milliseconds:

config:
  target: "http://lab.artillery.io"
  phases:
    - duration: 600
      arrivalRate: 50
  ensure:
    maxErrorRate: 1
    max: 500
 
scenarios:
  - name: "Get a list of movies"
    flow:
      - get:
          url: "/movies"
          expect:
            - statusCode: 200

You can run an example of this test script and see it in action. (opens in a new tab)

Setting up Jenkins Pipelines

This guide assumes you have an accessible Jenkins server running. The examples in this guide use Docker as a Jenkins agent (opens in a new tab) for running Jenkins Pipelines, and build integration with GitHub (opens in a new tab) to run the Artillery load test from a GitHub repository. Jenkins supports various build agents and integration with different source code management services, so you can easily modify the sample configuration files for your needs.

For this guide, the Artillery load test for the Socket.IO service will run after pushing new code to the main branch of an existing GitHub repository with our Artillery test. We'll also use the official Docker image (opens in a new tab) to execute the load test without setting up Artillery as part of the Pipeline.

We'll use a Jenkins Pipeline to run our Artillery load tests. You can create a Jenkins Pipeline through the Jenkins UI or the newer Blue Ocean interface (opens in a new tab). For this guide, we'll initialize the Pipeline through the Jenkins interface and later create the Pipeline's configuration inside of the code repository (opens in a new tab).

The Jenkins Pipeline setup offers multiple options to set up a build and how to trigger it. The initial setup also asks how to define the Pipeline. The Pipeline can be set up directly through the Jenkins UI. However, for better control and traceability, a good choice is to create a Pipeline script in the code repository for your test, which we'll do in this example.

To have Jenkins look in our code repository, choose the "Pipeline script from SCM" Pipeline definition option and configure the repository based on your source code management service. Jenkins looks for a file called Jenkinsfile in the root of the code repository. The Jenkinsfile configuration supports two different ways to configure a Pipeline — Declarative and Scripted (opens in a new tab). Both formats work similarly, with some differences under the hood. For this guide, we'll follow the Declarative Pipeline syntax (opens in a new tab) for the Jenkinsfile configuration.

After saving the Jenkins Pipeline, create a new file called Jenkinsfile in the root of your code repository with the following contents:

pipeline {
    agent {
        docker {
            image 'artilleryio/artillery:latest'
            args '-u root:root -i --entrypoint='
        }
    }
 
    stages {
        stage('Load Test') {
            steps {
                sh '/home/node/artillery/bin/run run tests/performance/socket-io.yml'
            }
        }
    }
}

Note that when setting up the Artillery Docker image under agent, we're adding additional arguments that Jenkins will use to start the build:

• -u root:root - By default, the Docker build agent will execute any actions inside the container using the jenkins user. This user does not have permission to run Artillery inside the container, so adding this argument will allow us to execute any actions as the root user inside the container.
• -i - This argument sets the container to run in 'interactive' mode to allow Jenkins to execute any commands in the Pipeline stages.
• --entrypoint= - Jenkins will use the Docker image's defined ENTRYPOINT when starting the container, which we don't need for continuous integration purposes. This argument overrides the ENTRYPOINT so we can execute commands in the container.

Commit this file to the main branch of your repository. You can manually trigger a build by clicking on the Build Now option inside of your Pipeline project on Jenkins:

Generating and viewing Artillery test reports

Artillery can output a JSON file with additional details from the load test and use it to generate a self-contained HTML report (opens in a new tab).

First, the job needs to create a directory to place the test reports. Next, you can generate a JSON report when executing the Artillery load test. You can then use the report command (opens in a new tab) to generate the HTML report from the JSON file. Finally, you'll need to archive the files as artifacts (opens in a new tab) to access the reports upon completion of the job.

The following configuration accomplishes these steps:

pipeline {
    agent {
        docker {
            image 'artilleryio/artillery:latest'
            args '-u root:root -i --entrypoint='
        }
    }
 
    stages {
        stage('Load Test') {
            steps {
                sh 'mkdir reports'
                sh '/home/node/artillery/bin/run run --output reports/report.json tests/performance/socket-io.yml'
                sh '/home/node/artillery/bin/run report --output reports/report reports/report.json'
            }
        }
    }
 
    post {
        success {
            archiveArtifacts 'reports/*'
        }
    }
}

After successful execution of the load test, Jenkins will store the files from the reports directory created as part of the build, which you can download from the build details page on Jenkins:

For more details on accessing the artifacts of a Jenkins build, read the "Recording tests and artifacts" (opens in a new tab) section in the Jenkins documentation.

Scheduling Artillery load tests

You can also set up Jenkins to run a build on a recurring schedule by setting up a trigger in the Pipeline definition (opens in a new tab), which is helpful if you want to execute your Artillery load tests at a specific time. For instance, you may wish to load-test your production applications outside of peak hours.

You can include the triggers directive in the Jenkinsfile configuration file with the cron parameter using the POSIX cron syntax (opens in a new tab) (with some minor differences (opens in a new tab)). The cron trigger uses the time zone set in your Jenkins server. The following configuration will automatically trigger the Jenkins build every day at midnight, based on the server's time zone:

pipeline {
    agent {
        docker {
            image 'artilleryio/artillery:latest'
            args '-u root:root -i --entrypoint='
        }
    }
 
    triggers {
        cron('0 0 * * *')
    }
 
    stages {
        stage('Load Test') {
            steps {
                sh 'mkdir reports'
                sh '/home/node/artillery/bin/run run --output reports/report.json tests/performance/socket-io.yml'
                sh '/home/node/artillery/bin/run report --output reports/report reports/report.json'
            }
        }
    }
 
    post {
        success {
            archiveArtifacts 'reports/*'
        }
    }
}

Distributed load testing on Jenkins

You can scale your load tests by using Artillery's support for running on AWS Lambda and AWS Fargate to execute your tests from different geographic regions within your AWS infrastructure.

You will need to set up the following environment variables as secret text values in the Jenkins global credentials section (opens in a new tab):

• AWS_ACCESS_KEY_ID - The access key ID of an IAM user with permissions to run Artillery. In this example, we'll set up this credential with the ID jenkins-aws-secret-key-id.
• AWS_SECRET_ACCESS_KEY - The secret access key of the IAM user. In this example, we'll set up this credential with the ID jenkins-aws-secret-access-key.

The following configuration will execute your Artillery test script using the AWS_ACCESS_KEY_ID (using the credential ID jenkins-aws-secret-key-id) and AWS_SECRET_ACCESS_KEY (using the credential ID jenkins-aws-secret-access-key) set as environment variables using the credentials specified in Jenkins (opens in a new tab):

pipeline {
    agent {
        docker {
            image 'artilleryio/artillery:latest'
            args '-u root:root -i --entrypoint='
        }
    }
 
    triggers {
        cron('0 0 * * *')
    }
 
    environment {
        AWS_ACCESS_KEY_ID = credentials('jenkins-aws-secret-key-id')
        AWS_SECRET_ACCESS_KEY = credentials('jenkins-aws-secret-access-key')
    }
 
    stages {
        stage('Load Test on AWS') {
            steps {
                sh '/home/node/artillery/bin/run run-fargate --region us-east-1 --count 5 my-script.yml'
            }
        }
    }
}