Running Artillery on CircleCI

Integrating Artillery with CircleCI allows you to continuously track your service's performance while developing your applications. The following guide will show you how to load-test your services using CircleCI workflows.

You’ll learn:

• How to set up a CircleCI workflow to run your Artillery tests
• How to generate and view Artillery test reports from CircleCI
• How to schedule your Artillery tests to run at a specific time

You can find the complete source code used in this guide in the Artillery CI/CD Examples GitHub repo (opens in a new tab).

Artillery test script example

In this guide, we'll use the following Artillery test script to run a load test on a simple API, creating 50 virtual users per second for 10 minutes and ensuring the aggregate maximum latency for the test is under 500 milliseconds:

config:
  target: 'http://lab.artillery.io'
  phases:
    - duration: 600
      arrivalRate: 50
  ensure:
    maxErrorRate: 1
    max: 500
 
scenarios:
  - name: 'Get a list of movies'
    flow:
      - get:
          url: '/movies'
          expect:
            - statusCode: 200

You can run an example of this test script and see it in action. (opens in a new tab)

Setting up CircleCI

For this guide, we'll use the cloud version of CircleCI and a GitHub code repository as an example. Currently, the cloud version of CircleCI only supports logging in to the service using your GitHub or Bitbucket credentials.

Go to the "Projects" section from the sidebar menu when logging in to CircleCI using either your GitHub or Bitbucket credentials. You'll see the code repositories available in your GitHub or Bitbucket account. Search for the code repository containing your tests and click on the "Set Up Project" button to prepare the repo for use with CircleCI:

Next, you'll have the option to choose a starter template configuration file or tell CircleCI to look for a configuration file in a branch from your code repository. CircleCI looks for the .circleci/config.yml configuration file placed into your code repository to run your CI workflow. The configuration file uses YAML syntax to define the events that will trigger your workflow and the jobs and steps to execute.

Before proceeding on CircleCI, we'll create a configuration file in the main branch of the code repository. The configuration file will run the Artillery load test for the Socket.IO service after pushing new code to your GitHub repository's main branch. We'll also use the official Docker image (opens in a new tab) to execute the load test without setting up Artillery as part of the workflow.

In your code repository, create a new file placed in .circleci/config.yml with the following contents:

version: 2.1
 
jobs:
  artillery:
    docker:
      - image: artilleryio/artillery:latest
    steps:
      - checkout
 
      - run:
          name: Execute load tests
          command: /home/node/artillery/bin/run run tests/performance/socket-io.yml
 
workflows:
  load-tests:
    jobs:
      - artillery:
        filters:
          branches:
            only: main

Commit this file to the main branch of your repository. Go back to CircleCI to finish setting up the project by telling the service to check the main branch of the repo. It will detect the .circleci/config.yml file in the branch:

Clicking on "Let's Go" will immediately execute the load test on CircleCI:

Generating and viewing Artillery test reports

Artillery can output a JSON file with additional details from the load test and use it to generate a self-contained HTML report (opens in a new tab).

First, your workflow needs to create a directory to place the test reports. Next, you can generate a JSON report when executing the Artillery load test. You can then use the report command (opens in a new tab) to generate the HTML report from the JSON file. Finally, you'll need to upload the files as artifacts (opens in a new tab) to access the reports upon completion of the workflow.

The configuration to accomplish these steps is the following:

version: 2.1
 
jobs:
  artillery:
    docker:
      - image: artilleryio/artillery:latest
    steps:
      - checkout
 
      - run:
          name: Make reports directory
          command: mkdir reports
 
      - run:
          name: Execute load tests
          command: /home/node/artillery/bin/run run --output reports/report.json tests/performance/socket-io.yml
 
      - run:
          name: Generate HTML report
          command: /home/node/artillery/bin/run report --output reports/report reports/report.json
 
      - store_artifacts:
          path: reports
 
workflows:
  load-tests:
    jobs:
      - artillery:
          filters:
            branches:
              only: main

After successful execution of the load test, CircleCI will store the contents of the reports directory as part of the executed job:

For more details on uploading and accessing artifacts of a CircleCI job, read "Storing Build Artifacts" (opens in a new tab) section in CircleCI's documentation.

Scheduling Artillery load tests

You can also schedule CircleCI to run a job on a recurring schedule using the schedule trigger configuration setting in the workflow, which is helpful if you want to execute your Artillery load tests at a specific time. For instance, you may wish to load-test your production applications outside of peak hours.

The schedule trigger requires two keys:

• cron - A string using the POSIX cron syntax (opens in a new tab) with the times in UTC. This setting will configure when to execute the job.
• filters - A map to define rules for executing the recurring scheduled job on specific branches.

The following example configuration will automatically trigger the Artillery job on CircleCI every day at midnight UTC, running the job only on the main branch of the code repository:

version: 2.1
 
jobs:
  artillery:
    docker:
      - image: artilleryio/artillery:latest
    steps:
      - checkout
 
      - run:
          name: Make reports directory
          command: mkdir reports
 
      - run:
          name: Execute load tests
          command: /home/node/artillery/bin/run run --output reports/report.json tests/performance/socket-io.yml
 
      - run:
          name: Generate HTML report
          command: /home/node/artillery/bin/run report --output reports/report reports/report.json
 
      - store_artifacts:
          path: reports
 
workflows:
  load-tests:
    jobs:
      - artillery:
          filters:
            branches:
              only: main
  nightly:
    jobs:
      - artillery
    triggers:
      - schedule:
          cron: '0 0 * * *'
          filters:
            branches:
              only:
                - main

For more information about scheduling jobs on CircleCI, read the "Scheduling a workflow" (opens in a new tab) section on CircleCI's documentation.

Distributed load testing on CircleCI

You can scale your load tests by using Artillery's support for AWS Lambda and AWS Fargate to execute your tests.

You will need set up the following environment variables as environment variables for your project (opens in a new tab):

• AWS_ACCESS_KEY_ID - The access key ID of an IAM user with permissions to run Artillery tests
• AWS_SECRET_ACCESS_KEY - The secret access key of the IAM user.

Once the environment variables are set up for your project in CircleCI, the container running Artillery will have access to its values. You won't need to specify these values in the configuration file.

The following configuration will execute your Artillery test script in the us-east-1 region (automatically using the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables set up for the project):

version: 2.1
 
jobs:
  artillery:
    docker:
      - image: artilleryio/artillery:latest
    steps:
      - checkout
 
      - run:
          name: Execute load tests on AWS (us-east-1 region)
          command: /home/node/artillery/bin/run artillery run-fargate --region us-east-1 --count 5 my-test.yml
 
workflows:
  load-tests:
    jobs:
      - artillery:
          filters:
            branches:
              only: main
  nightly:
    jobs:
      - artillery
    triggers:
      - schedule:
          cron: '0 0 * * *'
          filters:
            branches:
              only:
                - main