Running Artillery on AWS CodeBuild

Overview

Integrating Artillery with AWS CodeBuild allows you to track your service's performance at scale while developing your applications on Amazon's AWS platform. The following guide will show you how to load-test your services using an AWS CodeBuild project.

You can find the complete source code used in this guide in the Artillery CI/CD Examples GitHub repo (opens in a new tab).

Artillery test script example

In this guide, we'll use the following Artillery test script to run a load test on a simple API, creating 50 virtual users per second for 10 minutes and ensuring the aggregate maximum latency for the test is under 500 milliseconds:

config:
  target: "http://lab.artillery.io"
  phases:
    - duration: 600
      arrivalRate: 50
  ensure:
    maxErrorRate: 1
    max: 500
 
scenarios:
  - name: "Get a list of movies"
    flow:
      - get:
          url: "/movies"
          expect:
            - statusCode: 200

You can run an example of this test script and see it in action. (opens in a new tab)

Setting up AWS CodeBuild project

This guide assumes you have an existing AWS account with permissions to access the different resources used when setting up AWS CodeBuild projects, such as S3 (for storing and retrieving build artifacts) and CloudWatch (for generating and viewing build logs). For details on the required credentials and permissions, read the "Identity and access management in AWS CodeBuild" (opens in a new tab) section on the AWS CodeBuild User Guide.

We'll go through the process of setting up a new AWS CodeBuild project on the AWS console. After signing in to the console, go to the CodeBuild service to create a build project. The Create build project page provides many setup options, depending on your needs. The following are the main sections you'll need to set up for new CodeBuild projects:

Source

AWS CodeBuild can pull in your source code from different providers, such as AWS CodeCommit, an S3 bucket, GitHub, and Bitbucket. Each source provider has different settings, both required and optional. As an example for this guide, we'll use GitHub as the source provider, using a repository from the connected account. This example also sets up a webhook that listens to any PUSH events in the GitHub repository to trigger a new build.

Environment

You can choose different environments for your builds. AWS CodeBuild provides options to use a managed image (opens in a new tab) or a custom Docker image. We'll use a managed image based on Ubuntu 20.04 (aws/codebuild/standard:5.0 image) for this guide, which contains Node.js version 14 as a runtime, which is needed to install Artillery.

This step also requires setting up a service role for your AWS CodeBuild project. This service role delegates permissions between the different AWS services used for builds, like generating logs in CloudWatch or sending artifacts to S3. You can create a new service role or use an existing one from your AWS account.

Buildspec

The buildspec is the configuration used by AWS CodeBuild to process the different settings and commands in each project build. The configuration file can be read from the source specified in the project setup or created directly in the AWS console. When using the source, CodeBuild looks for a file named buildspec.yml in the source code root directory by default. If needed, you can change the file's name that AWS CodeBuild will look for in the repo. For this guide, we'll create a buildspec.yml file in the GitHub repository.

Logs

Setting up build logs is optional for AWS CodeBuild. However, it's a good practice to set up logging to know what's happening inside your build environment. AWS CodeBuild can upload build logs to CloudWatch of an S3 bucket. For this guide, we'll use CloudWatch.

Creating the buildspec for AWS CodeBuild

After creating the project on AWS CodeBuild, we can set up a buildspec in the source code repository to run our tests. For this guide, the buildspec will set up Artillery and run a load test for the Socket.IO service after pushing new code to the GitHub repository.

Create a buildspec configuration file placed inside the source code repository called buildspec.yml with the following contents:

version: 0.2
 
phases:
  install:
    commands:
      - npm install -g artillery@latest
  build:
    commands:
      - artillery run tests/performance/socket-io.yml

Commit this file to the repository. Once you push the update to GitHub, AWS CodeBuild will execute the load test:

Generating and viewing Artillery test reports

Artillery can output a JSON file with additional details from the load test and use it to generate a self-contained HTML report (opens in a new tab).

First, the job needs to create a directory to place the test reports. Next, you can generate a JSON report when executing the Artillery load test. You can then use the report command (opens in a new tab) to generate the HTML report from the JSON file. Finally, AWS CodeBuild can send the generated files as artifacts to an S3 bucket upon completion of the job.

Before updating the buildspec to generate and save the reports, we need to ensure our AWS CodeBuild project is set up to publish artifacts to S3. In this guide, we'll set up our project to send artifacts to an S3 bucket that belongs to the AWS account, using the service role initially set up with the project.

Once the AWS CodeBuild project handles build artifacts, we can update the buildspec.yml file with the following configuration to generate the Artillery load test report and send it to S3:

version: 0.2
 
phases:
  install:
    commands:
      - npm install -g artillery@latest
  pre_build:
    commands:
      - mkdir reports
  build:
    commands:
      - artillery run --output reports/report.json tests/performance/socket-io.yml
  post_build:
    commands:
      - artillery report --output reports/report reports/report.json
 
artifacts:
  files:
    - "reports/*"
  name: artifacts/$CODEBUILD_BUILD_NUMBER

The configuration now uses the pre_build and post_build phases to handle the steps needed to create a new directory and generate the HTML report. These steps can also be placed in the build phase instead if desired. The artifacts section also uses the $CODEBUILD_BUILD_NUMBER variable as a namespace to better associate the reports with a specific build when sending to the S3 bucket.

After running the build successfully, AWS CodeBuild will send all the generated files to the S3 bucket specified when setting up the artifacts for the project:

For more details on how to configure artifacts in a project build, read the artifacts section (opens in a new tab) in the AWS CodeBuild buildspec reference guide.

Scheduling Artillery load tests

You can also automatically schedule builds to run on AWS CodeBuild with Amazon EventBridge, which is helpful if you want to execute your Artillery load tests at a specific time. For instance, you may wish to load-test your production applications outside of peak hours. You can create an Amazon EventBridge rule to target an AWS CodeBuild project and trigger a new build at a specific schedule.

When choosing a schedule, you can specify either a fixed rate (like every 12 hours) or use a cron expression (opens in a new tab) for more fine-grained control. The cron expression syntax is different from the standard POSIX cron syntax. For instance, the following cron expression sets up the schedule to trigger the build every day at midnight UTC:

You'll need to specify CodeBuild project as the target for the EventBridge rule and set the correct Project ARN (opens in a new tab) for your AWS CodeBuild project.

For more information, read the "Schedule automated builds using AWS CodeBuild" (opens in a new tab) tutorial on the Amazon EventBridge User Guide.

Distributed load testing

You can scale your load tests by using Artillery's AWS Lambda and AWS Fargate integrations to execute your tests.

Once your AWS CodeBuild project has the correct permissions, you can set up your builds to run distributed tests.

The following configuration will install the Artillery CLI and execute the test script in the us-east-1 region:

version: 0.2
 
phases:
  install:
    commands:
      - npm install -g artillery@latest
  build:
    commands:
      - artillery run-fargate --region us-east-1 --count 5 tests/performance/socket-io.yml