Integrating Artillery with AWS CodeBuild


What you’ll learn

  • How to set up an AWS CodeBuild project to run your Artillery tests
  • How to generate Artillery test reports from AWS CodeBuild
  • How to automatically upload Artillery test reports to Amazon S3
  • How to schedule your Artillery tests to run at a specific time
  • How to use Artillery Pro in an AWS CodeBuild project

Overview

Integrating Artillery with AWS CodeBuild allows you to track your service’s performance at scale while developing your applications on Amazon’s AWS platform. The following guide will show you how to load-test your services using an AWS CodeBuild project.

You can find the complete source code used in this guide in the Artillery CI/CD Examples GitHub repo.

Artillery test script example

In this guide, we’ll use the following Artillery test script to run a load test on a running Socket.IO service, sending 50 virtual users per second for 10 minutes and ensuring the aggregate maximum latency for the test is under 500 milliseconds:

config:
target: "<http://lab.artillery.io>"
phases:
- duration: 600
arrivalRate: 50
ensure:
maxErrorRate: 1
max: 500

scenarios:
- name: "Emit an event"
engine: "socketio"
flow:
- emit:
channel: "echo"
data: "Hello from Artillery"
response:
channel: "echoResponse"
data: "Hello from Artillery"

You can run an example of this test script and see it in action.

Setting up AWS CodeBuild project

This guide assumes you have an existing AWS account with permissions to access the different resources used when setting up AWS CodeBuild projects, such as S3 (for storing and retrieving build artifacts) and CloudWatch (for generating and viewing build logs). For details on the required credentials and permissions, read the “Identity and access management in AWS CodeBuild” section on the AWS CodeBuild User Guide.

We’ll go through the process of setting up a new AWS CodeBuild project on the AWS console. After signing in to the console, go to the CodeBuild service to create a build project. The Create build project page provides many setup options, depending on your needs. The following are the main sections you’ll need to set up for new CodeBuild projects:

Source

AWS CodeBuild can pull in your source code from different providers, such as AWS CodeCommit, an S3 bucket, GitHub, and Bitbucket. Each source provider has different settings, both required and optional. As an example for this guide, we’ll use GitHub as the source provider, using a repository from the connected account. This example also sets up a webhook that listens to any PUSH events in the GitHub repository to trigger a new build.

AWS CodeBuild Build Project Setup - Source

Environment

You can choose different environments for your builds. AWS CodeBuild provides options to use a managed image or a custom Docker image. We’ll use a managed image based on Ubuntu 20.04 (aws/codebuild/standard:5.0 image) for this guide, which contains Node.js version 14 as a runtime, which is needed to install Artillery.

This step also requires setting up a service role for your AWS CodeBuild project. This service role delegates permissions between the different AWS services used for builds, like generating logs in CloudWatch or sending artifacts to S3. You can create a new service role or use an existing one from your AWS account.

AWS CodeBuild Build Project Setup - Environment

Buildspec

The buildspec is the configuration used by AWS CodeBuild to process the different settings and commands in each project build. The configuration file can be read from the source specified in the project setup or created directly in the AWS console. When using the source, CodeBuild looks for a file named buildspec.yml in the source code root directory by default. If needed, you can change the file’s name that AWS CodeBuild will look for in the repo. For this guide, we’ll create a buildspec.yml file in the GitHub repository.

Logs

Setting up build logs is optional for AWS CodeBuild. However, it’s a good practice to set up logging to know what’s happening inside your build environment. AWS CodeBuild can upload build logs to CloudWatch of an S3 bucket. For this guide, we’ll use CloudWatch.

AWS CodeBuild Build Project Setup - Logs

Creating the buildspec for AWS CodeBuild

After creating the project on AWS CodeBuild, we can set up a buildspec in the source code repository to run our tests. For this guide, the buildspec will set up Artillery and run a load test for the Socket.IO service after pushing new code to the GitHub repository.

Create a buildspec configuration file placed inside the source code repository called buildspec.yml with the following contents:

version: 0.2

phases:
install:
commands:
- npm install -g artillery@latest
build:
commands:
- artillery run tests/performance/socket-io.yml

Commit this file to the repository. Once you push the update to GitHub, AWS CodeBuild will execute the load test:

AWS CodeBuild Build Success

Generating and viewing Artillery test reports

Artillery can output a JSON file with additional details from the load test and use it to generate a self-contained HTML report.

First, the job needs to create a directory to place the test reports. Next, you can generate a JSON report when executing the Artillery load test. You can then use the report command to generate the HTML report from the JSON file. Finally, AWS CodeBuild can send the generated files as artifacts to an S3 bucket upon completion of the job.

Before updating the buildspec to generate and save the reports, we need to ensure our AWS CodeBuild project is set up to publish artifacts to S3. In this guide, we’ll set up our project to send artifacts to an S3 bucket that belongs to the AWS account, using the service role initially set up with the project.

AWS CodeBuild Artifacts Setup

Once the AWS CodeBuild project handles build artifacts, we can update the buildspec.yml file with the following configuration to generate the Artillery load test report and send it to S3:

version: 0.2

phases:
install:
commands:
- npm install -g artillery@latest
pre_build:
commands:
- mkdir reports
build:
commands:
- artillery run --output reports/report.json tests/performance/socket-io.yml
post_build:
commands:
- artillery report --output reports/report.html reports/report.json

artifacts:
files:
- 'reports/*'
name: artifacts/$CODEBUILD_BUILD_NUMBER

The configuration now uses the pre_build and post_build phases to handle the steps needed to create a new directory and generate the HTML report. These steps can also be placed in the build phase instead if desired. The artifacts section also uses the $CODEBUILD_BUILD_NUMBER variable as a namespace to better associate the reports with a specific build when sending to the S3 bucket.

After running the build successfully, AWS CodeBuild will send all the generated files to the S3 bucket specified when setting up the artifacts for the project:

AWS CodeBuild - Uploaded artifacts in S3 bucket

For more details on how to configure artifacts in a project build, read the artifacts section in the AWS CodeBuild buildspec reference guide.

Scheduling Artillery load tests

You can also automatically schedule builds to run on AWS CodeBuild with Amazon EventBridge, which is helpful if you want to execute your Artillery load tests at a specific time. For instance, you may wish to load-test your production applications outside of peak hours. You can create an Amazon EventBridge rule to target an AWS CodeBuild project and trigger a new build at a specific schedule.

When choosing a schedule, you can specify either a fixed rate (like every 12 hours) or use a cron expression for more fine-grained control. The cron expression syntax is different from the standard POSIX cron syntax. For instance, the following cron expression sets up the schedule to trigger the build every day at midnight UTC:

AWS CodeBuild - Setting up EventBridge schedule using cron expression

You’ll need to specify CodeBuild project as the target for the EventBridge rule and set the correct Project ARN for your AWS CodeBuild project.

AWS CodeBuild - Setting up EventBridge target

For more information, read the “Schedule automated builds using AWS CodeBuild” tutorial on the Amazon EventBridge User Guide.

Geographically distributed load testing with Artillery Pro

You can scale your load tests by using Artillery Pro to execute your tests. Artillery Pro allows you to run your Artillery tests from different geographic regions within your AWS infrastructure.

To use Artillery Pro on AWS CodeBuild, you must have previously deployed the Artillery Pro backend to your AWS account. You also must provide the correct permissions to allow your CodeBuild project to access the Artillery Pro backend. The simplest way to enable access to your Artillery Pro backend is to attach the role generated by Artillery Pro to the AWS CodeBuild role created with the project.

Once your AWS CodeBuild project has the correct permissions, you can set up your builds to use Artillery Pro. The following configuration will install the Artillery Pro CLI and execute the test script on an Artillery Pro backend deployed in the us-east-1 region:

version: 0.2

phases:
install:
commands:
- npm install -g artillery@latest artillery-pro@latest
build:
commands:
- artillery run-test --cluster artillery-pro-cluster --region us-east-1 --count 5 tests/performance/socket-io.yml

For more options on executing your tests using Artillery Pro, check the Pro Commands section in the Command Line guide.