Artillery

When running Artillery load tests, you'll want to go beyond testing locally or a single instance inside a continuous integration environment. It's essential to have the ability to scale your tests to generate as many requests as you need to ensure your services are stable and reliable. You'll also want to have a more realistic view of real-world usage for your applications by distributing your load test traffic geographically from different areas across the world.

While it's helpful to quickly run an Artillery load test locally, you need to be aware of a few caveats. Running your tests on a single instance won't scale well since the generated load depends on the underlying hardware. Unless you're running your tests from a high-performance system, it isn't easy to spin up the kind of load to match your testing needs. Also, running load tests on limited-performing hardware from only one place limits you from getting a clear picture of how well your services perform when someone accesses them from another region.

You can overcome all of these potential issues with Artillery Pro.

Artillery Pro to the rescue!

Artillery Pro is a layer built on top of the Artillery Core toolkit that allows you to run your load tests on the cloud using your own AWS account. At its core, Artillery Pro leverages the power and flexibility of Amazon AWS with the Amazon Elastic Container Service (ECS) to run your existing Artillery tests. It gives you the ability to scale and parallelize your tests from different geographic regions in a cost-effective manner, all within your AWS infrastructure.

Artillery Pro handles all the complex parts of running realistic and robust load tests while significantly reducing the overhead of time and money. Want to push the limits of your services? Artillery Pro can spin as many workers as you need to send more traffic on-demand. Need to check how your services perform with users on the opposite side of the world? Just run your tests from one of the 13 supported regions. Worried about managing infrastructure or the cost? Artillery Pro can use AWS Fargate, so you don't have to deal with servers while only paying for the resources used during testing.

In this article, you'll see how easy it is to get your existing Artillery load tests set up to use Artillery Pro. We'll show you how to set up your AWS environment and execute your tests from different regions around the world.

Setting up Artillery Pro

To get started with Artillery Pro, you need a license key provided by an active subscription or a trial. You can get started for free by signing up for the Dev plan on the Artillery Pro website. The Dev plan gives you access to essentially all of the Pro features with some limitations, such as running one test at a time, limited to one region and a limited number of workers. If you need to execute simultaneous test runs from different regions with more workers, you'll need a paid plan (Scale or Enterprise). For this article, we'll use Artillery Pro under the Scale plan.

As mentioned earlier, Artillery Pro builds on top of Artillery Core, so you'll need to install both Artillery and Artillery Pro. Provided you have Node.js version 12.0 and above installed, you can install both packages via npm:

npm install -g artillery@latest artillery-pro@latest

Before using Artillery Pro, you'll need to set up your environment with AWS credentials. You can handle this in one of the following ways:

• Use the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables to set up the correct access and secret key associated with an IAM user in your AWS account.
• Install the AWS CLI and configure your credentials using the command-line interface.

Either way will work, but we recommend using the AWS CLI tool to help you attach an AWS user policy to an IAM user to run your Artillery tests.

Once both packages are installed and your AWS credentials are set up, you'll need to set up the Artillery Pro backend in your AWS account. The backend is a fully serverless environment, using services such as DynamoDB, SQS, and S3 to handle the work of running your load tests on your AWS infrastructure without the burden of managing servers.

Artillery Pro handles this setup for you using the artillery deploy command. The command requires two settings: the AWS region where you want to deploy the backend and the license key for your Pro subscription. For instance, if you want to deploy the backend to the us-east-1 region of AWS, use the following command:

artillery deploy artillery deploy \
  --region us-east-1 \
  --license-key-file artillery-pro-license-key.txt

artillery deploy uses AWS CloudFormation to provision the different AWS services and create the necessary roles and policies required to run your tests on the cloud. You'll have the deployment progress shown in your terminal when running the command so you can view the created resources. This command will detect if you previously installed the Artillery Pro backend in your region, so you can safely run it more than once without accidentally deleting an existing setup.

After deploying the backend, you'll get prompted to attach a newly-created policy called artilleryio-cli-user to an existing IAM user in your AWS account. The policy gives an IAM user access to the resources created by Artillery Pro. If you installed the AWS CLI tool in your system, you can run the command shown in your terminal, changing the IAM user name to the one you want to use in your AWS account. You can also manually attach the policy from your browser by logging in to the AWS console and going to the IAM section.

With this initial setup completed, you're now ready to move on to the fun part - running your tests on AWS.

Running your tests from AWS

If you've run an Artillery load test in your local system, running your tests on AWS using Artillery Pro will look familiar. However, instead of creating virtual users and generating traffic from your computer, Artillery Pro will offload these tasks to an AWS ECS cluster with a defined number of workers using either EC2 instances or Fargate. Having your tests run on AWS allows you to run your tests from different regions in the world and scale your tests quickly and easily.

The first step for running Artillery tests on AWS is to create an ECS cluster in your AWS account. Since Artillery Pro supports both classic ECS clusters (with managed EC2 instances) and Fargate, you can create either type. To determine which type of cluster is best for your organization, read "Should you use ECS or Fargate?" in the Artillery documentation.

You can create the cluster from any of the AWS regions supported by Artillery Pro, but keep in mind that you can only run your tests in a single region with the Dev plan. You also need to ensure that the IAM user with the attached artilleryio-cli-user policy has the proper AWS permissions to use the ECS cluster.

Once your cluster is set up, you can now run your tests on AWS using Artillery Pro. For this article, we'll assume that you created an ECS cluster called artillery-pro-cluster in the us-east-1 region and have a test script called my-load-test.yml in your local directory.

To run the test script on AWS, you can use artillery run-test, specifying the cluster (using the --cluster flag), the AWS region (using the --region flag), and the number of workers (using the --count flag) as follows:

artillery run-test \
  --cluster artillery-pro-cluster \
  --region us-east-1 \
  --count 5 \
  my-load-test.yaml

By default, artillery run-test executes your tests in a classic ECS cluster using EC2 instances. If you want to use Fargate instead, you need to specify the cluster type with the --launch-type flag:

artillery run-test \
  --cluster artillery-pro-cluster \
  --region us-east-1 \
  --launch-type ecs:fargate \
  --count 5 \
  my-load-test.yaml

These commands work similarly to executing your load tests with artillery run. However, artillery run-test will use the deployed backend to set up your test run using the defined cluster. Your test script and any dependencies used in the script, such as payload files or custom JavaScript code, get packaged up and sent to the cluster. Your tests will then run the same way as they do locally.

Adding the --count flag lets you leverage the power of AWS to scale up your load tests in an instance. By spinning up additional workers in your cluster, you can generate the load you need without worrying about server management or bandwidth capacity. And since you only pay for what you use on AWS, scaling up your tests with more workers won't cost you an arm and a leg.

Once the test run completes, Artillery Pro will clean up the workers and report your test results in the same way as if you executed your tests on your computer. However, you'll now have better insight into how your services perform from another section of the globe, which you can't observe when running your tests locally. This functionality is a must for today's globally accessible world.

Scaling your Artillery tests up and abroad

The example above shows how simple it is to run your tests from a different region. Seeing how your services perform from another country can give you a glimpse at how your application works for everyone.

With the Scale plan, you can use Artillery Pro to go even further with your load tests by generating traffic from multiple points of the globe simultaneously. Instead of checking how your service behaves with traffic in one place, you can quickly run up to ten tests from different regions concurrently with just a simple command-line change.

For instance, you may have your application hosted in a data center in Europe, and you want to see how your services work with your customers from the United States and Asia. Assuming you have ECS clusters set up in both the us-west-2 (Oregon) and ap-northeast-1 (Tokyo) regions, all you need to do is run your tests on those regions and let Artillery Pro handle the rest.

In one shell, run your test script in the us-west-2 region:

artillery run-test \
  --cluster artillery-pro-cluster \
  --region us-west-2 \
  --launch-type ecs:fargate \
  --count 5 \
  my-load-test.yaml

Then, in another shell, execute the test in the ap-northeast-1 region:

artillery run-test \
  --cluster artillery-pro-cluster \
  --region ap-northeast-1 \
  --launch-type ecs:fargate \
  --count 5 \
  my-load-test.yaml

With only two commands, you now have geographically distributed load testing from across the globe on AWS. It can't get any simpler than that. Running distributed tests like this is especially useful when done from a continuous integration service, ensuring that your services work well for everyone in the world after every change to your application.

One thing to keep in mind is that when you run your tests in a distributed fashion, each test runs independently of the other. While it's helpful to see the results for each region, you may want to aggregate and keep track of all the results. You can use the [artillery-publish-metrics plugin](https://artillery.io/docs/guides/plugins/plugin-publish-metrics.html) to send the results for each test run to an external monitoring and observability system like Datadog or Lightstep.

For even more resilient load testing, you can increase the number of workers for running your tests. By default, Artillery Pro runs your tests on a single worker on your cluster. If you need to increase this capacity, use the --count flag to increase the number of workers you want to provision for your test. For example, the following command executes a test using 10 cluster workers:

artillery run-test \
  --cluster artillery-pro-cluster \
  --region us-east-1 \
  --count 10 \
  my-load-test.yaml

It doesn't get easier to run load tests across the globe than with Artillery Pro

These days, making your applications and web services speedy and resilient isn't a nice-to-have feature - it's vital for any business that caters to users across the world. Your systems must work well for everyone, not just a subset of people nearest to your servers. Without knowing how your application behaves in other regions of the globe, you may be losing customers without realizing it.

Artillery Pro will help you ensure performance and reliability for your services by taking advantage of your AWS infrastructure to run globally distributed load tests. With just a few commands on top of the Artillery CLI, you can execute your existing Artillery test scripts on different ECS clusters located in distinct regions. You can also scale up your testing traffic as needed by provisioning additional workers.

The best part about it is that your team won't have to spend any time dealing with testing infrastructure or worrying about costs. Everything you need for testing will get set up on your own cloud environment. With Artillery Pro, you won't find an easier way to run load tests at scale worldwide.