Artillery Best Practices

Overview

We designed Artillery with production-grade load testing in mind:

• Load testing frequently, e.g. on automated schedules or as part of CI/CD pipelines
• Load testing at high-scale

Artillery comes with support for best practices baked-in. This guide outlines some of the best practices that we see most of our users adopt to make the most out of load testing.

Fundamentals

Set up automated checks

An Artillery load test can generate many pages worth of metric reports. Whilst all that data is invaluable when you want to investigate the root cause of a performance issue, most of the time we just want to see if everything was OK or not at a glance.

Artillery gives you multiple ways to summarize the results of a load test:

1. Use the Apdex plugin for automated scoring of HTTP API performance
2. Use the ensure plugin to create flexible success/fail conditions for all load tests
3. Advanced: use Artillery's JSON reports to create custom checks

Setting up automated checks frees up your time from inspecting and summarizing load test results manually, and makes it clear when a load test may have identified an issue that needs to be investigated.

Connect monitoring & observability

Artillery integrates with multiple monitoring and observability systems out of the box.

Connecting Artillery to your monitoring system makes it possible to correlate metrics from backend services and APIs with metrics reported by Artillery load tests, which is useful for debugging issues uncovered by load tests.

Put load testing in your CI/CD pipeline

Load testing from CI/CD pipelines helps detect performance regressions and downward trends early on.

There's a couple of ways to run load tests from a CI/CD pipeline:

• As part of a release workflow, where a load test is triggered by deployment events, such as deployment to a pre-production environment
• As part of a release workflow, where a load test serves as a pre-promotion quality gate, such as running a load test on the staging environment before a limited canary roll-out into production
• Running load tests on a schedule, such as running a nightly load test on the staging environment

Learn more about integrating Artillery with your CI/CD system in Artillery in CI/CD guides.

Run distributed tests from the cloud

A realistic load test is a distributed load test that runs from the cloud.

Realistic load test that produce meaningful results do not run from developer machines, or directly from CI/CD nodes. This is a common anti-pattern seen on teams who are early in their load testing adoption journey.

A developer's local machine is great for great for developing test scripts themselves, but for running realistic and meaningful load tests always scale them out horizontally. Artillery comes with distributed load testing at scale built in.

Spike or soak, avoid middle ground

As a rule of thumb, your load tests should fall into one of two groups:

1. Spike tests (also know as flash tests) - these are tests that run for short periods of time (<30m) and ramp up traffic quickly to create a large burst of traffic quickly.

   This type of load testing is especially good for uncovering issues with:

   • autoscaling configurations
   • service startup time issues
   • misconfigurations of service liveness/readiness checks
   • CPU bottlenecks in application code

2. Soak tests - these are tests that run over extended periods of time, typically between 6-12 hours at a time (although longer soak tests of 24 hours or longer are not uncommon either). These tests usually run at no more than 10-20% above baseline load.

   This type of load testing is employed to find lifecycle issues, such as:

   • misconfigured resource pools
   • memory leaks and other types of resource leaks
   • resource saturation issues

Make your application perform an all-out sprint or run a marathon.

Organizing test code

Use config.environments

Use config.environments to create different configuration profiles for different environments or types of load tests.

Separate configuration and scenario definitions

Separate scenarios into individual files and use the --config flag to combine test configs with individual scenarios. This makes reusing scenario definitions in multiple different load test configurations easier.

Keeping scenario definitions in isolated files also makes it easier to re-use those scenarios for end-to-end testing and synthetic monitoring with Skytrace (opens in a new tab).

Use a single repo for all performance tests

We find that it's best to place all load test specs and configurations into the same repo, even though the tests may target different services or sub-systems.

This simplifies both code reuse between different tests, and dependency management (such as Artillery itself, plugins and extensions, and custom code used by the tests).

Advanced

Load test in production

Load testing in production is a practice with a very high payoff when it comes to increasing performance and reliability of production systems. Contrary to popular belief, load testing in production does not have to be risky and can be done safely.

We outline the theory and reasoning behind production load testing, and provide a hands-on guide for going from zero to a production load test in a two-part guide on our blog:

• Part I: https://www.artillery.io/blog/load-testing-in-production (opens in a new tab)
• Part II: https://www.artillery.io/blog/load-testing-in-production-part-2 (opens in a new tab)

Artillery at enterprise scale

Use Artillery Dashboard

We designed the Artillery Dashboard for teams that run load tests continuously and in large volumes to provide a central location for visualizing, reviewing, analyzing and collaborating on performance reports.

→ More information on https://www.artillery.io/cloud (opens in a new tab)

Consider creating a convenience wrapper

We find that teams that offer Artillery as part of an internal developer platform often create a company-specific CLI wrapper around the Artillery CLI (or integrate it into an existing internal CLI) to help with things like:

• Making common team or company-specific tests easier to run (e.g. running a test by a simple short name rather than by specifying paths to config and scenario files).
• Adding a set of tags automatically to every test, such as the environment and the author.
• Integrating Artillery tests with other systems. For example by injecting appropriate secrets from Vault (opens in a new tab) into Artillery tests.