Distributed load testing on Azure Container Instances

Licensing

Artillery's Azure-specific extensions are dual-licensed under BUSL / commercial license. Running Artillery tests on Azure for non-evaluation, continuous and/or production purposes requires a commercial license.

This guide describes how to run distributed load tests with Artillery on Azure Container Instances (ACI).

You’ll learn:

How to scale out your Artillery tests with Azure Container Instances
How to configure Artillery to run distributed tests on Azure
What Azure resources Artillery creates on your behalf to run your tests

Pre-requisites

Azure support is available in Artillery v2.0.19 or later. Install the latest version with:

npm install -g artillery@latest

How it works

Artillery uses the following Azure services to run load tests:

Azure Container Instances (ACI) - https://azure.microsoft.com/en-us/products/container-instances (opens in a new tab)
Azure Blob Storage (ABS) - https://azure.microsoft.com/en-us/products/storage/blobs (opens in a new tab)
Azure Queue Storage (AQS) - https://azure.microsoft.com/en-us/products/storage/queues (opens in a new tab)

There is no long-running infrastructure that needs to be created to run tests. The Artillery CLI creates all resources it needs on-the-fly and removes them at the end of the test.

To run a test, the Artillery CLI will:

Create an AQS queue. You provide the storage account name where the queue will be created.
Upload test artifacts to ABS (test scripts, JavaScript/TypeScript code, CSV payload files and anything else needed to run the test) . You provide the storage account and blob container that will be used for storing test artifacts.
Create ACI container groups to execute the test. Worker containers will download test artifacts from ABS, and send logs and metrics to the AQS queue. You provide the resource group name in which the container groups are created.
Collect logs and metrics from running container groups and aggregate and print reports
Delete AQS queue, files uploaded to ABS, and ACI container groups when the test is finished

Azure configuration

Artillery needs the following configuration to be able to run tests on Azure on your behalf:

Azure tenant ID and Azure subscription ID
Azure storage account and blob container
Azure resource group
A client ID and client secret for a service principal with the necessary permissions to create and manage resources in your Azure subscription

Azure tenant ID and subscription ID

Refer to https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id (opens in a new tab) for instructions on how to find your Azure tenant ID and subscription ID.

Azure storage account and blob container

Refer to the following guides for instructions on how to create a storage account and a blob container:

Azure resource group

Refer to https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal (opens in a new tab) for instructions on how to create a resource group.

Client ID and client secret

Refer to the following guide for instructions on how to register a new application: https://learn.microsoft.com/en-us/azure/healthcare-apis/register-application (opens in a new tab)
- Save the application ID (client ID) and generate and save a new client secret

Assign permissions to the new application to be able to create and manage resources in your Azure subscription. There are multiple ways to do this, which depend on your organization's policies and practices around IAM policies and security. The following instructions are an example for getting started quickly, and not a recommendation for production use:

Assign built-in "Storage Blob Data Contributor (opens in a new tab)" and "Storage Queue Data Contributor (opens in a new tab)" roles to the new application

Create a new role scoped to the resource group you created earlier, and assign it to the app. Include the following actions in the role definition:

  "actions": [
    "Microsoft.ContainerInstance/register/action",
    "Microsoft.ContainerInstance/containerGroups/read",
    "Microsoft.ContainerInstance/containerGroups/write",
    "Microsoft.ContainerInstance/containerGroups/delete",
    "Microsoft.ContainerInstance/containerGroups/restart/action",
    "Microsoft.ContainerInstance/containerGroups/stop/action",
    "Microsoft.ContainerInstance/containerGroups/refreshDelegatedResourceIdentity/action",
    "Microsoft.ContainerInstance/containerGroups/start/action",
    "Microsoft.ContainerInstance/containerGroups/containers/exec/action",
    "Microsoft.ContainerInstance/containerGroups/containers/attach/action",
    "Microsoft.ContainerInstance/containerGroups/containers/buildlogs/read",
    "Microsoft.ContainerInstance/containerGroups/containers/logs/read",
    "Microsoft.ContainerInstance/containerGroups/detectors/read",
    "Microsoft.ContainerInstance/containerGroups/outboundNetworkDependenciesEndpoints/read",
    "Microsoft.ContainerInstance/containerGroups/providers/Microsoft.Insights/diagnosticSettings/read",
    "Microsoft.ContainerInstance/containerGroups/providers/Microsoft.Insights/diagnosticSettings/write",
    "Microsoft.ContainerInstance/containerGroups/providers/Microsoft.Insights/metricDefinitions/read",
    "Microsoft.ContainerInstance/containerGroups/operationResults/read",
    "Microsoft.ContainerInstance/containerGroupProfiles/read",
    "Microsoft.ContainerInstance/containerGroupProfiles/write",
    "Microsoft.ContainerInstance/containerGroupProfiles/delete",
    "Microsoft.ContainerInstance/containerGroupProfiles/revisions/read",
    "Microsoft.ContainerInstance/containerGroupProfiles/revisions/deregister/action",
    "Microsoft.ContainerInstance/containerScaleSets/read",
    "Microsoft.ContainerInstance/containerScaleSets/write",
    "Microsoft.ContainerInstance/containerScaleSets/delete",
    "Microsoft.ContainerInstance/containerScaleSets/containerGroups/restart/action",
    "Microsoft.ContainerInstance/containerScaleSets/containerGroups/start/action",
    "Microsoft.ContainerInstance/containerScaleSets/containerGroups/stop/action",
    "Microsoft.ContainerInstance/containerScaleSets/containerGroups/delete/action",
    "Microsoft.ContainerInstance/operations/read",
    "Microsoft.ContainerInstance/serviceassociationlinks/delete",
    "Microsoft.ContainerInstance/locations/validateDeleteVirtualNetworkOrSubnets/action",
    "Microsoft.ContainerInstance/locations/deleteVirtualNetworkOrSubnets/action",
    "Microsoft.ContainerInstance/locations/cachedImages/read",
    "Microsoft.ContainerInstance/locations/capabilities/read",
    "Microsoft.ContainerInstance/locations/operationResults/read",
    "Microsoft.ContainerInstance/locations/operations/read",
    "Microsoft.ContainerInstance/locations/usages/read"
]

Run a test on Azure

The following example shows how to run a test on Azure using the configuration created in the previous steps. The test will run with 10 workers in the eastus region.

Client ID and secret values are set via environment variables. Other configuration values - Azure tenant ID, subscription ID, storage account, blob container, and resource group - are passed as command-line arguments.

tenant-abcd-efgh-ijkl-mnop is the placeholder Azure tenant ID
acme-inc-storage-account is the placeholder storage account name
artillery-test-data is a placeholder blob container name
ArtilleryIo is the placeholder resource group name

export AZURE_CLIENT_ID=abcd-efgh-ijkl-mnop
export AZURE_CLIENT_SECRET=abcd-efgh-ijkl-mnop
 
artillery run-aci my-script.yml \
    --tenant-id tenant-abcd-efgh-ijkl-mnop \
    --subscription-id subscription-abcd \
    --storage-account acme-inc-storage-account \
    --blob-container artillery-test-data \
    --resource-group ArtilleryIo \
    --region eastus \
    --count 10

See the reference for the run-aci command for more details on supported flags and options.

Supported regions

Tests can be run from any region that supports Azure Container Instances. The following regions are currently supported:

australiacentral
australiacentral2
australiaeast
australiasoutheast
brazilsouth
canadacentral
canadaeast
centralindia
centralus
eastasia
eastus
eastus2
francecentral
francesouth
germanynorth
germanywestcentral
israelcentral
italynorth
japaneast
japanwest
jioindiawest
koreacentral
koreasouth
mexicocentral
northcentralus
northeurope
norwayeast
norwaywest
polandcentral
qatarcentral
southafricanorth
southafricawest
southcentralus
southeastasia
southindia
spaincentral
swedencentral
switzerlandnorth
switzerlandwest
uaecentral
uaenorth
uksouth
ukwest
westcentralus
westeurope
westindia
westus
westus2

Limits

A single test run is currently limited to 12 hours. This limit will be raised in future.
ACI container groups have a hard limit of 4 CPUs and 16 GB of memory per container group (source (opens in a new tab)). If you need to run tests which are CPU-heavy (such as Playwright-based tests), scale out horizontally by running multiple workers with the --count flag.
If you're planning to run large-scale tests frequently, consider filing a support request with Azure to raise the following limits on your subscription:
1. Container group creates and deletes per 5 minutes
2. Container group creates and deletes per hour
Refer to https://learn.microsoft.com/en-us/azure/container-instances/container-instances-resource-and-quota-limits#changeable-limits-eligible-for-quota-increases (opens in a new tab) for more information on how to request a limit increase.

Licensing

Artillery's Azure-specific extensions are dual-licensed under BUSL / commercial license. Running Artillery tests on Azure in for non-evaluation, continuous and/or production purposes requires a commercial license.

Purchase a license via our self-check out portal on https://pay.artillery.io/b/4gweYDc5vd3Eal27sD (opens in a new tab).

If you have any questions about licensing get in touch on sales@artillery.io.

Questions?

If you have a commercial license, get in touch with us via support@artillery.io
Post your questions on the community forum on GitHub at https://github.com/artilleryio/artillery/discussions (opens in a new tab)
Report a bug or raise an issue via https://github.com/artilleryio/artillery/issues (opens in a new tab)

AWS Fargate Docker Guide