Distributed Load Testing on AWS Fargate

Overview

This guide describes how to run high-scale distributed load tests with Artillery on AWS Fargate.

AWS credentials

To execute tests on AWS Fargate the Artillery CLI makes use of the official AWS SDK (opens in a new tab) to create the resources needed to run your tests (see the AWS Resources seciton for details on what Artillery creates).

The SDK requires AWS credentials (opens in a new tab) to be present to work. Please refer to the official AWS documentation if you don't have one set up already: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html (opens in a new tab)

Please see the IAM Permissions section for details on permissions required to run tests from AWS Fargate.

Running tests from AWS Fargate

To run an existing test script from AWS Fargate, use run-fargate command instead of the run command.

For example, if you have a test script saved in blitz.yml and you want to run it from eu-west-1 region, run the following command:

artillery run-fargate \
   --region eu-west-1 \
   blitz.yml

How it works

When you run a load test with AWS Fargate, Artillery tries to make it as seamless as possible.

AWS resources created

Artillery will create a number of AWS resources behind the scenes to be able to execute your tests. All resources created by Artillery are serverless and created on-demand. There are no long-running infrastructure components involved.

• An S3 bucket to store test bundles
• An SQS queue for communication between Fargate containers executing your test and the Artillery CLI. This queue is deleted once the test run completes.
• (Optional) An IAM role named artilleryio-ecs-worker-role for Fargate tasks that execute the test. If a role with that name already exists, Artillery will use it instead of creating it.

IAM permissions

The AWS profile that the Artillery CLI runs under needs to have sufficient permissions to be able to create the resources listed above.

If running in a sandbox/developer account, the easiest way is to run with admin privileges using the default AWS AdministratorAccess (opens in a new tab) managed policy.

An definition of IAM policy to use instead of AdministratorAccess is below. The AWS IAM user you're using will need permissions to assume a role which makes use of the policy created with this template.

Note that 123456789000 will need to be replaced with the id of the AWS account you'll be using.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CreateOrGetECSRole",
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:GetRole",
                "iam:AttachRolePolicy"
            ],
            "Resource": "arn:aws:iam::123456789000:role/artilleryio-ecs-worker-role"
        },
        {
            "Sid": "CreateECSPolicy",
            "Effect": "Allow",
            "Action": [
                "iam:CreatePolicy"
            ],
            "Resource": "arn:aws:iam::123456789000:policy/artilleryio-ecs-worker-policy"
        },
        // Allow Artillery CLI to create AWS service role for ECS when creating a Fargate cluster
        // https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using-service-linked-roles.html#create-service-linked-role
        {
          "Effect": "Allow",
          "Action": ["iam:CreateServiceLinkedRole"],
          "Resource": [
            "arn:aws:iam::*:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS*"
          ],
          "Condition": {
            "StringLike": {
              "iam:AWSServiceName": "ecs.amazonaws.com"
            }
          }
        },
        {
          "Effect": "Allow",
          "Action": ["iam:PassRole"],
          "Resource": ["arn:aws:iam::123456789000:role/artilleryio-ecs-worker-role"]
        },
        {
            "Sid": "SQSPermissions",
            "Effect": "Allow",
            "Action": [
                "sqs:*"
            ],
            "Resource": "arn:aws:sqs:*:123456789000:artilleryio*"
        },
        {
            // ListQueues cannot be scoped to individual resources
            // https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonsqs.html#amazonsqs-queue
            "Sid": "SQSListQueues",
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues"
            ],
            "Resource": "*"
        },
        {
            "Sid": "ECSPermissionsGeneral",
            "Effect": "Allow",
            "Action": [
                "ecs:ListClusters",
                "ecs:CreateCluster",
                "ecs:RegisterTaskDefinition",
                "ecs:DeregisterTaskDefinition"
            ],
            "Resource": "*"
        },
        {
            "Sid": "ECSPermissionsScopedToCluster",
            "Effect": "Allow",
            "Action": [
                "ecs:DescribeClusters",
                "ecs:ListContainerInstances"
            ],
            "Resource": "arn:aws:ecs:*:123456789000:cluster/*"
        },
        {
            "Sid": "ECSPermissionsScopedWithCondition",
            "Effect": "Allow",
            "Action": [
                "ecs:SubmitTaskStateChange",
                "ecs:DescribeTasks",
                "ecs:ListTasks",
                "ecs:ListTaskDefinitions",
                "ecs:DescribeTaskDefinition",
                "ecs:StartTask",
                "ecs:StopTask",
                "ecs:RunTask"
            ],
            "Condition": {
                "ArnEquals": {
                    "ecs:cluster": "arn:aws:ecs:*:123456789000:cluster/*"
                }
            },
            "Resource": "*"
        },
        {
            "Sid": "S3Permissions",
            "Effect": "Allow",
            "Action": [
              "s3:CreateBucket",
              "s3:DeleteObject",
              "s3:GetObject",
              "s3:GetObjectAcl",
              "s3:GetObjectTagging",
              "s3:GetObjectVersion",
              "s3:PutObject",
              "s3:PutObjectAcl",
              "s3:ListBucket",
              "s3:GetBucketLocation",
              "s3:GetBucketLogging",
              "s3:GetBucketPolicy",
              "s3:GetBucketTagging",
              "s3:PutBucketPolicy",
              "s3:PutBucketTagging",
              "s3:PutMetricsConfiguration",
              "s3:GetLifecycleConfiguration",
              "s3:PutLifecycleConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::artilleryio-test-data-*",
                "arn:aws:s3:::artilleryio-test-data-*/*"
            ]
        },
        {
            "Sid": "LogsPermissions",
            "Effect": "Allow",
            "Action": [
                "logs:PutRetentionPolicy"
            ],
            "Resource": [
                "arn:aws:logs:*:123456789000:log-group:artilleryio-log-group/*"
            ]
        },
        {
          "Effect": "Allow",
          "Action": ["secretsmanager:GetSecretValue"],
          "Resource": ["arn:aws:secretsmanager:*:123456789000:secret:artilleryio/*"]
        },
        {
          "Effect": "Allow",
          "Action": [
            "ssm:PutParameter",
            "ssm:GetParameter",
            "ssm:GetParameters",
            "ssm:DeleteParameter",
            "ssm:DescribeParameters",
            "ssm:GetParametersByPath"
          ],
          "Resource": [
            "arn:aws:ssm:us-east-1:123456789000:parameter/artilleryio/*",
            "arn:aws:ssm:us-west-1:123456789000:parameter/artilleryio/*",
            "arn:aws:ssm:eu-west-1:123456789000:parameter/artilleryio/*",
            "arn:aws:ssm:eu-central-1:123456789000:parameter/artilleryio/*",
            "arn:aws:ssm:ap-south-1:123456789000:parameter/artilleryio/*",
            "arn:aws:ssm:ap-northeast-1:123456789000:parameter/artilleryio/*"
          ]
        },
        {
          "Effect": "Allow",
          "Action": [
            "ec2:DescribeRouteTables",
            "ec2:DescribeVpcs",
            "ec2:DescribeSubnets"
          ],
          "Resource": ["*"]
        }
    ]
}

Setting up the IAM Policy and Role from AWS

To help you get started, here's a small guide to set up the necessary role and policy in AWS. It's especially important to have this created if you're planning on using Artillery in CI/CD.

IAM Policy: You should first create the policy. You can do that from the AWS UI or CLI. To do it from the UI:

1. Go to IAM -> Policies and select Create Policy. Select JSON.
2. Copy the JSON policy from the section above. You’ll need to change the following:
   • Change the AWS account from 123456789000 to your account ID;
   • Remove any comments from the JSON policy, as that is not allowed;
3. Press Next and give the policy a meaningful name.

IAM Role: After the policy created, you'll need to use that policy in a role:

1. Go to IAM -> Roles and select Create Role;
2. Select Custom Trust Policy. You can keep the default generated or customise it based on your organisational needs;
3. After, on the Add Permissions UI, select the policy created in the previous step;
4. Finally, press Next and give the role a meaningful name.

That's it! You should now be able to use this role to run Artillery tests in Fargate.

Accessing worker logs in Cloudwatch

Artillery automatically outputs the test run metrics to your console while the test runs. However, if you run into unexpected errors in the worker containers where Artillery runs, you may need to access the logs in Cloudwatch to debug this.

Via Artillery Cloud

If your test ran using Artillery Cloud, direct links to worker logs will be available inside the test report by clicking Logs -> Worker logs.

Via ECS (Fargate) UI

This is the simplest way to access the logs if not using Artillery Cloud. However, since Stopped tasks only show in ECS for 1 hour, you'll only be able to see the logs from this UI if the test has run in the last hour. Otherwise, you'll need to use the Cloudwatch UI.

1. Go to AWS ECS (Elastic Container Service) in the region you used (default is us-east-1), and select the cluster that was created for the test (artilleryio-cluster by default);
2. Select the Tasks tab and change Filter desired status to Stopped if the task has already stopped/failed. Otherwise, leave it as Running;
3. Select the task that you want to see the logs for;
4. Click the Logs tab and then click the View in Cloudwatch dropdown, selecting the first one. The first one should contain logs for the artillery container, and the second one logs for the datadog-agent container that runs alongside it.
5. That should take you straight to the relevant Cloudwatch log stream for that task.

Note: Logs may not be immediately available when you start the test. The workers need to be running before you can access logs.

Via Cloudwatch UI

1. Go to Cloudwatch in AWS, in the region you're using. Then click Log groups.
2. Search for the Artillery log group (artilleryio-log-group/artilleryio-cluster) and click it.
3. You'll now be in the Log Streams tab, which will be ordered by Last event time.
4. The streams you need will be in the format artilleryio/<testID>/artillery/<taskID>. If you search by the testID (it's outputted when you ran the test), it should show you 2 streams per worker (so you would see 2 with the default --count of 1).
5. You'll need to select the one that has artillery in the name (not datadog).

Note: Logs may not be immediately available when you start the test. The workers need to be running before you can access logs.

Retention Settings

S3 Lifecycle Configuration

Objects created in Artillery's s3 bucket in your AWS account have the following lifecycle rules by default:

• Test Run artifacts (e.g. custom npm dependencies): 2 days
• Test Run metadata: 7 days

Cloudwatch Retention Policy

Cloudwatch logs (under the artilleryio-log-group) are retained within your AWS account for 180 days. You can configure this value with the ARTILLERY_LOGGROUP_RETENTION_DAYS environment variable, making sure to use a valid number (opens in a new tab).

Troubleshooting

Unable to assume the service linked role

`InvalidParameterException`: Unable to assume the service linked role. Please verify that the ECS service linked role exists

You may get the error above if, in some situations, your AdministratorAccess role may be misconfigured from AWS. This seems more likely to happen when no ECS cluster has been created before in that AWS account.

To resolve this, try to run the ECS wizard to create a Fargate cluster manually from the AWS console (by going to ECS in the console). Doing so seems to do something internally in AWS to resolve the misconfiguration.

Could not find public subnets in default VPC

If you're getting this error from the CLI, then it's likely that there isn't a default VPC or that the default VPC doesn't have public subnets in the region you're trying to run Fargate tests from.

AWS Regions should come with a default VPC, but if for some reason you don't have it, you can create one from the AWS console yourself (opens in a new tab). Artillery needs it to be there for Fargate to work.

Questions?

• Post your questions on the community forum on GitHub at https://github.com/artilleryio/artillery/discussions (opens in a new tab)
• Report a bug or raise an issue via https://github.com/artilleryio/artillery/issues (opens in a new tab)