CloudWatch

This reporter supports sending metrics and traces to CloudWatch (opens in a new tab).

In order to enable the Cloudwatch reporter set type to cloudwatch, and region to the AWS region that you want metrics to be sent to:

config:
  plugins:
    publish-metrics:
      - type: cloudwatch
        region: eu-west-1

Important: AWS credentials have to be present in the environment and have sufficient IAM permissions to send metrics to Cloudwatch and/or traces to AWS X-Ray.

Metrics

By default, all Artillery metrics will be sent to CloudWatch. Each Artillery metric will create a custom CloudWatch metric (opens in a new tab), which incur extra AWS fee for each metric. You can configure a specific list of metrics to send with the includeOnly setting (see Configuration section below).

You can learn more about AWS CloudWatch custom metric pricing in this blog post from Vantage.sh: https://www.vantage.sh/blog/cloudwatch-metrics-pricing-explained-in-plain-english (opens in a new tab)

Configuration options

Example configuration

The following configuration enables CloudWatch reporting. All metrics generated by the test will be sent to us-east-1 region, and tagged with extra dimensions to specify the team running the tests (SQA), and the name of the service under test (checkout-svc).

config:
  plugins:
    publish-metrics:
      - type: cloudwatch
        region: us-east-1
        namespace: continous-testing
        dimensions:
          - name: Team
            value: SQA
          - name: Service
            value: checkout-svc

Name	Description
`region`	CloudWatch region where metrics will be sent.
`namespace`	Metric namespace. Defaults to "artillery".
`name`	The value of the default `Name` dimension attached to every metric. Defaults to "loadtest".
`excluded`	A list of metric names which should not be sent to CloudWatch. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.
`includeOnly`	A list of specific metrics to send to CloudWatch. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.
`dimensions`	A list of extra dimensions to attache to every metric as `name`/`value` pairs. Defaults to an empty list.
`traces` Added inv2.0.9	Set to send traces to AWS X-Ray. See traces specific configuration

Traces

Added inv2.0.9

Powered by: OpenTelemetry reporter

⚠️

Experimental This feature is under active development. We aim to keep it stable, but it may need to introduce breaking changes. Please open an issue (opens in a new tab) or discussion (opens in a new tab) for any feedback.

With the CloudWatch reporter you can also send traces to AWS X-Ray (opens in a new tab). Tracing is available for both HTTP and Playwright engines, and the tracing data available is different depending on the engine used.

An additional plugins.publish-metrics.spans.exported metric will be recorded and will appear in your reports when tracing is enabled. It represents the number of spans exported to AWS X-Ray.

Configuration options

CloudWatch requires a running instance of the CloudWatch Agent (opens in a new tab) (or the AWS Distro for OpenTelemetry (opens in a new tab)) in order to ingest traces. When you run a test on AWS Fargate, Artillery will take care of this for you automatically, but for testing on your local machine you will need to run the agent yourself.
Note: AWS Lambda is not currently supported.

Example configuration

  plugins:
    publish-metrics:
      - type: "cloudwatch"
        traces:
          sampleRate: 0.5
          replaceSpanNameRegex:
            - pattern: get_user_[a-zA-Z0-9]+
              as: get_user_id
          annotations:
            testType: LoadTest
            tool: Artillery

Name	Description
`serviceName`	Name of your service. Defaults to `Artillery-test`
`sampleRate`	Sample rate. Percentage of traces to send represented by a value between 0 and 1. (defaults to 1 - all traces are sent)
`useRequestNames`	If set to `true` the request `name`s provided in test script will be used as span names
`annotations`	Custom annotations to be added to each span in `key: value` pairs.
`replaceSpanNameRegex`	A list of replacement maps that consist of: `pattern` - regex pattern to match against the span names `as` - string to replace the matched pattern in the span names

AWS Fargate - additional setup

Artillery sets up the AWS Distro for OpenTelemetry (opens in a new tab) configured with the AWS X-Ray exporter as a sidecar container.

Important: If you ran Artillery tests on AWS Fargate in the past on the account you are using, you will need to delete the existing artillery-worker-role IAM role and artilleryio-ecs-worker-policy IAM policy from your account prior to using this feature.
Artillery will automatically recreate the role and policy with the correct permissions to send traces to AWS X-Ray.

To run the test run:

artillery run-fargate scenario.yaml

Local runs - additional setup

You'll need to install and configure a CloudWatch Agent (opens in a new tab) or AWS Distro for OpenTelemetry (opens in a new tab) to ingest OTLP traces:

CloudWatch Agent

You can find the installation instructions for the CloudWatch Agent here (opens in a new tab), and for the configuration here (opens in a new tab).
The agent's IAM role or IAM user must have the AWSXrayWriteOnlyAccess policy to send trace data to AWS X-Ray. For more information, see Create IAM roles and users for use with CloudWatch agent (opens in a new tab)].
To configure the agent to receive traces from Artillery set up the otlp receiver in the CloudWatch Agent configuration file as follows:

"traces": {
  "traces_collected": {
    "otlp": {}
  }
}

AWS Distro for OpenTelemetry

You can find the installation and configuration instructions here (opens in a new tab).

Debugging

Set DEBUG=plugin:publish-metrics:cloudwatch when running your tests to print out helpful debugging messages when sending metrics to Datadog.

DEBUG=plugin:publish-metrics:cloudwatch artillery run my-script.yaml

Examples

config:
  plugins:
    publish-metrics:
      - type: cloudwatch
        region: us-east-1
        namespace: continous-testing
        dimensions:
          - name: Team
            value: SQA
        traces:
          serviceName: my-service
          useRequestNames: true
          annotations:
            testType: LoadTest
            tool: Artillery

Datadog New Relic