Changelog
Feature updates and improvements to Artillery.
Core & CLI
- Add
--env-file flag as an alternative for --dotenv flag. This makes it consistent with the Node.js --env-file flag. The --dotenv flag will be deprecated in a future release (#3376)
- Add tracking of response times by HTTP status code. A new set of metrics (e.g.
http.response_time.2xx or http.response_time.5xx) is now reported to provide more granular view of response times in a test (#3326)
- Fix an issue that caused incorrect "multiple batches of metics" warnings when running tests with
pause phases (#3331)
Playwright
- Upgrade Playwright to v1.48.0
Azure
- Fix: Make values loaded from an env file with
--dotenv / --env-file flag available to workers containers (rather than just the Artillery process running inside the worker) (#3376)
- Stagger startup of containers in large load tests to prevent rate limit errors from Azure services (#3371)
OpenTelemetry
- Add support for setting resource-level attributes (#3335)
Artillery Cloud
- Send CI related information to Artillery Cloud. This makes the following information available in Artillery Cloud:
- Whether a test run was triggered in CI or not, and which CI service was used
- For tests triggered in GitHub Actions - provide a link back to the job run on GitHub Actions
Dependencies
- Upgrade
json-plus to address a critical security vulnerability (#3369)
Hassy Veldstra
Core & CLI
- New: add ability to set a custom content type for
multipart/form-data form fields #3316 — docs
- Fix: don't print an unnecessary warning when
loadAll is not set by @hassy in #3303
- Remove legacy Artillery Pro integration #3320
Artillery Cloud
- New: add
--name option to set the name of the test to be shown in Artillery Cloud dashboard #3317 — docs
- Improve pre-flight checks when recording reports to Artillery Cloud to detect potential firewall/proxy issues #3314
- Fix: include organization IDs in the test report URLs
Azure
- Fix: bundle separate config files provided with
--config option correctly #3312
- Fix: bundle dotenv files provided with
--dotenv correctly #3313
- Fix: bundle custom
.npmrc and other dotfiles correctly #3312
- Improve error handling and reporting for container provisioning errors #3313
Hassy Veldstra
CLI
- Fix bug preventing custom code using ES modules from loading on Windows (#2662)
- Prevent setting individual CSV rows in context when using
loadAll (#3277)
Fargate
- Add ability to set DNS servers using the
--container-dns-servers flag when running on ECS/EC2 (#3301)
- Add ability to override the ephemeral storage amount for each task via the
--task-ephemeral-storage flag (#3301)
Playwright
- Upgrade Playwright to v1.45.3 (#3294)
Bernardo Guerreiro
CLI
- Group metrics emitted from dynamic URL’s by default
- Fix bug causing Fargate tests to crash when recording to alternate Artillery Cloud URL
Playwright
- Upgrade Playwright to v1.45.2
Bernardo Guerreiro
HTTP
- Add support for file uploads
Playwright
- Upgrade Playwright to v1.45.0
Hassy Veldstra
AWS ECS
- Fix issue preventing running tests on AWS ECS when using an EC2 launch type
- Wait for IAM role for ECS workers to be ready when running tests for the first time
- Add
run:ecs and run-ecs aliases to run-fargate command
CLI
- Update
tough-cookie package to v5 to prevent usage of deprecated package
- Prevent setting the wrong value when using the
--platform flag
Bernardo Guerreiro
AWS Lambda
- AWS Lambda support is now considered stable
- Fix error happening when multiple Lambda tests were created in parallel for the first time
- Remove
--container flag (all run-lambda tests now run using Lambda Container Images)
- Exit with non-zero exit code when expectations fail in AWS Lambda workers
AWS Fargate
- Prevent CLI hanging when workers are terminated early (e.g. due to early termination when running tests on Fargate Spot)
- Exit with non-zero exit code when expectations fail in AWS Fargate workers
Playwright
- Update Playwright version to 1.44.1
CLI
- Make HTTP tests take
strictCapture option into account when set in defaults
Bernardo Guerreiro
AWS Lambda
Lambda Container Images
Artillery will now always use Lambda Container Images to run tests on AWS Lambda. The --container flag introduced in v2.0.12 is no longer required and will be removed in the next release. Creating Lambda functions from .zip archives is no longer supported.
All existing tests should continue running normally without needing any changes. If you run into any issues please open an issue.
This change brings improved startup times for tests running on AWS Lambda and brings AWS Lambda support to parity with AWS Fargate.
Fixes
- Increase startup timeout time for Lambdas running in a VPC
- Exit with non-zero code if a Lambda worker fails
- Fix issue that could lead tests to crash when triggered from a Windows machine
- Expose current worker ID to Lambda workers as
WORKED_ID environment variable
Playwright
- Accept
0 values for defaultNavigationTimeout and defaultTimeout settings
- A warning will be printed if a Playwright
testFunction cannot be found
Monitoring & observability (publish-metrics plugin)
- Add
url attribute to page spans when tracing is enabled for Playwright tests
- Fix issue which could cause Core Web Vitals to not be set as span attributes
- A
test_id attribute is no longer set on all metrics. The built-in $testId variable may be used to set that attribute when needed
- Request timing phases such as DNS lookups and SSL handshake timings are now recorded as attributes instead of spans for HTTP tests
Slack integration
Fix a number of issues that prevented the Slack plugin from working with tests running on AWS Fargate and AWS Lambda
WebSocket
- Errors thrown by hook functions will now be tracked and reported
Hassy Veldstra
AWS Lambda
Artillery can now use Lambda Container Images to run tests on AWS Lambda. Pass the --container flag to run-lambda command to use a container instead of a zip file. No further configuration is required.
This change brings two main benefits:
- Much faster startup times for tests
- Ability to use certain features that were previously only available in tests running on AWS Fargate:
- Run test scripts that TypeScript hooks
- Use third-party dependencies
- Use the built-in publish-metrics plugin
Container-based Lambda functions will become the default option in a future release of Artillery.
Playwright
- Update Playwright to v1.44.0
- Fix tests failing on Fargate with
ENOSPC error with Playwright tracing enabled
- Improve reliability of trace collection:
- Reduce trace collection cooldown period to 1-5m
- Increase the number of concurrent trace recordings to 3 up from 1
- Track the number of collected/discarded recordings with
browser.traces_collected and browser.traces_discarded metrics
CLI & Core
- Make path to the config directory available to test scripts via the special
$dirname variable - docs
- The
--scenario-name flag will now default to using an exact string match, and fall back to using a regex
- Clarify the meaning of
--tags flag
- Remove an unnecessary dependency
Slack integration
Hassy Veldstra
Playwright
- Display errors from the Playwright engine in Artillery reports and the console - docs
- Report errors from failed Playwright assertions - docs
Fargate
- Fix flag allowing custom IAM role in Fargate tasks (
--task-role-name) - docs
CLI & Core
- Log an error message when using an unsupported expectation with the
expect plugin
Bernardo Guerreiro
CLI & Core
- Add
parallel action for making parallel requests in HTTP tests
Fargate
- Add ability to suppress metrics output with
--quiet flag - docs
- Add
eu-north-1 to supported regions
Lambda
- Add ability to suppress metrics output with
--quiet flag - docs
Playwright
- Fix issue preventing stack traces for errors from being displayed
- Update Playwright version to
1.43.1 - docs
Publish Metrics
- Make reporters using OpenTelemetry compatible with parallel requests
Publish Metrics
- Sending traces to AWS X-Ray is now available through the CloudWatch reporter
- Allow replacing of all tracing span names with
replaceSpanNameRegex config option - Docs
- Fix bug in Datadog reporter causing last batches of traces to occasionally drop
- Fix errors happening when
sampleRate tracing option set
- Fix errors in tracing happening when using HTTP streaming
- Display number of spans exported in the Artillery report (
plugins.publish-metrics.spans.exported ) when tracing is enabled
CLI & Core
- Fix incorrectly declared dependencies in Artillery CLI
Artillery Cloud
- Fix issue preventing
ARTILLERY_CLOUD_API_KEY environment variable from being picked up by Fargate when using --dotenv
Metrics by Endpoint
- Report request URLs for requests that resulted in a network error such as
ETIMEOUT or ECONNRESET - docs
CLI and Core
- Fix issue causing some intermediate reports to not be reported
- Fix issue with incorrect formatting of time units in “phase started” messages
CLI and Core
- Add support for loading custom code from ES modules - docs
- Add support for using
async functions as hooks - docs
Playwright
- Update Playwright version to 1.42.1
- Fix bug that prevented traces from Playwright tests from being sent to Datadog and Honeycomb
AWS Fargate
Playwright
- Make
expect from @playwright/test work natively with Typescript support - Documentation
- Use
target as Playwright’s baseURL by default, allowing test scripts to include only relative paths in page navigations (e.g. page.goto('/docs') ) - Documentation
- Update Playwright version to 1.41.2 - Documentation
Artillery Cloud
- Fix missing phase events in Artillery Cloud when running Fargate tests
CLI & Core
- Improve Typescript support to work with more packages, by exposing
config.bundling.external to set packages that can’t be bundled as external - Documentation
- Make test run id available to test runs via
{{ $testId }} in scenarios/config and vuContext.vars.$testId in custom code
- Routine dependency updates
Publish Metrics
- Automatically send all traces and metrics sent via the
opentelemetry reporter with a test_id attribute, making it easier to search for all relevant data from a particular test run
Fargate
- Improve error message and handling for putting CloudWatch retention policy
Artillery will now aim to release a new CLI version every first and third Monday of each month, when possible.
Artillery Cloud
- Fix bug causing only one of the links to worker logs to be displayed in Artillery Dashboard
Playwright
- Update Playwright version used by Artillery to v1.41.0 - Documentation
- Add support for configuring Playwright’s
testIdAttribute setting - Documentation
- Add INP to core web vital metrics reported by Artillery - Documentation
- Enable
headless mode by default when running in Fargate, as headed mode is not compatible with Fargate
Fargate
- Improve Fargate startup logic to prevent bug where workers would hang
- Fix bug where Fargate tests running on Windows would crash trying to resolve paths to dependencies in folders
- Set default bucket lifecycle policy for S3 bucket containing artifacts from test runs - Documentation
- Set default retention policy (180 days) on Cloudwatch Logs generated during runs, with override possible - Documentation
- Add support for running using Fargate Spot by using the
--spot flag - Documentation
Lambda
- Fix Lambda bundle size issue preventing all Lambda tests from running
- Add default bucket lifecycle policy (2-7 days retention) for S3 bucket containing artifacts from test runs - Documentation
CLI & Core
- Fix
$randomString(length) utility to return correct number of characters and allow length larger than 10
Publish Metrics
- Fix bug in Fargate tests where traces sent with OpenTelemetry reporter would not arrive or arrive inconsistently formatted to observability platforms
- Add tracing support for the Datadog, New Relic and Dynatrace reporters using OpenTelemetry
- Improve traces sent via Honeycomb reporter to be proper traces with full metadata, exceptions, events, etc
Fargate
- Enable support for 12 more AWS regions (
eu-west-2, eu-west-3, us-east-2, us-west-2, ca-central-1, ap-east-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, ap-northeast-1, me-south-1, sa-east-1 )
- Add support for sending Playwright and HTTP traces to Datadog APM - Documentation
- Add support for
--variables flag in run-fargate command - Documentation
- Show supported regions when using the
--region flag
CLI & Core
- Add TypeScript support for custom code for tests running locally or on AWS Fargate. This is an experimental feature. - Documentation
Playwright
- VUs will now run in isolated browsers contexts rather than new browser instances by default for better performance - Documentation
- HTTP status codes are now tracked and reported through the
browser.page.codes.<code> metric - Documentation
- Fix bug in
before/ after hooks that prevented the Playwright engine from working - Documentation
- Add experimental TypeScript support. See the docs for current limitations - Documentation
OpenTelemetry
- Fix issue with virtual users not failing when using
test.step() with OpenTelemetry Playwright tracing enabled
- OpenTelemetry tracing can now be used to send HTTP and Playwright engine traces to Datadog in Fargate tests - Documentation
- Add support for smart sampling when using HTTP tracing with the OpenTelemetry reporter
CLI & Core
- Add time unit support for
think steps in scenarios
metrics-by-endpoint plugin is now enabled by default to capture per-URL metrics in HTTP tests. The metrics will be available only in the JSON report and not output to the console, unless the plugin is loaded explicitly by the user.- Configure CPU and memory for Fargate tests with
--cpu and --memory flags
- Fix issue with
$env not resolving correctly in the config section
- Fix an issue with
--scenario-name not picking up the correct scenario
- Add a new
fake-data plugin to allow for realistic test data to be generated in your tests - docs
- Fix handling of metrics with non-alphanumeric characters in
ensure checks
- Improve error handling in the
expect plugin
OpenTelemetry
- Add experimental tracing support for Playwright tests, with tracing of
test.step() calls, page navigation events, and Core Web Vitals metrics
- Improve conformance to OTel semantic conventions:
- Include error messages for spans with HTTP response codes of
400 or higher
- Include
HTTP_REQUEST_CONTENT_LENGTH, HTTP_USER_AGENT and HTTP_FLAVOR attributes
- Fix issue with virtual user ID attribute (
vu.uuid) not always being set correctly
- Fix inconsistencies in error handling
Playwright
- Add experimental tracing support via OpenTelemetry with
publish-metrics plugin
- Fix
FID Core Web Vital metric not being emitted as expected
ECS/Fargate
- Configure CPU and memory for Fargate tests with
--cpu and --memory flags
Socket.IO
- Allow validation of any number of arguments in response/acknowledge with new
args/on options
- More flexible validations with ability to wait for multiple responses
- Fix issue with displaying
match errors in the report
New features & improvements
Engines
playwright: Update Playwright version to latest (1.39.0).playwright: Add a convenience test.step() helper, allowing you to break your tests into steps that emit custom timing metrics - Docs.
CLI/Core
- New CLI flag
--scenario-name, allowing you to specify which scenario to run by its name in the test script - Docs.
- Expose
mean as a metric.
- Include scenario object in all engine hooks - you can now access this under
context.scenario - Docs.
Plugins
publish-metrics - OpenTelemetry
- Traces for HTTP requests are now nested under individual scenarios.
- Include all request metrics in traces: dns_lookup, tcp_handshake, tls_negotiation, request, first_byte, download.
- Include VU errors in traces.
Fixes
Artillery Dashboard
- Ensure all Artillery console logs are sent correctly to Artillery Cloud.
- Fix issue with phase information not being sent correctly.
Engines
socketio: Allow nested options (e.g. namespace) in emit without throwing validation error - Docs.socketio: Allow emit action to be an array of strings - Docs.
CLI/Core
fargate: Fix issue with --packages flag.fargate: Include apdex plugin and playwright engine by default in image, preventing additional installation in workers.- Allow unnamed capture groups to be used in
capture.
- Display meaningful error when engine doesn’t match the engine provided in
config.engine.
Plugins
apdex - Allow frustrated, satisfied and tolerated metrics to be checked with ensure plugin.apdex - Prevent plugin from crashing when using processor functions.publish-metrics - Set headers as metadata for otel-grpc exporter in OpenTelemetry reporter - Docs.publish-metrics - Create endpoint path correctly so it works with managed instances of Dynatrace.publish-metrics: Fix OpenTelemetry reporter debug logging.
New features
- Dynatrace support - the official
publish-metrics plugin can now be configured to send Artillery metrics and events to Dynatrace. Read more in the docs
config.http.defaults - default headers and cookies for all requests in a test may now be set via config.http.defaults
Fixes & improvements
Artillery Dashboard
- Send load phase start/end timings for visualization
- Send CLI exit code to display in test run results
- Fix test run status not always getting reported correctly
- Fix bug which caused test runs on AWS Lambda to not be tagged correctly
Core
- Allow variables set with
--variables to be used anywhere in config
- Fix for encoding of array values in query strings
- Make sure CLI exit handlers are always run
- Fix bug affecting match groups in regexp captures
Plugins
publish-metrics - Exit early if required config is missing for publish-metricspublish-metrics - Reduce memory usage of the pluginexpect - add useRequestNamesOnly optionensure - fix bug where ensure config under config.plugins.ensure would not get picked up
The load test summary view will now calculate and display Apdex scores for load tests that make use the built-in apdex plugin.
Apdex (Application Performance Index) is an open standard for measuring performance of HTTP services. The Apdex score converts response times into a single score that reflects user satisfaction with the service.
Fixes & Improvements
App-wide
- Update colors to improve legibility and consistency
- Add exit animations for all dropdown-like elements
- Improve loading states on all buttons
- Improve consistency of dialogs everywhere
- Improve consistency of spacing between elements across the app
Main view
- Tweak the display of Artillery’s logo in the navbar
- Fix responsiveness on Load Tests view
- Improve UX for Saved Views and the Filter Bar with long input strings
- Match test run ID icon in filter dropdown to one used elsewhere
Load test view
- Make font sizes consistent in metadata panel
- Improve styling of text log view, JSON log view, and table components
- Fix issue with loading default CLI avatar images on notes
- Fix issue with scroll gradient covering first note in the notes list
- Improve spacing on Edit Note dialog
Charts
- Add intelligent X axis scaling depending on duration of the test
- Improve Y axis layout on Load Summary chart
AWS Fargate support
This release adds support for running tests on AWS Fargate. Scale out your load tests horizontally from any of the six supported regions.
artillery run-fargate --count 10 --region us-east-1 my-script.yml
Supported regions:
us-east-1us-west-1eu-west-1eu-central-1ap-south-1ap-northeast-1
AWS Fargate is a serverless solution that requires no infrastructure to be managed, and unlike AWS Lambda does not impose a restriction on the duration of your load tests.
Artillery’s Playwright integration is fully supported on Fargate, to let you run browser-based load tests at scale with ease.
Playwright support
Playwright support is now built into Artillery. The integration in artillery-engine-playwright is now bundled into the Artillery CLI.
- The version of Playwright used by the integration has been updated to v1.3.4
- Web Vitals tracking has been improved to provide more accurate measurements
- Playwright scenario file may now be loaded via
testFunction attribute rather than flowFunction
flow attribute may now be omitted in Playwright engine scenarios
Other fixes & improvements
- Pass
lambdaRoleArn to AWS Lambda engine correctly
- Fix issue with returning matched groups from regexp capture
- Track & report downloaded data in HTTP tests via
http.downloaded_bytes metric
- Environment variables may now be accessed through the special
$env variable (the current way of accessing those via $processEnvironment is still supported but will be deprecated in future)
- Dependency updates to address deprecation and security notices
New features
- AWS Lambda - when running tests on AWS Lambda, Artillery will automatically bundle any plugins loaded via
config.plugins and engines loaded via config.engines
- Checks & assertions: new
notStatusCode check in the expect plugin
- Datadog integration:
includeOnly and excluded options may be used to configure the list of Artillery metrics which will be sent to Datadog
Fixes & improvements
- When
artillery run is used with the --output option, Artillery will check if destination exists before running the test
- Miscellaneous dependency updates to address deprecation warnings and security advisories
We have released Artillery v2.0.0-28, which adds support for AWS CloudWatch, and features a number of performance and stability improvements.
GitHub release page ›
Breaking changes
- AWS Lambda: the default IAM role and policy are now versioned. If you attached any custom permissions to the
artilleryio-default-lambda-role role, you will need to re-attach it to the new artilleryio-default-lambda-role-20230116 role
- CLI: The
probe command has been removed from the CLI. The functionality has been moved to the new Skytrace CLI
New features
- Add support for publishing test metrics to AWS CloudWatch
- HTTP - compressed responses (e.g.
gzip) will be decompressed by default now. (Set gzip: false to disable automatic decompression if required to improve performance.)
- CLI - suggest closest match if a command name is mistyped
AWS Lambda
- Print custom role ARN if provided
- Add versioning to default Lambda role and policy created for running tests
Fixes & improvements
- Fix arrival phases with
arrivalRate and rampTo set to 0
- Improve
rampTo logic to result in more predictable and deterministic ramps
- Fix issue with
maxVusers setting that would lead to unexpected number of VUs to be generated
- Fix wildcard message handling support in Socket.io
- Fix issue with setting some properties on the global
artillery object
AWS Lambda
- Improve test startup logic: workers will wait for each other to be ready before kicking off the test run
- Fix some intermediate metrics not being flushed & reported
- Fix handling of
memory-size and security-group-ids options
Artillery CLI
New HTML reports
We have improved the layout and styling of reports generated with artillery report.
--dotenv flag
You can now load environment variables from a file with the --dotenv flag. Those env vars are available in Artillery scenarios through the $env variable.
Dashboard
Visual polish
- Improved homepage responsiveness
- Updated load test table with new context tags and new flexible tags
- Updated context menus
- Updated header styling in load test view.
- New and improved selected tags filter bar
- Show loading skeletons while data is being fetched
CDK installer
We released an official AWS CDK-based installer for Artillery dashboard.
Artillery AWS CDK installer
Flexible deployment modes
The dashboard can now be deployed in more than one way to fit the exact requirements of your organization:
- As an internal-facing, inside a VPC. This is the default deployment mode, and it will require an existing VPN setup to access the app. TLS is optional with this setup.
- As a publig public-facing, which requires TLS with a pre-provisioned ACM certificate.
- Custom domain may be configured on top of the provisioned ALB DNS name.
GitHub Authentication
We added support for authentication through GitHub.
Artillery Probe
We have launched the first version of Artillery Probe, a smart and user-friendly HTTP client built into Artillery
- Send HTTP requests from the command line
- Send JSON, arbitrary request bodies, set custom headers, querystrings, forms, and more
- See request performance waterfalls
- Syntax highlighting for responses
- Query JSON responses with JMESPath
- Set expectations and checks on the command-line for quick acceptance testing and smoke testing
More details on our blog ›
Artillery Probe is available in the most recent release of Artillery:
npm install -g artillery@latest
Artillery CLI
- Improve validation of test scripts
- Add
wait action to WebSocket scenarios
- Fix an issue with aggregating summary metrics under certain conditions
Detailed changelog for the CLI on GitHub releases page.
Dashboard
- Improve test status indicators
- Fix issue with tooltip positioning on metric charts
- Fix keyboard navigation in tag search dropdown
Website and Docs
- Typography & readability improvements