Reference
Engines
Playwright

Playwright engine

Built-in sincev2.0.0-33

Overview

Artillery's Playwright engine is the glue between Artillery & Playwright. The engine takes care of setting up headless browsers, running your Playwright test code, and collecting and emitting performance metrics.

This page provides a reference for all configuration options of Artillery's Playwright integration. You can also learn more about load testing with Playwright in the detailed guide for using Playwright at scale.

Usage

The Playwright engine is built into Artillery and does not need to be installed separately.

Artillery config - hello-world.yml

Create a new Artillery test definition in hello-world.yml:

config:
  target: https://www.artillery.io
  # Load the Playwright engine:
  engines:
    playwright: {}
  # Path to JavaScript file that defines Playwright test functions
  # TypeScript is also supported
  processor: './flows.js'
scenarios:
  - engine: playwright
    testFunction: helloWorld

Playwright test functions - flows.js

Create a test function in flows.js:

module.exports = { helloWorld };
 
async function helloWorld(page) {
  // The "page" argument is instance of Playwright Page class:
  // https://playwright.dev/docs/api/class-page/
 
  // Go to https://artillery.io/
  await page.goto('https://www.artillery.io/');
  // Click text=Docs
  await page.click('text=Docs');
}

Run your test

artillery run hello-world.yml

Artillery automatically records front-end performance metrics for perceived load speed (opens in a new tab) such as LCP and FCP.

--------------------------------
Summary report @ 11:24:53(+0100)
--------------------------------

vusers.created.total: ....................................... 1
vusers.completed: ........................................... 1
vusers.session_length:
  min: ...................................................... 5911.7
  max: ...................................................... 5911.7
  mean: ..................................................... 5911.7
  median: ................................................... 5944.6
  p95: ...................................................... 5944.6
  p99: ...................................................... 5944.6
browser.page.FCP.https://artillery.io/:
  min: ...................................................... 1521.1
  max: ...................................................... 1521.1
  mean: ..................................................... 1521.1
  median: ................................................... 1525.7
  p95: ...................................................... 1525.7
  p99: ...................................................... 1525.7
browser.page.LCP.https://artillery.io/:
  min: ...................................................... 1521.1
  max: ...................................................... 1521.1
  mean: ..................................................... 1521.1
  median: ................................................... 1525.7
  p95: ...................................................... 1525.7
  p99: ...................................................... 1525.7
browser.page.FCP.https://artillery.io/docs/:
  min: ...................................................... 205.3
  max: ...................................................... 205.3
  mean: ..................................................... 205.3
  median: ................................................... 206.5
  p95: ...................................................... 206.5
  p99: ...................................................... 206.5
browser.page.LCP.https://artillery.io/docs/:
  min: ...................................................... 205.3
  max: ...................................................... 205.3
  median: ................................................... 206.5
  p95: ...................................................... 206.5
  p99: ...................................................... 206.5

Configuration

Playwright / browser configuration options

NameValid OptionsDescription
launchOptionsPlaywright's browserType.launch() (opens in a new tab) optionsConfigure the browser instance started by Playwright
contextOptionsPlaywright's browser.newContext() (opens in a new tab) optionsConfigure browser contexts created for each virtual user
defaultNavigationTimeoutNumber (in seconds)Shorthand for setting setDefaultNavigationTimeout() (opens in a new tab) and setDefaultTimeout() (opens in a new tab)
extendedMetrics
  • false (default)
  • true
Report additional metrics.
aggregateByName
  • false (default)
  • true
Group metrics by scenario name rather than by URL.
showAllPageMetrics
  • false (default)
  • true
Send Web Vital metrics for all pages. By default, Artillery only displays Web Vital metrics for a URL that starts with the config.target URL. This avoids reporting metrics for third-party pages and iframes.
useSeparateBrowserPerVU
Added inv2.0.4
  • false (default)
  • true
Use a separate browser for each virtual user (instead of a new browser context (opens in a new tab)). This will require a lot more CPU and memory and is not recommended for most tests.
testIdAttribute
Added inv2.0.5		StringThe attribute used by locator page.getByTestId (opens in a new tab) in Playwright.
traceObject (see below)Configuration for recording Playwright Traces

Tracing configuration

Artillery can automatically record Playwright traces for virtual users that fail. This can be helpful for debugging any issues uncovered by the load test. Recorded traces are uploaded to Artillery Cloud (opens in a new tab) and can be viewed under the Traces tab in the test report.

You will need to create an Artillery Cloud account and run your test with --record flag to use this feature.

NameValid OptionsDescription
enabled
  • false (default)
  • true
Enable or disable recording of Playwright traces
maxConcurrentRecordingsNumber (default: 3)The number of virtual users that will be recording a trace at the same time

For performance reasons, only a subset of active virtual users record traces at any given time. If a virtual user succeeds, the trace is discarded. This means that some tests with only a small number of failing virtual users may not produce any traces. You can tweak the maxConcurrentRecordings setting to increase the likelihood of capturing traces for failing virtual users.

Customizing maxConcurrentRecordings

You can increase the value of maxConcurrentRecordings to increase the likelihood of capturing traces for failing VUs.

In distributed tests this setting is applied per worker, e.g. if you run a distributed load test on Azure ACI with 20 workers like this:

artillery run-aci hello-world.yml --count 20

Each of those 20 workers will be recording up to maxConcurrentRecordings traces at a time.

See the load testing with Playwright guide for more information on how trace recordings work.

$rewriteMetricName hook

It's possible to customize the names of metrics emitted by Playwright load tests by defining a $rewriteMetricName hook function.

This can be used to group metrics from distinct URLs & pages requested during the test under a single metric name in the report. A common use-case is to report metrics for a page that's loaded with many different query parameters under a single metric name.

If your custom code (loaded with config.processor) defines a $rewriteMetricName function, it will be called for each metric emitted by the Playwright engine.

The function should return the new metric name as a string.

Example of a $rewriteMetricName function that groups metrics by the first part of the URL:

function $rewriteMetricName(metricName, metricType) {
  // metricType is one of: histogram, counter, rate
  // For more details on Artillery metric types, see: https://www.artillery.io/docs/reference/reported-metrics
  if (metricName.includes('/checkout?promoid=')) {
    return 'browser.page.checkout';
  } else {
    return metricName;
  }
}

config.target is Playwright's baseURL

Added inv2.0.6

The config.target is automatically set as the baseURL (opens in a new tab) for the Playwright test. This means that you can use relative URLs (e.g. page.goto('/docs')) in your Playwright scripts, and they will be resolved relative to the config.target URL.

If you are not using relative URLs in your test script, full URLs will still work as usual.

Test function API

Page argument

By default, only the page argument (see Playwright's page API (opens in a new tab)) is required for functions that implement Playwright scenarios, e.g.:

module.exports = { helloFlow };
 
async function helloFlow(page) {
  // Go to https://artillery.io/
  await page.goto('https://artillery.io/');
}

Virtual user context and events arguments

The functions also have access to the virtual user context, which can be used for several purposes:

  • Accessing scenario (and environment) variables for different virtual users (vuContext.vars);
  • Getting the current virtual user ID (vuContext.vars.$uuid);
  • Getting the scenario definition for the scenario currently being run by the virtual user (vuContext.scenario), including its name.

Additionally, the events argument can be used to track custom metrics.

module.exports = { helloFlow };
 
async function helloFlow(page, vuContext, events) {
  // Increment custom counter:
  events.emit('counter', `user.${vuContext.scenario.name}.page_loads`, 1);
  // Go to https://artillery.io/
  await page.goto('https://artillery.io/');
}

test.step argument

Added inv2.0.0-38

The final argument of the function is test, which contains the step property. The API for test.step is similar to Playwright's own test.step, which allows you to re-use similar code. The purpose in Artillery is slightly different: to emit custom metrics that represent how long each step takes.

async function loginSearchAndLogout(page, vuContext, events, test) {
  //1. add this line to your scenario function to use test.step helper:
  const { step } = test;
 
  const userid = vuContext.vars.userid;
  const recordid = vuContext.vars.recordid;
 
  //2. wrap any logic you have in steps (sometimes you might already have something like this done from existing Playwright tests)
  await step('landing_page', async () => {
    await page.goto('https://internaltesturl.com/landing');
  });
 
  await step('submit_login', async () => {
    await page.getByLabel('id-label').fill(`${userid}`);
    await page.getByLabel('Password').fill(`${password}`);
    await page.getByRole('button', { name: 'Submit' }).click();
  });
 
  await step('search_record', async () => {
    await page.getByPlaceholder('enter request id').fill(`${recordid}`);
    await page.getByRole('button', { name: 'Go' }).click();
    await page.locator('css=button.avatar-button').click();
  });
 
  await step('logout', async () => {
    await page.getByText('Logout').click();
  });
}

The above code will now emit custom metrics for each step in addition to the default metrics:

browser.step.landing_page:
  min: ......................................................................... 87
  max: ......................................................................... 150
  mean: ........................................................................ 118.5
  median: ...................................................................... 89.1
  p95: ......................................................................... 89.1
  p99: ......................................................................... 89.1
browser.step.submit_login:
  min: ......................................................................... 300
  max: ......................................................................... 716
  mean: ........................................................................ 571.6
  median: ...................................................................... 561.2
  p95: ......................................................................... 561.2
  p99: ......................................................................... 561.2
browser.step.search_record:
  min: ......................................................................... 287
  max: ......................................................................... 801
  mean: ........................................................................ 544.6
  median: ...................................................................... 290.1
  p95: ......................................................................... 290.1
  p99: ......................................................................... 290.1
browser.step.logout:
  min: ......................................................................... 52
  max: ......................................................................... 334
  mean: ........................................................................ 193.1
  median: ...................................................................... 140.2
  p95: ......................................................................... 200.4
  p99: ......................................................................... 200.4

Metrics reported by the engine

In addition to the default metrics reported by Artillery, the Playwright engine reports the following metrics:

MetricTypeDescription
browser.http_requestsCounter

(count)

Number of HTTP requests made by all virtual users during this time period.
browser.page.codes.<code>
Added inv2.0.4		Counter

(count)

Number of different HTTP status codes, e.g. browser.page.codes.200 is the number of 200 OK requests.
errors.pw_failed_assertion.<assertion_type>
Added inv2.0.11		Counter

(count)

When available, Artillery will display the name of failed assertions (e.g. toBeVisible). Defaults to errors.<error.message> if not possible to parse the assertion error.
browser.page.TTFB.<page_url>.<aggregation>Histogram

(milliseconds)

Time To First Byte (opens in a new tab) (Web Vital metric) measurement for a specific page_url.
browser.page.FCP.<page_url>.<aggregation>Histogram

(milliseconds)

First Contentful Paint (opens in a new tab) (Web Vital metric) measurement for a specific page_url.
browser.page.LCP.<page_url>.<aggregation>Histogram

(milliseconds)

Largest Contentful Paint (opens in a new tab) (Core Web Vital metric) measurement for a specific page_url.
browser.page.FID.<page_url>.<aggregation>Histogram

(milliseconds)

First Input Delay (opens in a new tab) (Core Web Vital metric) measurement for a specific page_url (if available).
browser.page.INP.<page_url>.<aggregation>
Added inv2.0.5		Histogram

(milliseconds)

Interaction to Next Paint (opens in a new tab) (Core Web Vital metric) measurement for a specific page_url (if available).
browser.page.CLS.<page_url>.<aggregation>Histogram

(shift score)

Cumulative Layout Shift (opens in a new tab) (Core Web Vital metric) measurement for a specific page_url (if available).

Extended metrics

If extendedMetrics is enabled, the following metrics are also reported:

MetricTypeDescription
browser.page.domcontentloadedCounter

(count)

Number of DOM Content Loaded (opens in a new tab) events across all pages.
browser.page.domcontentloaded.<page_url>Counter

(count)

Number of DOM Content Loaded (opens in a new tab) events for a specific page_url.
browser.page.dominteractive.<aggregation>Histogram

(milliseconds)

Measurement of time taken for DOM to become interactive (opens in a new tab), across all pages.
browser.page.dominteractive.<page_url>.<aggregation>Histogram

(milliseconds)

Measurement of time taken for DOM to become interactive (opens in a new tab), for a specific page_url.
browser.memory_used_mb.<aggregation>Histogram

(megabytes)

Measurement of usedJSHeapSize.

If test.step() API is used, the following additional histogram is reported:

  • browser.step.<step_name>.<aggregation> (milliseconds) - measurement of time taken for each step in the scenario.

Auto-instrumentation of Web Vitals

Artillery automatically tracks the following Web Vitals metrics for every page load in your test:

The metrics are tracked using Google's web-vitals (opens in a new tab) library. Artillery injects the library into the context of each page and Page event handlers (opens in a new tab) are set up to collect those metrics.

⚠️

If you have any code that modifies the Page object provided as the first argument to the test function, you need to make sure that it doesn't overwrite the page context, or reset event handlers. If either of those is reset, then automatic collection of Web Vitals metrics won't work.

Examples

Example 1: turn off headless mode

You can turn off the default headless mode to see the browser window for local debugging by setting the headless (opens in a new tab) option.

config:
  engines:
    playwright:
      launchOptions:
        headless: false

For tests executing in distributed mode, e.g. on AWS Fargate or Azure ACI, headless mode is enforced regardless of config.

Example 2: set extra HTTP headers

This example sets the extraHTTPHeaders (opens in a new tab) option for the browser context that is created by the engine.

config:
  engines:
    playwright:
      contextOptions:
        extraHTTPHeaders:
          x-my-header: my-value

Example 3: using TypeScript

Added inv2.0.4

You can use a TypeScript file in config.processor.

config:
  target: https://www.artillery.io
  engines:
    playwright: {}
  processor: './flows.ts'
scenarios:
  - engine: playwright
    testFunction: 'helloFlow'
import { Page } from 'playwright';
import { expect } from '@playwright/test';
 
export async function helloFlow(page: Page) {
  await page.goto('https://www.artillery.io/');
  await expect(page.getByText('Never Fail To Scale')).toBeVisible();
}

Example 4: Aggregate metrics by scenario name

By default metrics are aggregated separately for each unique URL. When load testing the same endpoint with different/randomized query params, it can be helpful to group metrics by a common name.

To enable the option pass aggregateByName: true to the playwright engine and give a name to your scenarios:

config:
  target: https://artillery.io
  engines:
    playwright: { aggregateByName: true }
  processor: './flows.js'
scenarios:
  - name: blog
    engine: playwright
    testFunction: 'helloFlow'

flows.js:

module.exports = { helloFlow };
 
function helloFlow(page) {
  await page.goto(`https://artillery.io/blog/${getRandomSlug()}`);
}

This serves a similar purpose to the useOnlyRequestNames option from the metrics-by-endpoint plugin.

Example 5: Record Playwright Traces

config:
  target: https://www.artillery.io
  engines:
    # Enable trace recordings:
    playwright:
      trace:
        enabled: true

You will need to create an Artillery Cloud account and run your test with --record flag to use this feature. Once enabled, traces for failing virtual users will be uploaded to the Artillery Cloud dashboard. Trace recording works for both tests running locally, and distributed tests (on AWS or Azure).

Playwright compatibility

Each release of Artillery includes a specific version of Playwright (the playwright (opens in a new tab) npm package), which is the version of Playwright that's used to run load tests. Playwright releases are usually backwards-compatible, but if your Playwright code depends on a feature that was introduced in a newer version of Playwright than the one bundled with Artillery, you will need to wait until a new version of Artillery is released.

Artillery versionPlaywright version
2.0.21 (opens in a new tab)1.48.0 (opens in a new tab)
2.0.20 (opens in a new tab)1.45.3 (opens in a new tab)
2.0.19 (opens in a new tab)1.45.3 (opens in a new tab)
2.0.18 (opens in a new tab)1.45.2 (opens in a new tab)
2.0.17 (opens in a new tab)1.45.0 (opens in a new tab)
2.0.16 (opens in a new tab) - 2.0.15 (opens in a new tab)1.44.1 (opens in a new tab)
2.0.14 (opens in a new tab) - 2.0.12 (opens in a new tab)1.44.0 (opens in a new tab)
2.0.11 (opens in a new tab) - 2.0.10 (opens in a new tab)1.43.1 (opens in a new tab)
2.0.9 (opens in a new tab) - 2.0.7 (opens in a new tab)1.42.1 (opens in a new tab)
2.0.6 (opens in a new tab)1.41.2 (opens in a new tab)
2.0.5 (opens in a new tab)1.41.0 (opens in a new tab)
2.0.4 (opens in a new tab) - 2.0.0-38 (opens in a new tab)1.39.0 (opens in a new tab)
HTTPSocket.IO