Why Artillery?

What you’ll learn

  • What Artillery is and why you’d want use it
  • Artillery’s philosophy
  • Key Artillery features

Artillery in a nutshell

Artillery is a modern, powerful & easy-to-use performance testing toolkit. Use it to ship scalable applications that stay performant & resilient under high load.

Artillery prioritizes developer productivity and happiness, and follows the “batteries-included” philosophy.

What do we mean by “performance testing”?

You can use Artillery to run two types of performance tests:

  • Tests that put load on a system, i.e. load tests, stress tests, and soak tests
  • Tests that verify that a system is working as expected, i.e. continuous functional tests, also known by a number of other names such as: synthetic monitoring, semantic monitoring, production scripted testing, and continuous verification. Think ping on steroids - automated probes running continuously against services & APIs to test key user journeys.

What kinds of services can I test with Artillery?

Artillery is designed for testing backend systems, such as API services, ecommerce backends, chat systems, game backends, databases, message brokers and queues, and anything else that can be communicated with over a network.

It can be used to test any backend regardless of what protocol it speaks or what language it’s written in. Artillery supports HTTP, WebSocket, and Socket.io out of the box, and many additional protocols such as HLS, Kinesis, and Kafka via plugins. Support for additional protocols can be added via Artillery’s plugin interface.

Can I test web UIs with Artillery?

No, Artillery is not designed for testing web front-ends (i.e. it is not a browser, and does not run client-side Javascript for example). If you need to test the UI of a web application or website, we would recommend using Cypress.io for that part of your testing stack.

Who uses Artillery?

Artillery is typically used across teams responsible for delivery, testing, and operating production backend systems: from application developers, to test & QA engineers, and ops/SREs.

Typical usage scenarios

Artillery is used in a large variety of scenarios, which typically include:

  • Running ad-hoc load tests against individual APIs or microservices as part of a development process to explore their performance characteristics, and make performance optimizations if needed (e.g. preventing memory leaks or optimizing CPU-heavy code)
  • Running tests against staging/feature environments as part of a CI/CD pipeline to catch performance regressions early, and to verify SLOs
  • Large-scale load tests before production releases of new services, or to prepare applications and infrastructure for heavy traffic, such as Black Friday/Cyber Monday traffic
  • Adding synthetic traffic in production to maintain a margin of safety against traffic spikes
  • Running synthetic monitoring against key APIs from multiple geographical locations to verify that key transactions & flows work as expected, and alert if anything breaks

Artillery ecosystem

Artillery consists of Artillery Core - a free, open source tool, which can be used to write and run load and functional tests, and Artillery Pro - a commercial subscription-based package which adds features for enterprise teams, such as the ability to run large-scale tests from your organization’s own AWS account.

Differences between Artillery Core and Artillery Pro

Artillery Pro builds on top of Artillery Core to provide a self-hosted (AWS) performance testing solution. It provides development teams with an internal self-service performance testing platform, with the ability to run large-scale tests with no resource limits, and additional enterprise-grade features, such as the ability to run tests from multiple geographical regions.

Every feature of Artillery Core is available in Artillery Pro, which provides a seamless upgrade path from tests running on a developer’s machine, to scaling up & running the same test scripts from your organization’s AWS account.

Our philosophy

Our mission is to help development teams ship performant, resilient, and scalable software systems.

Software is eating the world

We believe that software is eating the world. Complex software systems underpin all aspects of our daily lives. There are no industries that don’t rely on software anymore - hundreds of software systems were involved in getting your breakfast cereal to your cupboard for example.

At the same time, software systems are only getting more complex, and complexity is the enemy of performance and reliability. As our dependence on software system grows, the importance of shipping performant and reliable software grows along with it.

Performance is a feature

We believe that performance is a feature. Software which is fast and reliable makes for happier users, and happier users make for more revenue.

We believe that performance is everyone’s job and requires cross-functional collaboration between product owners, developers, QA engineers, ops & SREs.

Developer productivity above all

Artillery prioritizes developer productivity above everything else. We believe that performance tests should be as easy to write and run as possible. Every product decision in Artillery is viewed through the lens of developer experience and a batteries-included philosophy.

Features

  • Functional and load testing in one package - Re-use the same scenarios to run either load or functional tests. Use assertions and expectations on the responses, and easily extract and store data from responses to re-use in subsequent requests.
  • Multiple protocols - Test any system, regardless of the protocol it speaks or the tech stack it’s built on. HTTP and HTTP/2, Socket.io, and WebSockets are supported out of the box, with support for many other protocols available via plugins.
  • Emulate complex user behavior with scenarios - Define complex user behavior with multiple steps and transactions, loops, conditionals and custom Javascript code. Artillery is designed to support testing the most complex of systems such as e-commerce backends, transactional APIs, IoT backends, chat services, game backends and all kinds of stateful services.
  • YAML-based scripts - readable test scripts that encourage collaboration across teams and functions. No know
    ledge of any specific programming language is needed to read and understand Artillery’s YAML-based scripts. We believe that performance is everybody’s job and YAML makes it easier for everyone to contribute.
  • Full power of JS and npm - having said that, sometimes it’s handy to be able to drop down to a real programming language for advanced customization of your test code. Artillery is built on top of Node.js, and lets you customize your tests with real JS code using any of the thousands of modules on npm. Whilst not everyone knows Javascript, it is the most widely used and known programming language on the planet, and it’s very likely to be well-known or at least familiar to most developers.
  • Extensibility & hackability - Artillery itself is easy to extend in Javascript with custom engines (for additional protocols), plugins (e.g. to generate data for tests or integrate with external systems), or custom reporters (to send metrics and test results to another location).
  • Plugins and extensions - Artillery ships with several plugins to make common tasks easier, and there’s a number of ready-to-use community plugins available
  • Scale up your tests easily - Artillery Pro makes it easy to go from running tests on a developer’s machine to running a large-scale distributed load test from multiple geographical regions