Using Artillery to Load Test GraphQL APIs
Dennis Martinez
See end-to-end example with Apollo and Prisma - artillery-examples/graphql-api-server.
The popularity of GraphQL for building and querying APIs has been on the rise for years, with no sign of slowing down. Developers can now build robust and flexible APIs that give consumers more choices when accessing their data. This flexibility, in turn, can help with application performance compared to traditional RESTful APIs. Providing API consumers the flexibility to fetch the data they need avoids the overhead of processing unnecessary information.
Despite all the advantages GraphQL provides to developers, it’s also easy to introduce potential performance issues when constructing APIs:
- Its flexibility can allow clients to craft inefficient queries when fetching server-side data, especially when using nested queries that need to get data from multiple database tables or sources. This problem is more prevalent when using ORMs with GraphQL since it can mask any inefficiencies in your database calls.
- Since GraphQL allows consumers to build different queries as they need, it isn’t easy to implement caching effectively. Since each request made to the API is potentially unique, developers can’t use standard caching techniques. Some libraries provide built-in caching, but it may not work in all scenarios.
Most developers building GraphQL APIs eventually run into one or both of these issues. While these performance problems are simple to deal with, they’re also easily overlooked. It’s also not uncommon to find yourself with a sluggish API after deploying to production when dealing with actual data. That’s why load testing your GraphQL APIs before and after deploying is more crucial than ever.
With Artillery, you can create test scripts that can test common workflows for GraphQL to smoke out any potential performance issues that can creep up when you least expect. Running regular load tests on your APIs both during development and against your production servers will keep your APIs performant and resilient for your customers. This article shows you how to use Artillery to keep your GraphQL services in check.
GraphQL API example
For this article, let’s assume we have a GraphQL API with a data store, some queries to fetch this data, and mutations to manage data manipulation based on this diagram:
The data store consists of two tables to store user information and messages associated with that user for performing standard CRUD operations. There’s a single query to fetch a specific user by their data store ID. The GraphQL API also has some mutations to create, update, and delete a user, along with creating messages associated with a single user.
Let’s imagine that most consumers of the GraphQL API perform two primary flows after creating and logging in as the new user. The first flow will create new messages through the API and later fetch the user’s account information along with their messages. The other flow handles updating a user. We want to make sure these flows remain performant under load for whoever accesses the API. Let’s create a few load tests using Artillery.
Creating our first GraphQL load test
Most GraphQL APIs handle requests via standard HTTP GET and POST methods, meaning we can easily use Artillery’s built-in HTTP engine to make requests for load testing. Both methods work similarly, with some differences related to how HTTP works. For instance, GET requests are easier to cache, while POST responses aren’t cacheable unless they include specific headers. For simplicity in our load tests, we’ll use the POST method to make our GraphQL API requests.
When served over HTTP, a GraphQL POST request accepts a JSON-encoded body with a query field containing the GraphQL query to execute on the server. The body can also include an optional variables field to set and use variables in the specified GraphQL query. We’ll use both of these fields for our load test.
For our first test, we’ll go through one of the flows described above:
- First, we’ll create a new user record through the GraphQL API and capture their email address from the GraphQL response. We’ll generate randomized usernames and email addresses using Artillery’s
$randomString()method to ensure we don’t run into uniqueness constraints when creating users during the test. - Next, we’ll log in as the created user to test the login process and capture their ID from the data store.
- With the user’s ID in hand, we’ll then use the
loopconstruct to create 100 new messages associated with the user. - After creating those messages, we’ll make a request to return the user data and their messages to verify how the API performs when fetching associated data between different database tables.
- Finally, to clean up the database, we’ll delete the user at the end of the flow, which will also delete the user’s associated messages.
Let’s see how this test script looks for executing this flow:
config:
target: 'https://graphql.test-app.dev'
phases:
- duration: 600
arrivalRate: 50
scenarios:
- name: 'Create and fetch messages flow'
flow:
- post:
url: '/'
json:
query: |
mutation CreateUserMutation($input: CreateUserInput) {
createUser(input: $input) {
email
}
}
variables:
input:
username: '{{ $randomString() }}'
email: 'user-{{ $randomString() }}@artillery.io'
password: 'my-useful-password'
capture:
json: '$.data.createUser.email'
as: 'userEmail'
- post:
url: '/'
json:
query: |
mutation LoginUserMutation($email: String!, $password: String!) {
loginUser(email: $email, password: $password) {
id
}
}
variables:
email: '{{ userEmail }}'
password: 'my-useful-password'
capture:
json: '$.data.loginUser.id'
as: 'userId'
- loop:
- post:
url: '/'
json:
query: |
mutation CreateMessageMutation($authorId: ID!, $body: String!) {
createMessage(authorId: $authorId, body: $body) {
id
body
}
}
variables:
authorId: '{{ userId }}'
body: 'Message Body {{ $loopCount }}'
count: 100
- post:
url: '/'
json:
query: |
query Query($userId: ID!) {
user(id: $userId) {
username
email
messages {
id
body
}
}
}
variables:
userId: '{{ userId }}'
- post:
url: '/'
json:
query: |
mutation DeleteUserMutation($deleteUserId: ID!) {
deleteUser(id: $deleteUserId) {
id
}
}
variables:
deleteUserId: '{{ userId }}'
For this example, we have our GraphQL API set up at https://graphql.test-app.dev. The Artillery test will run for ten minutes, generating 50 virtual users per second. In our scenario, we have our defined flow with a few POST requests made to the root URL (/) where the GraphQL API handles our requests.
Each step in our flow section works similarly. we’ll have a JSON request to the GraphQL server with the query and variables fields (where necessary). The query field contains the GraphQL query or mutation sent to the server for processing. we’ll use a pipe (|) when sending the data to make the body more readable in our test script.
Each query and mutation in our script is set up to handle variables since we’ll use them for keeping track of the dynamic data generated throughout the test phase. That’s where the variables field comes into play. This field can set up the variables we’ll use through a standard JSON object. The GraphQL server will handle setting up the values where specified in a query or mutation.
For instance, in the step where we create a user, the createUser mutation expects an input argument, which is an object containing the username, email, and password fields. The variables field will construct the input object using fields required in the mutation argument and pass it to the query. It allows us to generate unique usernames and emails for each virtual user dynamically.
Assuming we save the test script in a file called graphql.yml, we can run it with Artillery using artillery run graphql.yml. After Artillery completes all the requests, you’ll see how the GraphQL API performs under this load:
All VUs finished. Total time: 10 minutes, 39 seconds
--------------------------------
Summary report @ 15:31:12(+0900)
--------------------------------
vusers.created_by_name.Create and fetch messages flow: ...... 30000
vusers.created.total: ....................................... 30000
vusers.completed: ........................................... 30000
vusers.session_length:
min: ...................................................... 947.6
max: ...................................................... 20767.2
median: ................................................... 1465.6
p95: ...................................................... 20136.3
p99: ...................................................... 20136.3
http.request_rate: .......................................... 29/sec
http.requests: .............................................. 3120000
http.codes.200: ............................................. 3120000
http.responses: ............................................. 3120000
http.response_time:
min: ...................................................... 173
max: ...................................................... 4870
median: ................................................... 186.8
p95: ...................................................... 1312
p99: ...................................................... 3899
This example shows how straightforward it is to create a fully-fledged load test for GraphQL services. Artillery’s HTTP engine has everything you need out of the box to check the most important actions of your GraphQL API. It’s a simple yet powerful way to ensure your GraphQL services are resilient to any spikes in traffic and remain performant throughout. Although the performance of our GraphQL service isn’t close to what we’d like, it gives us an excellent starting point for improvement.
Expanding the GraphQL load test
Let’s add another scenario to validate the GraphQL API’s performance further to expand our load test. For this flow, we want to validate that the service correctly handles password updates. For security purposes, the GraphQL service uses bcrypt hashing to ensure we don’t store plain-text passwords in the database. Depending on how a service uses bcrypt, the hashing function can slow down password-related functionality, so it’s an excellent candidate to check for performance purposes.
The second scenario for our test goes through the following steps:
- we’ll create a new user record and log in as that user, the same way we did in the first scenario.
- Next, we’ll log in as that user to test the login process and capture their ID from the data store.
- With the user’s ID in hand, we’ll call the mutation for updating a user’s database record to update their password. This step helps us verify that the request works as expected along with testing its performance.
- Just like before, we’ll delete the user at the end of the flow to keep the database clean of any test records.
The entire test script with both flows is as follows:
config:
target: 'https://graphql.test-app.dev'
phases:
- duration: 600
arrivalRate: 50
scenarios:
- name: 'Create and fetch messages flow'
flow:
- post:
url: '/'
json:
query: |
mutation CreateUserMutation($input: CreateUserInput) {
createUser(input: $input) {
email
}
}
variables:
input:
username: '{{ $randomString() }}'
email: 'user-{{ $randomString() }}@artillery.io'
password: 'my-useful-password'
capture:
json: '$.data.createUser.email'
as: 'userEmail'
- post:
url: '/'
json:
query: |
mutation LoginUserMutation($email: String!, $password: String!) {
loginUser(email: $email, password: $password) {
id
}
}
variables:
email: '{{ userEmail }}'
password: 'my-useful-password'
capture:
json: '$.data.loginUser.id'
as: 'userId'
- loop:
- post:
url: '/'
json:
query: |
mutation CreateMessageMutation($authorId: ID!, $body: String!) {
createMessage(authorId: $authorId, body: $body) {
id
body
}
}
variables:
authorId: '{{ userId }}'
body: 'Message Body {{ $loopCount }}'
count: 100
- post:
url: '/'
json:
query: |
query Query($userId: ID!) {
user(id: $userId) {
username
email
messages {
id
body
}
}
}
variables:
userId: '{{ userId }}'
- post:
url: '/'
json:
query: |
mutation DeleteUserMutation($deleteUserId: ID!) {
deleteUser(id: $deleteUserId) {
id
}
}
variables:
deleteUserId: '{{ userId }}'
- name: 'Update password flow'
flow:
- post:
url: '/'
json:
query: |
mutation CreateUserMutation($input: CreateUserInput) {
createUser(input: $input) {
email
}
}
variables:
input:
username: '{{ $randomString() }}'
email: 'user-{{ $randomString() }}@artillery.io'
password: 'my-useful-password'
capture:
json: '$.data.createUser.email'
as: 'userEmail'
- post:
url: '/'
json:
query: |
mutation LoginUserMutation($email: String!, $password: String!) {
loginUser(email: $email, password: $password) {
id
}
}
variables:
email: '{{ userEmail }}'
password: 'my-useful-password'
capture:
json: '$.data.loginUser.id'
as: 'userId'
- post:
url: '/'
json:
query: |
mutation UpdateUserMutation($updateUserId: ID!, $input: UpdateUserInput) {
updateUser(id: $updateUserId, input: $input) {
id
}
}
variables:
updateUserId: '{{ userId }}'
password: 'my-new-password'
- post:
url: '/'
json:
query: |
mutation DeleteUserMutation($deleteUserId: ID!) {
deleteUser(id: $deleteUserId) {
id
}
}
variables:
deleteUserId: '{{ userId }}'
Running the test script with Artillery as we did previously (artillery run graphql.yml) shows the generated VUs going through both scenarios:
All VUs finished. Total time: 11 minutes, 20 seconds
--------------------------------
Summary report @ 15:43:19(+0900)
--------------------------------
vusers.created_by_name.Update password flow: ................ 18469
vusers.created.total: ....................................... 30000
vusers.created_by_name.Create and fetch messages flow: ...... 11531
vusers.completed: ........................................... 30000
vusers.session_length:
min: ...................................................... 899.1
max: ...................................................... 30451.4
median: ................................................... 1300.1
p95: ...................................................... 29445.4
p99: ...................................................... 29445.4
http.request_rate: .......................................... 28/sec
http.requests: .............................................. 1966900
http.codes.200: ............................................. 1966900
http.responses: ............................................. 1966900
http.response_time:
min: ...................................................... 172
max: ...................................................... 5975
median: ................................................... 186.8
p95: ...................................................... 1620
p99: ...................................................... 4867
As seen in our less-than-ideal performance results, GraphQL helps developers build flexible APIs with ease, but that flexibility comes at a cost if you’re not careful. It’s easy to accidentally introduce N+1 queries or implement ineffective caching that brings your GraphQL API to a grinding halt with only a handful of consumers. This example test script gives you an idea of how to use Artillery to run robust load tests against your GraphQL services to find any bottlenecks or other issues during periods of high traffic.
End-to-end example
An end-to-end example with Apollo and Prisma can be seen on artillery-examples/graphql-api-server.