Artillery

If you're building a real-time backend in Node.js such as a chat server or the backend for a collaborative app, chances are you're using Socket.io (and you're in good company - Microsoft, Zendesk, Trello and countless others use it too).

We won't go into the details of what Socket.io does - if you're reading this, you probably know it very well already. What we'll look into is performance testing of Socket.io-based applications with Artillery.

The Situation

You have a Socket.io-based app. Maybe it's still in development, or maybe it's already running in production. You'd like to test your app under heavy load.

Perhaps you're interested in how much traffic it can handle before falling over. Maybe you want to test if you can scale horizontally by adding more app servers behind the load balancer. Or perhaps you're interested in doing some CPU profiling with perf or dtrace to improve runtime performance.

Whatever the reason may be, step one is to generate some load on your application, and that's exactly what this article covers. Read on to find out how to use Artillery to load test a Socket.io app.

Artillery 101

In case you haven't used Artillery before and need a quick intro to what it is: Artillery is a modern load testing toolkit that focuses on ease of use and developer happiness. It's designed for testing complex apps with multi-step scenarios (such as e-commerce backends, chat servers and all kinds of transactional APIs), and comes with batteries included for testing a variety of protocols and integrating with a variety of monitoring tools. It's very lightweight and very easy to get started with.

Artillery Socket.io

Artillery offers first-class native support for Socket.io out of the box. The Socket.io engine in Artillery lets you send data to the server and optionally wait for and verify responses that come back. You can also mix HTTP and Socket.io actions in the same scenario.

Let's look at an example

# simple-socketio-load-test.yaml
config:
  target: "http://my-backend.local"
  phases:
    - duration: 600
      arrivalRate: 5
scenarios:
  - name: "Connect and send a bunch of messages"
    flow:
      - loop:
          - emit:
              channel: "send message"
              data: "hello world!"
          - think: 1
        count: 50

(Note that we are connecting to the Socket.io endpoint, not an underlying transport endpoint, e.g. the WebSocket URL exposed by Socket.io)

What will happen when we run this script with artillery run simple-socketio-load-test.yaml?

The target config value tells us that Artillery will connect to the application running on http://my-backend.local.

Looking at the phases definition, we see that Artillery will simulate 5 new users arriving to use the application every second for 600 seconds (resulting in 3000 users arriving in the space of 10 minutes).

What will those users do?

Each user spawned by Artillery will pick and run through one of the scenarios defined in the test script. In our case, there's just one scenario, which will have each user send 50 messages with a second's pause in between and disconnect from the server.

Testing the demo chat app

Let's see how we would test a real Socket.io application. We'll use the chatroom demo bundled with Socket.io.

The app is a simple chatroom where users can choose a nickname, join the room, and publish and receive messages.

The source code for the chatroom is available on Github: https://github.com/socketio/socket.io/tree/master/examples/chat.

Writing the test

Our test isn't going to be complex, but we will try to make it realistic by modeling three different kinds of users:

1. Lurkers, who typically make up the majority of users in any given chatroom. They may send the occasional message, but are mostly receiving messages from others.
2. Mostly-quiet users, that will engage now and then, but stay quiet most of the time.
3. Chatty users, who will send a lot of messages.

The scenario for a lurker is very simple:

 # A lurker - join, listen for a while, and leave.
- name: "A user that just lurks"
  weight: 90
  engine: "socketio"
  flow:
    - get:
        url: "/"
    - emit:
        channel: "add user"
        data: "lurker-{{ $randomString() }}"
    - think: 60

The nickname for the user is set with a template which generates a random alpanumeric string ($randomString() is a built-in function in Artillery templates).

The scenario for a chatty makes use of the ability to run custom logic written in Javascript as part of a scenario:

  - name: "A chatty user"
    weight: 10
    engine: "socketio"
    flow:
      - get:
          url: "/"
      - emit:
          channel: "add user"
          data: "chatty-{{ $randomString() }}"
      - emit:
          channel: "new message"
          data: "{{ greeting }}"
      - loop:
          - function: "setMessage"
          - emit:
              channel: "new message"
              data: "{{ message }}"
          - think: 10
        count: 10
      - think: 60

After joining the chatroom and greeting other users (the greeting message is picked from a number of predefined strings at random), the user will send 10 messages with a 10 second pause between each message.

The setMessage function is pretty straightforward:

const MESSAGES = [
  'what a nice day',
  'how\'s everybody?',
  'how\'s it going?',
  'what a lovely socket.io chatroom',
  'to be or not to be, that is the question',
  'Romeo, Romeo! wherefore art thou Romeo?',
  'now is the winter of our discontent.',
  'get thee to a nunnery',
  'a horse! a horse! my kingdom for a horse!'
];

function setMessage(context, events, done) {
  // pick one of the messages
  const index = Math.floor(Math.random() * MESSAGES.length);
  // make it available to templates as "message"
  context.vars.message = MESSAGES[index];
  return done();
}

All scenarios in the test script also make use of the weighing feature in Artillery, which allows us to "weigh" the scenarios relative to each other making some more likely to be picked by spawned users than others. For this test script, we have assigned a weight of 75 to lurkers, 15 to mostly-quiet users, and 10 to chatty users, meaning that on average 75% of all users created during a test run will be of the lurker type for example.

To see how everything fits together, check out the code Github under hassy/socketio-load-test-artillery-example.

Running the tests

Running the tests is easy:

artillery run socketio-chat-load-test.yaml

Or if you cloned the repository above:

npm install # install dependencies
npm run load # run the load test

Questions or comments?

If you have any questions or commments, feel free to reach out on Github Discussions.

Happy load testing!