If you’re building a real-time backend in Node.js such as a chat server or the backend for a collaborative app, chances are you’re using Socket.io (and you’re in good company - Microsoft, Zendesk, Trello and countless others use it too).
We won’t go into the details of what Socket.io does - if you’re reading this, you probably know it very well already. What we’ll look into is performance testing of Socket.io-based applications with Artillery.
You have a Socket.io-based app. Maybe it’s still in development, or maybe it’s already running in production. You’d like to test your app under heavy load.
Perhaps you’re interested in how much traffic it can handle before falling over. Maybe you want to test if you can scale horizontally by adding more app servers behind the load balancer. Or perhaps you’re interested in doing some CPU profiling with
dtrace to improve runtime performance.
Whatever the reason may be, step one is to generate some load on your application, and that’s exactly what this article covers. Read on to find out how to use Artillery to load test a Socket.io app.
In case you haven’t used Artillery before and need a quick intro to what it is: Artillery is a modern load testing toolkit that focuses on ease of use and developer happiness. It’s designed for testing complex apps with multi-step scenarios (such as e-commerce backends, chat servers and all kinds of transactional APIs), and comes with batteries included for testing a variety of protocols and integrating with a variety of monitoring tools. It’s very lightweight and very easy to get started with.
Artillery offers first-class native support for Socket.io out of the box. The Socket.io engine in Artillery lets you send data to the server and optionally wait for and verify responses that come back. You can also mix HTTP and Socket.io actions in the same scenario.
(Note that we are connecting to the Socket.io endpoint, not an underlying transport endpoint, e.g. the WebSocket URL exposed by Socket.io)
What will happen when we run this script with
artillery run simple-socketio-load-test.yaml?
target config value tells us that Artillery will connect to the application running on
Looking at the
phases definition, we see that Artillery will simulate 5 new users arriving to use the application every second for 600 seconds (resulting in 3000 users arriving in the space of 10 minutes).
What will those users do?
Each user spawned by Artillery will pick and run through one of the scenarios defined in the test script. In our case, there’s just one scenario, which will have each user send 50 messages with a second’s pause in between and disconnect from the server.
Let’s see how we would test a real Socket.io application. We’ll use the chatroom demo bundled with Socket.io.
The app is a simple chatroom where users can choose a nickname, join the room, and publish and receive messages.
The source code for the chatroom is available on Github: https://github.com/socketio/socket.io/tree/master/examples/chat.
Our test isn’t going to be complex, but we will try to make it realistic by modeling three different kinds of users:
- Lurkers, who typically make up the majority of users in any given chatroom. They may send the occasional message, but are mostly receiving messages from others.
- Mostly-quiet users, that will engage now and then, but stay quiet most of the time.
- Chatty users, who will send a lot of messages.
The scenario for a lurker is very simple:
# A lurker - join, listen for a while, and leave.
The nickname for the user is set with a template which generates a random alpanumeric string (
$randomString() is a built-in function in Artillery templates).
- name: "A chatty user"
After joining the chatroom and greeting other users (the greeting message is picked from a number of predefined strings at random), the user will send 10 messages with a 10 second pause between each message.
setMessage function is pretty straightforward:
const MESSAGES = [
All scenarios in the test script also make use of the weighing feature in Artillery, which allows us to “weigh” the scenarios relative to each other making some more likely to be picked by spawned users than others. For this test script, we have assigned a weight of 75 to lurkers, 15 to mostly-quiet users, and 10 to chatty users, meaning that on average 75% of all users created during a test run will be of the lurker type for example.
To see how everything fits together, check out the code Github under hassy/socketio-load-test-artillery-example.
Running the tests is easy:
artillery run socketio-chat-load-test.yaml
Or if you cloned the repository above:
npm install # install dependencies
Happy load testing!