Real-time connections at scale
Read firstLoad balancingMessage queues
Most of your servers forget a request the moment they answer it. Any server can take the next one, and you can restart any of them whenever you like. A server holding open connections to 100,000 phones is different. Each of those phones is attached to that one machine, and a message for any of them has to find its way there.
That one change, from forgetting to remembering, is what this chapter is about. Which protocol carries the bytes (WebSocket, server sent events or long polling) is the smaller question, and WebSockets vs SSE vs long polling answers it. Here it is about everything around the connection: how many fit on a server, how a message finds the right one, and what happens when 100,000 of them drop at once.
You deploy your connection servers one at a time, and each restart drops its 100,000 connections. Those phones reconnect at once. What happens to the servers that were not restarted?
A connection is state#
An open connection is memory on a server: the operating system's buffers for the socket, the TLS session, and whatever the app keeps per user, like who they are and which chats they are watching. Say tens of kilobytes each. It is small per user and large per server.
The last row is the one to remember. More connections per server means fewer servers, and also a bigger reconnect wave each time one restarts. 100,000 phones reconnecting together is the failure the rest of this chapter is built around.
In January 2012 WhatsApp wrote that it had pushed one FreeBSD server, running Erlang, past two million TCP connections. A couple of years later it was running about one million per server in production, because the servers were doing much more work for each connection. The test number and the working number are different things, and a design should use the working one.
Finding the server a user is on#
A message arrives for Priya. She is connected to one of a few hundred servers. Which one? There are three answers, and each suits a different shape of traffic.
Design WhatsApp uses the registry, because most messages go to one person. Live cricket scores uses topics, because thirty million people watch the same match. Picking by the shape of the traffic, rather than by habit, is most of the answer.
Presence is a fan out problem in disguise#
"Online" and "last seen" look like a small feature. Do the arithmetic. A user with 500 contacts who opens and closes the app 20 times a day causes 10,000 presence updates a day, if every contact is told every time. Across tens of millions of users, presence can outweigh the messages.
Three things keep it sane:
- Heartbeat and expiry. A user is online while their connection sends a heartbeat, every 30 seconds or so, and offline when the entry expires. Nobody has to send a goodbye, which matters because a phone that loses signal never does.
- Only tell people who are looking. Send presence for the chats someone has open, not for their whole contact list. Most presence updates are about people nobody is looking at.
- Smooth the flapping. A phone on a train connects and drops every minute. Wait a few seconds before announcing "offline", or every tunnel is two updates to everyone.
Balancing connections that last for hours#
A load balancer is built for short requests: spread them out and the load evens itself. Long connections do not even out. Three problems come with them.
- A new server gets nothing. Add a server and it sits idle, because nobody reconnects until something breaks. Balance new connections by "fewest open connections", not round robin, and let busy servers ask a slice of their clients to move.
- Deploys drop everyone on a server. Drain instead: stop accepting new connections, then tell existing clients to reconnect over the next minute or two, each at a random moment.
- Every server needs a ceiling. Past a set number of connections, refuse new ones quickly so they go elsewhere, rather than accepting until memory runs out.
Long lived connections are usually balanced at layer 4, on TCP, or by a layer 7 proxy that understands the upgrade to WebSocket. L4 vs L7 load balancing covers the difference.
The reconnect storm#
Now the failure every real-time system meets. A server dies, or a whole zone does, and every phone it held notices in the same second. They all reconnect. The servers that are left now take a wave of new connections, and setting one up is expensive: a TCP handshake, a TLS handshake and an authentication check, far more work than a message.
Two fixes, and they live in different places.
- On the client: backoff with jitter. Wait before reconnecting, double the wait after each failure, and pick a random moment inside that wait. The randomness is what matters. Without it, a hundred thousand phones double in step and arrive together every time.
- On the server: refuse quickly. When new connections arrive faster than you can set them up, turn the extra away at once with a hint to come back later, instead of half setting them up and dropping them. A fast refusal costs far less than a slow one.
And when a phone does get back, it should not start from nothing. It sends the last message or version it saw and gets only what it missed: a sequence number in chat, a version in live scores. Reconnecting then costs a small catch up, not a reload.
Break it#
Read every step as text
- normal · Healthy. Connections are long lived, so very few are being set up at any moment. The servers spend their time moving messages, and the registry sees mostly heartbeats.
- deploy · Degraded. A server restarts without draining. 100,000 phones reconnect within a second or two. The others absorb it, just, and the registry takes 100,000 writes in a burst while it still points some users at the server that just went away.
- zone lost · Down. A whole zone fails, and a third of all phones reconnect together. Every client retries on the same schedule, so the remaining servers spend their CPU on handshakes they then drop. Connections fail, so phones retry again, and the storm keeps going long after the zone came back.
- repaired · Healthy. Clients back off with jitter, servers refuse new connections quickly above a set rate, and deploys drain each server over a couple of minutes. Phones that get back send the last sequence they saw and fetch only what they missed. The same failure now takes a minute to recover instead of an hour.
Trade-offs#
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| A registry of user to server | One lookup and one delivery for a message to one person. | A store that is briefly wrong after every crash, so delivery needs a fallback. | Chat and anything else sent to one person at a time. |
| Topics that servers subscribe to | Publish once, and only servers with a watcher get a copy. | A pub/sub system to run, and subscribe and unsubscribe churn as people come and go. | Many people watching the same thing: scores, rooms, live documents. |
| Many connections per server | Fewer servers to run and pay for. | A bigger reconnect wave whenever one restarts. | Up to the working limit you have measured, not the test limit. |
| Draining deploys | Reconnects spread over minutes instead of one second. | Deploys take longer, and old and new versions run side by side meanwhile. | Always, for a tier that holds connections. |
Interview replay#
Checkpoint#
1. 50 million users connected at once, 100,000 connections per server. How many servers just to hold the connections?
2. Phones reconnect with exponential backoff but no randomness. Why can that be worse than you expect?
3. Same system, but now one live event has ten million viewers. What changes first?
- An open connection is state, tens of kilobytes on one specific server.
- 50 million connected users at 100,000 per server is 500 servers just to hold sockets.
- One person: look them up. Many people watching: a topic.
- Backoff without jitter keeps the herd together. The randomness is the fix.
A server holding open connections is stateful, so a message has to find the one server holding its recipient, a deploy disconnects real people, and adding a server does not rebalance anything. Connections cost tens of kilobytes each, so at a hundred thousand per server, fifty million users is five hundred servers just to keep sockets open, and I size by the working limit, not the test limit. For routing, a message to one person uses a registry from user to server, kept alive by heartbeat, with a fallback to push notifications when it is stale, and something many people watch uses a topic that servers subscribe to. Presence is a fan out problem, so I only send it for chats people have open and smooth out flapping. Long lived connections need balancing by connection count and draining deploys. The failure to design for is the reconnect storm: clients back off exponentially with jitter, servers refuse new connections quickly above a safe rate, and a returning client asks only for what it missed since its last sequence number.