Load balancing
You have four app servers. Three are healthy. The fourth shares its hardware with somebody else’s noisy job and is running four times slower than it should. Nothing has crashed, so no alarm has fired.
Your load balancer is set to round robin, so it hands that server every fourth request, same as the others. One user in four now waits ten seconds for a page. The other three servers are half idle the whole time.
A load balancer is the box that decides which server gets each request. This page is about how it decides, and what each way of deciding does on a bad day.
Four servers, one of them four times slower than the rest, and nobody has noticed yet. What would a good load balancer do about it without being told? What information would it need to do that?
What it is for
A load balancer gives your users one address and hides many servers behind it. That one idea buys you three things.
Spreading work. No single server gets everything.
Hiding failure. It checks each server every few seconds and stops sending traffic to one that does not answer. Users never see the dead server.
Changing the fleet. You can add servers, remove them, or replace them all with a new version, and the address your users know never changes.
How it chooses a server
There are three ideas, and every algorithm you will meet is a version of one of them.
Round robin. Take turns. Server 1, then 2, then 3, then 4, then back to 1. It needs no information about anything.
Least connections. Send the request to whichever server is working on the fewest requests right now. It needs a count of open requests per server, which the load balancer already has.
Hash. Compute a hash of something in the request, usually the user ID or the client’s IP address, and use it to pick the server. The same user always lands on the same server.
Below are 600 requests sent to four servers. On a healthy day you will struggle to see a difference between the three. Then flip the two switches.
Three things to take from it.
On a good day, round robin and least connections make the same choices. That is why round robin is a perfectly sensible default and why nobody thinks about it until something is uneven.
When one server is slow, least connections protects you without any configuration. A slow server finishes work slowly, so it always has more requests open, so it gets picked less. The load balancer does not know why the server is slow. It does not need to.
Hashing is the only one that keeps a user on one server, and it is the only one that breaks when a single client is loud. You are trading balance for stickiness.
Round robin and least connections both come with weights. Give a 16 core server a weight of 2 and an 8 core server a weight of 1, and the big one gets twice the turns. Weights are also how you roll out a new version safely: give the new servers a weight of 1 in 100, watch the error rate, then raise it.
When you want the same server every time
Sometimes sending a user to a different server each time is wasteful. A chat server holds an open connection per user. A media server has that user’s video segments warm in memory. Then hashing earns its place.
Plain hash(user) % servers has a nasty property: add or remove one server and almost
every user gets moved to a different one. The fix is
consistent hashing, which moves only about one server’s
worth of users when the fleet changes. If you use hashing in a load balancer, use that
form of it.
Layer 4 and layer 7
Load balancers come in two kinds, named after layers of the network model. The names sound academic. The difference is simply how much of the request the balancer reads.
For a web product, the answer is nearly always layer 7 at the front, because routing by path and handling HTTPS in one place are too useful to give up. Layer 4 is for when you are balancing something that is not HTTP, or you need raw speed in front of the layer 7 tier.
Health checks, and how long a dead server hurts
The load balancer finds dead servers by asking. Every few seconds it sends a small
request, usually to a path like /health, and expects a quick 200 back.
Two settings decide how long users suffer when a server dies: how often you ask, and how many failures in a row you need before you believe it. Ask every 5 seconds and require 3 failures, and a dead server keeps receiving traffic for up to 15 seconds. With four servers, that is a quarter of your users getting errors for 15 seconds.
You can shorten it, but not for free. Require only one failure and a server that had one slow moment gets thrown out, then let back in, then thrown out. That flapping is worse than the 15 seconds.
The most common mistake is a /health route that returns 200 as long as the process is
running. The server in the opening story would pass that check all day. A useful check
touches the things a real request needs, like the database connection, and it has a
timeout short enough that “slow” counts as “failed”.
The load balancer is also a box
You added a load balancer so that no single server could take you down. Now every request goes through one load balancer. You have moved the single point of failure, not removed it.
The standard fix is two of them. One is active and the other is watching. If the active one stops responding, the standby takes over its address within a few seconds. Managed cloud load balancers do this for you, across more than one data centre, which is the main reason to use them instead of running your own.
Break it
Trade-offs
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Round robin | Nothing to tune and nothing to go wrong inside the balancer. Perfectly even when servers are alike. | Blind to slow servers and to requests that take very different amounts of work. | Identical servers and short, similar requests. This is the right default for a plain web API. |
| Least connections | Shifts work away from a slow or busy server on its own, with no configuration. | A new empty server gets flooded for a moment, since it has the fewest connections of all. | Requests vary a lot in length, or connections stay open a long time, such as WebSockets. |
| Consistent hash by user or URL | The same user or the same URL keeps landing on the same server, so its cache is warm. | One loud client or one popular URL overloads one server, and nothing rebalances it. | Servers hold something per user or per item: open connections, sessions, cached files. |
| Layer 4 instead of layer 7 | Faster, cheaper per connection, and works for any protocol. | No routing by path, no HTTPS handling, no retries, since it never reads the request. | Non HTTP traffic, or very high connection counts in front of a layer 7 tier. |
Interview replay
Checkpoint
1. Health checks run every 5 seconds and a server is removed after 3 failures in a row. A server dies. Roughly how long does it keep receiving traffic?
2. Why does least connections cope with a slow server when round robin does not?
3. You balance by hash of user ID so each user keeps a warm cache on one server. One customer’s script starts sending 40% of all requests. What happens?
- Round robin for identical servers and short requests. Least connections when either is uneven.
- Detection time is interval times threshold. 5 seconds and 3 failures means up to 15 seconds of errors.
- Layer 4 sees IP and port. Layer 7 reads the HTTP request. Path routing and HTTPS handling need layer 7.
- Two load balancers, never one. Otherwise you have only moved the single point of failure.
A load balancer gives clients one address, spreads requests over many servers and stops sending traffic to servers that fail health checks. I would use a layer 7 balancer for a web product, since it can route by path and handle HTTPS. For the algorithm I start with round robin, which is right when servers are identical and requests are short. If requests vary in cost or connections are long lived, I switch to least connections, because it moves work away from a slow server without being told. I only hash by user when servers keep something per user, and then I use consistent hashing so a resize does not reshuffle everyone. Health checks decide how long a dead server hurts: interval times failure threshold, typically around 15 seconds. And the balancer itself must be a pair or a managed service, or it is the new single point of failure.
