Scaling from one server
Your app runs on one small cloud machine. The code, the database and the uploaded photos all live on it, and it costs you less than a pizza a month. Then a creator with two million followers posts your link. Within an hour the CPU is pinned at 100%, pages take twelve seconds, and the machine starts refusing connections.
You have two ways out. Make the machine bigger, or add more machines. Almost every system design answer is some mix of those two, applied to the right box in the right order. This page is that order.
Your single server is at 95% CPU. You can double its size in ten minutes with a restart, or put a second server behind a load balancer in about a day. Which do you do first, and what would have to be true for you to pick the other one?
Two ways to grow
Scaling up (also called vertical scaling) means a bigger machine. More CPU cores, more memory, a faster disk. Think of a tea stall swapping a small kettle for a huge one.
Scaling out (also called horizontal scaling) means more machines doing the same job side by side. The tea stall opens a second counter.
Scaling up is the easy one. You change nothing in your code. You stop the machine, pick a bigger size and start it again. Do this first, more often than people admit. Stack Overflow famously ran for years on a handful of web servers and a few very large database machines.
It stops working for two reasons. There is a biggest machine you can rent, a few hundred cores and a few terabytes of memory, and after that there is nothing left to buy. And one machine is still one machine: when it dies or restarts for an update, your whole product is down.
Scaling out has no ceiling and no single machine that matters. The cost is that your software has to be built for it, and the rest of this page is about what “built for it” means.
Growing one step at a time
Nobody draws the final architecture on day one. You add one piece when one thing hurts. Click through the stages in order. Each one fixes the problem named in the stage before it and creates the next one.
Each stage has its own page when you want the detail: load balancing, replication, caching and sharding.
Stateless servers, the rule that makes stage 3 work
A server is stateless when it remembers nothing between one request and the next. Everything it needs arrives with the request or sits in a shared store. That sounds like a small detail. Here is what happens when you get it wrong.
The fix is to move that memory out of the server. Keep the session in a shared store such as Redis, where every server can read it. Or put it in a signed token that the browser sends with each request. Either way, any server can now handle any request, and that is the property that lets you add and remove servers freely.
The same rule covers uploaded files (keep them in object storage, not on the server’s disk) and anything else a second server would need to know.
You can tell the load balancer to always send Riya to server 1. That is called a sticky session and it does hide the bug. But when server 1 dies, everyone stuck to it is logged out together, and you cannot remove a server without throwing users off it. Use it to buy time. Do not build on it.
Single points of failure
A single point of failure is any one box that takes the product down when it dies. To find them, point at each box in your drawing and ask: what happens if this disappears right now?
In stage 3 above, losing one app server is fine. Losing the database is the end. Losing the load balancer is also the end, which people forget, because it looks like plumbing. The fix is always the same shape: have a second one ready to take over.
The availability page puts numbers on this.
How many servers
Once the app tier is stateless, sizing it is division. The only judgement is how full you are willing to run each server.
If one of the 11 dies at peak, the other 10 run at about 60% and nobody is paged. Never plan to run servers near 100%. A full server has no room for a traffic bump, a slow query or a neighbour dying.
Break it
Here is the stage 3 design with a cache added but not yet switched on. Raise the traffic and watch which box goes red. It is not the one most people expect.
The lesson from the dial: scaling moves the bottleneck, it does not remove it. Fix the box that is red, then look again.
Trade-offs
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Scale up: a bigger machine | No code changes at all. Done in minutes. Fewer moving parts to operate. | A hard ceiling, a restart to resize, and still one machine that takes everything down. | First, almost always. Especially for the database, which is the hardest thing to scale out. |
| Scale out the app tier | No ceiling, and one dead server stops mattering. Capacity can follow traffic up and down. | Needs a load balancer and stateless servers, so sessions and files must move out first. | As soon as one server dying would be an outage, or traffic changes a lot through the day. |
| Read replicas | More read capacity and a standby that can take over from the primary. | Replicas lag behind, so users can briefly read old data. Writes do not get any faster. | Reads are the load, and the app can live with data that is a moment old. |
| A cache and a CDN | Often removes most of the load for the least money, and makes pages faster too. | Cached data can be stale, and you must decide when to throw it away. | The same things are read again and again, which is true of nearly every product. |
| Sharding | Write capacity that grows with the number of machines. | Queries across shards are slow, moving data between shards is painful, and it is hard to reverse. | Last. Only when writes or data size have outgrown the largest single machine. |
Interview replay
Checkpoint
1. Users are randomly logged out after you add a second app server. What is the most likely cause?
2. You have 12 app servers and the database is at 100% CPU. What does adding 12 more app servers do?
3. Which of these does a read replica NOT help with?
- Scale up first, scale out second, shard last. Each step costs more than the one before.
- Stateless means any server can answer any request. Sessions and files live outside the app server.
- Run servers at about 60% and keep one spare. N + 1 is the smallest honest answer to “how many”.
- Replicas and caches help reads only. Every write still goes to one primary until you shard.
I would start on one server and scale up first, because resizing a machine costs nothing in code. The limits are a ceiling on machine size and the fact that one machine is a single point of failure. So next I separate the database from the app, make the app stateless by moving sessions to Redis and files to object storage, and run several app servers behind a load balancer with one spare. At that point the database is the bottleneck, so for reads I add a cache in front and read replicas behind, and put static files on a CDN. Those help reads only. If writes outgrow one primary I shard the data, and I do that last because it is the hardest step to undo. At every stage I check which box is actually full before adding anything.
