Vertical vs horizontal scaling
Published
Your chai stall is getting too busy. You can buy a bigger stove with more burners and keep one person running it: vertical scaling, or scaling up. Or you can open three more stalls with three more people: horizontal scaling, or scaling out. The bigger stove is simpler, until there is no bigger stove for sale, and if it breaks the whole business stops. More stalls have no ceiling, but now you have to split the customers between them and keep the recipe the same in all four.
Your Postgres database is at 85% CPU on a 16 core machine. Your Node.js API servers are at 85% CPU too, four of them. Would you scale both the same way? Why or why not?
Vertical: simple, until it is not#
A bigger machine needs no new architecture, and modern machines are enormous. AWS sells instances with hundreds of vCPUs and terabytes of memory. A single well tuned Postgres on a large machine handles tens of thousands of transactions per second. Stack Overflow famously served one of the busiest sites on the internet for years from a small number of powerful SQL Server machines.
The limits:
- A ceiling. At some point there is no bigger machine.
- Cost curve. The largest sizes cost more per core than mid sized ones, and the jump between sizes is a doubling.
- Downtime to resize, usually a restart onto new hardware.
- One failure domain. A single machine is a single point of failure, so you still need a standby replica for availability, which is a second machine anyway.
Horizontal: unlimited, with conditions#
Adding machines is the only way past the biggest box, and it brings redundancy for free: lose one of ten servers and you lose 10% of capacity, not the service. Cloud autoscaling adds and removes them with traffic. The conditions:
- Statelessness. Any server must be able to handle any request, so sessions, uploads and caches move out of process into Redis, object storage or the database. This is easy for application servers and the reason they are the first thing scaled out.
- Distribution. A load balancer spreads requests, health checks remove dead machines (see L4 vs L7 load balancing).
- Data is the hard part. Scaling a database out means read replicas (stale reads) or sharding (cross shard queries and transactions become hard). That is why databases are scaled up first and out last. Sharding covers how.
- Coordination costs. More machines doing a shared job spend more time agreeing. Ten times the servers is rarely ten times the throughput for anything that needs coordination.
No single machine you are willing to buy carries this. If these are app servers, add more behind the load balancer. If it is a database, split reads onto replicas, cache, and only then shard.
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Vertical (scale up) | No code or architecture change, keeps transactions and joins local, fewer things to run. | Hard ceiling, steep prices at the top, restart to resize, single failure domain. | Databases first, early stage products, workloads hard to split (a big in memory cache, a single writer). |
| Horizontal (scale out) | Practically unlimited, redundancy, autoscaling, small cheap machines. | Needs stateless design, load balancing, and for data, replication or sharding with their consistency costs. | Stateless app servers and workers always; databases once a big machine plus replicas and caches run out. |
Adding API servers when the database is the bottleneck makes things worse: more servers open more connections and send more queries at the same overloaded database. Find the saturated resource first (CPU, memory, disk I/O, connections, a lock) and scale that. A connection pooler like PgBouncer is often the first fix when app servers multiply.
- AWSAuto Scaling groups; instance sizes up to hundreds of vCPUs
- KubernetesHorizontal Pod Autoscaler (out), Vertical Pod Autoscaler (up)
- PostgreSQLUsually scaled up, then read replicas, then sharding (Citus)
- RedisOut of process state so app servers can scale out
In the interview#
Checkpoint#
1. What must be true of app servers before you can scale them out freely?
2. Why are databases usually scaled up before out?
3. Even if one server carries all your traffic, why run at least two?
- Vertical: bigger machine. Simple, ceilinged, single failure domain.
- Horizontal: more machines. Unlimited, redundant, needs statelessness.
- App tier out, data tier up first, then replicas, then shard.
- Find the saturated resource before scaling anything.
- Two servers minimum, for availability.
Vertical scaling means a bigger machine: no code changes and data stays local, but there is a ceiling, prices climb steeply at the top, resizing needs a restart, and it remains one failure domain. Horizontal scaling means more machines behind a load balancer: practically unlimited and redundant, with autoscaling, but the servers must be stateless, so sessions and files move to shared stores, and for data it means replicas or sharding with their consistency costs. In practice I scale stateless app servers and workers out from the start, running at least two for availability, and scale the database up first, adding caching and read replicas, sharding only when that runs out. And before scaling anything, I find the saturated resource, because adding app servers in front of an overloaded database makes it worse.