Availability and the nines
“We are up 99.9% of the time” sounds like nearly perfect. It is almost nine hours of being down every year. And those hours do not arrive as one quiet second a day. They arrive as one bad evening, usually the evening you ran a sale.
Availability is the number every design gets judged on, and most people quote it without being able to do the arithmetic behind it. The arithmetic is two rules. Both fit on a sticky note.
A request passes through five services, one after another. Each one is up 99.9% of the time. Is the whole path better than 99.9%, the same, or worse? Guess how many hours a year it is down.
What the number means
Availability is the share of time your system does what users ask. If 1,000 requests come in and 999 get a correct answer, that is 99.9%.
People count the nines as shorthand. 99.9% is “three nines”. 99.99% is “four nines”. Each extra nine means ten times less downtime, and in practice something like ten times more effort.
Guess the downtime for each before you reveal it. A year has 8,760 hours, which is all you need.
The useful anchor is three nines: about nine hours a year, or 43 minutes a month. Every other row is that one moved by a factor of ten.
Rule one: a chain is weaker than its weakest link
A request rarely touches one thing. It goes through a load balancer, an app server, an auth service, a database. If any one of them is down, the request fails. Parts that all have to work are said to be in series.
For parts in series, you multiply. Two services at 99.9% each give you 0.999 x 0.999, which is 99.8%. You did nothing wrong and you already lost some.
Drag the first slider and watch the bottom number. Leave “copies” at 1 for now.
Every service you add to the path costs you availability, even though each one looks fine by itself. With 5 in a row you are at 99.50%. Now raise the copies slider.
Five services at three nines each is 99.5%. That is 44 hours a year, five times worse than any single one of them. This is the real cost of a microservice architecture that nobody mentions on the diagram: every box you add in series is something else that can be down.
Rule two: copies multiply the other way
Now put two copies of a service side by side, with a load balancer that uses whichever one is alive. The service is only down when both copies are down together. Parts where any one is enough are said to be in parallel.
For parts in parallel, you multiply the chances of failure. A server that is up 99% of the time is down 1% of the time. Two of them are both down 1% of 1%, which is 0.01%. So two mediocre 99% servers make one 99.99% service.
Go back to the calculator and raise “copies” to 2. Five three-nines services with two copies each come out at about 99.9995%. That is the entire reason every box in a production diagram is drawn as a stack of three.
The parallel rule only holds if one copy failing tells you nothing about the other. In real systems copies share things: the same bad deploy, the same config file, the same expired certificate, the same data centre, the same database behind them. When the shared thing breaks, all copies go at once and the arithmetic is worthless. The big outages you read about are almost never one server dying. They are one change applied to every copy.
So redundancy has a second half that the formula leaves out: make the copies as unrelated as you can. Put them in different data centres. Roll out changes to one copy first and wait. Figure 1 is what that looks like.
How long it takes to recover matters as much as how often you fail
There is another way to look at the same number. Availability is the time you are up, divided by the time you are up plus the time you spend recovering.
Say your database primary fails twice a year. If a standby takes over automatically in 60 seconds, that is 2 minutes of downtime a year, and you are comfortably inside four nines. If instead a human has to be paged, wake up, log in and promote the standby by hand, call it 30 minutes each time. That is an hour a year, and four nines (53 minutes) is already gone from this one cause.
Same hardware. Same number of failures. The only difference was whether recovery needed a person. That is why four nines and above always means automatic failover.
Three words you will be asked about
SLI (indicator) is the thing you measure. “The share of requests that returned successfully in under 300 ms.”
SLO (objective) is the target you set for it. “99.9% over each month.”
SLA (agreement) is a promise to a customer with a penalty attached. “Below 99.9%, we refund 10% of the bill.” The SLA is always looser than the SLO, so you have room.
From the SLO you get an error budget. At 99.9% a month you are allowed about 43 minutes of failure. That budget is something you spend. A risky deploy that causes 10 minutes of errors used a quarter of it. When the budget is gone, you stop shipping features and fix reliability until next month.
Speed has the same problem
A system can be up and still be useless, if it takes eight seconds to answer. So alongside availability you track latency, and the mistake everyone makes first is tracking the average.
Averages hide the people who are suffering. If 99 requests take 100 ms and one takes 10 seconds, the average is about 200 ms and looks healthy. One user in a hundred just waited ten seconds.
So you use percentiles. The p50 is the time half of requests beat. The p99 is the time 99 out of 100 requests beat, so it describes your slowest 1%. Good teams set targets on p99, not on the average.
Then the chain rule comes back. A page that makes many backend calls is slow if any one of them is slow.
Each call is slow only 1.0% of the time, which sounds rare. Across 20 calls it is not rare at all. The slowest 1% of your backend becomes a normal experience for your users, which is why p99 matters more than the average.
Set it to 100 calls at 1% and you get 63%. Most page loads are hitting your “rare” slow case. The search engine page is built around exactly this effect.
Break it
The design from Figure 1, with failures of increasing size. Notice which ones the copies absorb and which one they cannot.
Trade-offs
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| One instance, restart on failure | Cheapest and simplest. Nothing to keep in sync. | Every failure and every deploy is downtime. Realistically two nines. | Internal tools, batch jobs, anything where an hour of downtime costs nobody money. |
| Active and standby | Survives the loss of a machine or a zone with about a minute of downtime. | You pay for a second copy that sits idle, and a failover that is never tested will not work. | Databases and anything else that can only have one writer at a time. |
| Several active copies | Losing one is invisible, and the spare capacity is doing useful work instead of idling. | Only possible for stateless services, and each copy must be able to carry extra load. | App servers, API gateways, caches. This is the normal shape for everything stateless. |
| Several regions | Survives an entire region going dark, and users far away get faster responses. | Data now has to be copied across continents, which forces hard consistency choices. | Four nines and above, or users on more than one continent. Rarely worth it before then. |
Interview replay
Checkpoint
1. Roughly how much downtime a year does 99.9% availability allow?
2. A request must pass through five services, each 99.9% available. What is the path’s availability?
3. Three copies of a service, each 99% available, should give 99.9999%. You measure 99.9%. What is the most likely reason?
- 99.9% is 8.8 hours a year, or 43 minutes a month. Each extra nine divides the downtime by ten.
- In series, multiply the availabilities. Five services at 99.9% make a 99.5% path.
- In parallel, multiply the failure rates. Two 99% copies make 99.99%, but only if they fail separately.
- Four nines is 53 minutes a year. No human recovers that fast, so failover must be automatic.
Availability is the share of requests that succeed, and I quote it in nines: three nines is about nine hours of downtime a year, and each extra nine is ten times less. Two rules do most of the work. Components in series multiply, so five services at three nines give a path of only 99.5 percent, which means I keep the number of hard dependencies small and let optional ones fail quietly. Components in parallel multiply their failure rates, so two copies of a 99 percent server give four nines, but only if they fail independently. Real copies share deploys and config, so I spread them across zones and roll out changes gradually. Recovery time matters as much as failure rate, so anything aiming at four nines needs automatic failover. And I track latency by p99 rather than the average, because a page that makes many calls hits the slow tail most of the time.
