Partial failure

Your checkout service calls the payment gateway to charge ₹1,499. Two seconds pass. Nothing comes back, and your HTTP client throws a timeout. Was the customer charged?

You do not know. You cannot know, from where you are standing. That one sentence is the whole difference between writing a program and building a distributed system, and every other chapter on this path is a way of living with it.

One machine fails whole. Many machines fail in parts.#

On your laptop, failure is mostly all or nothing. If the process crashes, every part of it stops at once. If the disk dies, you know. A function call either returns or throws, and it never "half returns".

Put the same program on two machines that talk over a network and a new kind of failure appears. One machine is fine, the other is down, and the first one cannot tell. Or both are fine and the cable between them is not. Or the other machine is up but so busy it will answer in nine seconds instead of nine milliseconds. This is partial failure: some parts of the system are working and some are not, and no single part sees the whole picture.

What the network does to a message#

Between two machines, a message can be:

Lost. A switch dropped it, a cable got unplugged, a firewall rule changed.

Delayed. It sat in a queue on a busy router, or behind a retransmission, and arrived three seconds later than usual. There is no upper limit on how late "late" can be.

Duplicated. Your client retried, a load balancer retried, or TCP reconnected and resent. The server sees the same request twice.

Reordered. Two messages sent one after the other arrive the other way round, because they took different paths or different connections.

The network does not tell you which one happened. It gives you silence, or it gives you an answer, and the answer itself might be a duplicate or a straggler from a request you already gave up on.

Engineers at Sun Microsystems wrote down the beliefs that get people into trouble here in 1994, as the fallacies of distributed computing. The first three cause most of the bugs: the network is reliable, latency is zero, bandwidth is infinite. The other five (the network is secure, topology does not change, there is one administrator, transport cost is zero, the network is the same everywhere) bite later, usually in production.

Silence has five meanings#

Back to the payment. The client sent "charge ₹1,499" and heard nothing for two seconds. Here is the trap: there are at least five different things that could have happened on the far side, and they all look identical from the client. Pick one, then pick what the client does, and watch the left half of the picture.

The left half never changes. That is the lesson. A timeout tells you that you stopped waiting. It does not tell you what the other side did.

So you have two choices, and both are wrong in some worlds:

• Give up, and in the worlds where the charge went through, you have taken money and told the customer it failed.
• Retry, and in the worlds where the charge went through, you have taken it twice.

The way out is to make the retry harmless: the server remembers the request by a key the client chose, and a second request with the same key gets the first answer back instead of a second charge. That is an idempotency key, and it is the single most common fix in this whole path. Once retries are harmless, the rule becomes simple: when in doubt, retry. The Exactly-once in practice chapter follows that idea through every hop of a real system.

Picking a timeout#

You need a timeout anyway, because without one a dead server holds your threads forever. The question is how long.

Start from what the other side normally does. If a service answers in 40 ms at the median and 300 ms at the 99th percentile, a timeout of around 1 second rarely fires on a healthy server and still frees you quickly when something is wrong. Then check the whole chain: if your own caller gives up after 2 seconds, there is no point in you waiting 5 seconds for something downstream.

Too short, and you declare healthy but slow servers dead. Every retry adds load to a server that was already struggling, which makes it slower, which causes more timeouts. This is how a small slowdown turns into an outage, and Retries, backoff and circuit breakers lets you watch it happen.

Too long, and a really dead server ties up your threads, connection pools and your users' patience for the full timeout on every call.

Clusters do the same thing between machines with heartbeats: each node sends "I am alive" every second or so, and a node that misses several in a row is suspected dead. Cassandra uses a smarter version called the phi accrual failure detector, which learns how regular a node's heartbeats normally are and raises its suspicion gradually instead of flipping at a fixed number. Either way it is a guess. A heartbeat can only ever say "I have not heard from you lately".

A node can stop without knowing it stopped#

The other machine is not the only thing that can freeze. Your own process can too.

A Java or Go program can pause for garbage collection. A virtual machine can be paused while the cloud provider moves it to another host. A laptop lid closes. The operating system swaps memory to disk. In every one of these, the process stops running for anything from a few milliseconds to many seconds, then carries on from exactly the next line, with no idea that time passed.

This is why "check, then act" is never safe across machines, and why locks need something more than an expiry time. The fix, a fencing token, is in Leader election, leases and locks.

The rules everyone assumes#

Papers and databases describe their guarantees against a system model, a statement of what can go wrong. The one almost everything in this path assumes:

• Nodes crash and recover. A node can stop at any moment and may come back later with whatever it wrote to disk, but not what was in memory.
• The network is partially synchronous. Most of the time messages arrive within some bound. Sometimes, for an unknown stretch, they do not. Algorithms must stay correct during the bad stretches, and are only expected to make progress during the good ones.
• Nodes do not lie. A node may be slow or dead, but it does not send deliberately wrong messages. Systems that must survive lying nodes (blockchains, some aircraft controls) solve the much harder Byzantine version of the problem, and they pay for it in messages and machines.

One famous result sits behind the "progress during the good stretches" wording. In 1985, Fischer, Lynch and Paterson proved that in a fully asynchronous network, where messages can be delayed without any bound, no algorithm can guarantee that machines always reach agreement if even one of them may crash. It is called FLP. Real systems get around it with timeouts: they stay safe no matter what, and they make progress whenever the network behaves for long enough. Raft, in Consensus with Raft, is built exactly this way.

Checkpoint#