Replication and quorums
Read firstCAP theorem
Every copy of your data exists for one of two reasons: so you can survive losing a machine, or so you can serve more reads. Those are different jobs with different failure modes, and conflating them is how people end up surprised that their read replica served a profile edit that had already been saved.
The whole subject reduces to one question. When a write lands on one copy and a read hits another, what does the reader see?
You write to the leader and immediately read from a follower, and the value is old. Is that a bug? What would you need to know to answer?
Leader and follower#
One replica accepts writes. The others copy from it and serve reads. This is what almost every relational database does by default and it is the right starting point.
Follower B in Figure 1 is not broken. Replication lag is normal, it varies constantly, and it spikes exactly when you least want it to: during a large write, a schema change, or a burst of traffic. Any design that assumes followers are current is a design that works in testing and fails in production.
Synchronous or not#
Quorums#
Leaderless systems like Dynamo, Cassandra and Riak drop the leader entirely. Write to several replicas, read from several replicas, and make the two sets overlap.
The rule is one line of arithmetic. Drag the sliders and watch the sets move.
The picture is the proof. Write to the first W nodes, read from the last R, and the number
they share is W + R - N. When that is at least one, every read touches a node that saw
the last write, so the newest value is always in the response set and a version number
picks it out. When it is zero or less, the sets can miss each other entirely.
R + W greater than N gives you a read that sees the latest completed write. It does not give you transactions, it does not order concurrent writes to the same key, and it does not prevent two clients writing at the same moment and producing two conflicting versions. That is a separate problem, solved with version vectors and either last write wins or application level merging.
Failover, and the two ways it hurts#
When the leader dies, something has to promote a follower. Both parts of that are hard.
Deciding it is dead. The only evidence is silence, and silence is indistinguishable from a slow network. Set the timeout aggressively and you promote a leader that was merely busy. Set it generously and you are down for that long every time.
Two leaders at once. The old leader was not dead, just unreachable, and it is still accepting writes from clients that can still see it. Now two nodes both believe they are the leader, and both are accepting conflicting writes to the same rows.
The defence is a fencing token: every leadership term gets a monotonically increasing number, and the storage layer rejects writes carrying an old one. The deposed leader tries to write, gets refused, and finds out it is no longer in charge. Without fencing, "we promote a follower" is a sentence that hides a data loss bug.
Consistency, said plainly#
CAP gets quoted more than it gets used. The useful version is short, and the CAP theorem page lets you make the choice yourself on four scenarios.
During a network partition you must choose between refusing requests and serving possibly stale data. There is no third option, and the choice is per operation rather than per system: a bank balance and an avatar image in the same product should answer differently.
The part people miss is what happens the rest of the time, which PACELC names. When there is no partition, you still trade latency against consistency, because a strongly consistent read means waiting for a quorum. That trade is live every single day, whereas partitions are rare, so it matters far more in practice.
Read your writes: you always see your own updates. Usually done by routing a user's reads to the leader for a short window after they write, which is cheap and solves the most visible symptom. Monotonic reads: you never see time go backwards, so pin a user to one replica. Consistent prefix: you never see an effect before its cause, which is what stops a reply appearing above the message it answers.
Break it#
Read every step as text
- healthy · Healthy. Lag under 10ms. Reads go to followers, writes to the leader, and nobody can tell the difference from a single machine.
- lag climbing · Degraded. A bulk import runs. Follower B falls seconds behind, because replication on most systems applies changes in a single stream and cannot parallelise past a certain point. Users who write and immediately read start seeing their own edits vanish.
- leader dies · Down. The leader is gone. Writes stop until a follower is promoted. If A is promoted, every write the leader had acknowledged but not yet shipped is lost, and with asynchronous replication that is a real window that grows with lag.
- split brain · Down. The old leader was only unreachable. Two nodes are now accepting writes to the same rows and both believe they are correct. Reconciling this loses acknowledged data no matter how it is resolved.
- fenced · Degraded. With fencing tokens the deposed leader’s writes are rejected by the storage layer the moment a new term begins. It discovers it has been replaced instead of quietly corrupting data. Writes still paused during failover, which is the cost you accept for not losing them.
Trade-offs#
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Asynchronous replication | Writes never wait on a replica, and a slow or dead follower cannot stop them. | A leader failure loses every write it acknowledged but had not yet shipped. | Read scaling and analytics replicas, where the leader is not the only copy that matters. |
| Semi synchronous | Every acknowledged write exists on at least two machines, so one failure loses nothing. | Writes pay one extra round trip, and if no follower can confirm, writes stall. | The default for anything you would be unhappy to lose. Most managed databases offer it. |
| Quorum with R + W > N | No leader to fail over, and reads always see the latest completed write. | Every operation talks to several nodes, so latency tracks the slowest of them, and concurrent writes still conflict. | Multi region, high write availability, and a workload that tolerates conflict resolution. |
| Read your writes routing | Removes the symptom users actually notice, at almost no cost. | Extra load on the leader for a short window after each write, and it needs sticky routing. | Any system with follower reads and users who edit their own data, which is most of them. |
Interview replay#
Checkpoint#
1. N = 5, W = 2, R = 2. Can a read miss the latest completed write?
2. What does a fencing token prevent?
3. Which guarantee stops a user seeing their own profile edit disappear?
- N=3, W=2, R=2 is the workhorse. R + W > N is the only formula worth memorising.
- Synchronous replication costs a round trip on every write and buys you zero data loss on failover.
- Asynchronous replication loses whatever had not shipped when the leader died. Alert on replication lag, in bytes.
- Failover costs twice: the pause while a new leader is chosen, and the writes the old leader accepted alone.
Replicas exist either for durability or for read scaling, and those are different jobs. I would default to a leader with semi synchronous replication, so every acknowledged write is on at least two machines without waiting for every replica. Followers lag, always, and that lag spikes exactly under load, so any read from a follower is a read from slightly in the past. The symptom users notice is their own edit disappearing, and the fix is read your writes: route them to the leader for a few seconds after a write. On failover the risk is split brain, where the old leader was only partitioned and is still taking writes, so I would use fencing tokens that let the storage layer reject a stale leadership term. If I needed leaderless writes I would use a quorum with W plus R greater than N, and I would say up front that this buys latest-write reads and not transactions or conflict resolution.