Replication and quorums
Every copy of your data exists for one of two reasons: so you can survive losing a machine, or so you can serve more reads. Those are different jobs with different failure modes, and conflating them is how people end up surprised that their read replica served a profile edit that had already been saved.
The whole subject reduces to one question. When a write lands on one copy and a read hits another, what does the reader see?
You write to the leader and immediately read from a follower, and the value is old. Is that a bug? What would you need to know to answer?
Leader and follower
One replica accepts writes. The others copy from it and serve reads. This is what almost every relational database does by default and it is the right starting point.
Follower B in Figure 1 is not broken. Replication lag is normal, it varies constantly, and it spikes exactly when you least want it to: during a large write, a schema change, or a burst of traffic. Any design that assumes followers are current is a design that works in testing and fails in production.
Synchronous or not
Quorums
Leaderless systems like Dynamo, Cassandra and Riak drop the leader entirely. Write to several replicas, read from several replicas, and make the two sets overlap.
The rule is one line of arithmetic. Drag the sliders and watch the sets move.
The picture is the proof. Write to the first W nodes, read from the last R, and the number
they share is W + R - N. When that is at least one, every read touches a node that saw
the last write, so the newest value is always in the response set and a version number
picks it out. When it is zero or less, the sets can miss each other entirely.
R + W greater than N gives you a read that sees the latest completed write. It does not give you transactions, it does not order concurrent writes to the same key, and it does not prevent two clients writing at the same moment and producing two conflicting versions. That is a separate problem, solved with version vectors and either last write wins or application level merging.
Failover, and the two ways it hurts
When the leader dies, something has to promote a follower. Both parts of that are hard.
Deciding it is dead. The only evidence is silence, and silence is indistinguishable from a slow network. Set the timeout aggressively and you promote a leader that was merely busy. Set it generously and you are down for that long every time.
Two leaders at once. The old leader was not dead, just unreachable, and it is still accepting writes from clients that can still see it. Now two nodes both believe they are the leader, and both are accepting conflicting writes to the same rows.
The defence is a fencing token: every leadership term gets a monotonically increasing number, and the storage layer rejects writes carrying an old one. The deposed leader tries to write, gets refused, and finds out it is no longer in charge. Without fencing, “we promote a follower” is a sentence that hides a data loss bug.
Consistency, said plainly
CAP gets quoted more than it gets used. The useful version is short.
During a network partition you must choose between refusing requests and serving possibly stale data. There is no third option, and the choice is per operation rather than per system: a bank balance and an avatar image in the same product should answer differently.
The part people miss is what happens the rest of the time, which PACELC names. When there is no partition, you still trade latency against consistency, because a strongly consistent read means waiting for a quorum. That trade is live every single day, whereas partitions are rare, so it matters far more in practice.
Read your writes: you always see your own updates. Usually done by routing a user’s reads to the leader for a short window after they write, which is cheap and solves the most visible symptom. Monotonic reads: you never see time go backwards, so pin a user to one replica. Consistent prefix: you never see an effect before its cause, which is what stops a reply appearing above the message it answers.
Break it
Trade-offs
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Asynchronous replication | Writes never wait on a replica, and a slow or dead follower cannot stop them. | A leader failure loses every write it acknowledged but had not yet shipped. | Read scaling and analytics replicas, where the leader is not the only copy that matters. |
| Semi synchronous | Every acknowledged write exists on at least two machines, so one failure loses nothing. | Writes pay one extra round trip, and if no follower can confirm, writes stall. | The default for anything you would be unhappy to lose. Most managed databases offer it. |
| Quorum with R + W > N | No leader to fail over, and reads always see the latest completed write. | Every operation talks to several nodes, so latency tracks the slowest of them, and concurrent writes still conflict. | Multi region, high write availability, and a workload that tolerates conflict resolution. |
| Read your writes routing | Removes the symptom users actually notice, at almost no cost. | Extra load on the leader for a short window after each write, and it needs sticky routing. | Any system with follower reads and users who edit their own data, which is most of them. |
Interview replay
Checkpoint
1. N = 5, W = 2, R = 2. Can a read miss the latest completed write?
2. What does a fencing token prevent?
3. Which guarantee stops a user seeing their own profile edit disappear?
Replicas exist either for durability or for read scaling, and those are different jobs. I would default to a leader with semi synchronous replication, so every acknowledged write is on at least two machines without waiting for every replica. Followers lag, always, and that lag spikes exactly under load, so any read from a follower is a read from slightly in the past. The symptom users notice is their own edit disappearing, and the fix is read your writes: route them to the leader for a few seconds after a write. On failover the risk is split brain, where the old leader was only partitioned and is still taking writes, so I would use fencing tokens that let the storage layer reject a stale leadership term. If I needed leaderless writes I would use a quorum with W plus R greater than N, and I would say up front that this buys latest-write reads and not transactions or conflict resolution.
