Consensus with Raft

A Kubernetes control plane stores everything it knows (every pod, every deployment, every secret) in etcd, usually on three machines. Lose one and nothing happens. Lose two and kubectl apply hangs, even though the last etcd node is healthy and holds a complete copy of the data.

A healthy machine with all the data, refusing to accept a write. That is not a bug. It is the rule that stops a cluster from ever disagreeing with itself, and this chapter is about why the rule is right.

What consensus is for#

Replication with one primary and some replicas works well until the primary dies. Then someone has to pick a new primary, and the dangerous moment is when the old one comes back still believing it is in charge: two primaries, two histories, split brain. Doing that failover safely, automatically, is the job of consensus.

The shape is always the same. A group of machines (almost always 3 or 5) keeps a replicated log: an ordered list of commands like "set x = 2". Every machine applies the same commands in the same order, so every machine ends up in the same state. Consensus guarantees that once an entry is committed at a position in the log, no machine will ever have a different entry at that position, whatever crashes or splits happen.

You rarely write this yourself. You use a system that does it, and put your critical, small, must-agree data in it: which node leads, which shard lives where, the cluster's configuration.