Design a globally consistent database
Most distributed databases ask you to give something up. Availability, or transactions, or knowing whether the row you just read is current. The interesting question is whether you can have SQL transactions across continents with a guarantee strong enough that nobody has to think about replicas at all.
Spanner’s answer, and the reason it is worth studying, is that the hard part is not consensus. Consensus is solved. The hard part is time: two machines cannot agree on what “now” means, and every strong ordering guarantee across regions eventually reduces to that.
Transaction A commits in Mumbai. Transaction B starts in Frankfurt one millisecond later and reads the same row. Should B see A's write, and how could Frankfurt possibly know?
What you are building
- SQL transactions across rows that live in different regions. Serialisable, not eventually consistent.
- External consistency. If A commits before B starts in real time, every observer sees A before B. This is the strongest ordering guarantee and it is the product.
- Survive losing a region. Without data loss and without a manual failover decision.
- Reads that do not block writes. Multi version storage, so a read never takes a lock.
- Data in at least three regions.
- Reads served from a nearby replica where possible.
- Commit latency measured in tens of milliseconds, not hundreds.
- No lost writes on the failure of any single region.
- Query planning and optimisation, which is a whole discipline and orthogonal to the distribution problem.
- Schema change coordination, which is genuinely hard and would take the whole session.
- Analytical queries. This is a transactional store, and mixing the two is a separate design.
The problem is time
Drag the clock uncertainty slider. At 7ms, which is what you get with GPS receivers and atomic clocks in every datacentre, commit wait costs about 14ms and is a rounding error next to the network. At 100ms, which is what ordinary NTP gives you, commit wait dominates everything and the design becomes unusable. That is why this system exists inside a company that could put atomic clocks in its buildings: tightening the clock bound is what makes the guarantee affordable.
That slider is the whole insight. Every strong consistency guarantee across regions is paid for in either coordination or waiting, and the cost of waiting is proportional to how badly you know the time.
The building blocks
Two layers, and keeping them separate is what makes this tractable. Within a partition, consensus replicates a log across regions and elects a leader, which is Paxos or Raft and is well understood. Across partitions, two phase commit coordinates those groups.
The reason two phase commit is acceptable here, when it was the wrong answer for the payment rail, is that every participant is a fault tolerant consensus group inside one operator’s control. A 2PC participant that can crash and block everyone forever is unacceptable. A participant that survives losing a machine or a region, and is operated by the same team, is a different proposition entirely.
Each partition replicates across regions and commits when a majority acknowledges. With five replicas a write survives two failures and the commit latency is the third fastest replica rather than the slowest, which is why an odd number and geographic spread both matter.
Deep dive one: why timestamps are the whole problem
Serialisable transactions need an order. The natural way to order them is by commit timestamp. But a timestamp comes from a clock, and two machines never agree.
If Mumbai’s clock is 5ms ahead of Frankfurt’s, a transaction that genuinely commits later in Frankfurt can carry an earlier timestamp. Order the log by timestamp and you have inverted causality: someone reading at that timestamp sees the later write and not the earlier one.
The fix is counterintuitive and elegant. Stop pretending the clock is a point and treat it
as an interval. The time service returns not a timestamp but a range, [earliest, latest],
and guarantees the true time lies inside it. Then, having chosen a commit timestamp, the
transaction waits until that timestamp is definitely in the past everywhere before
releasing its locks.
That is external consistency, and the trade is stated plainly: you buy a global ordering guarantee by making every commit wait a few milliseconds. The tighter your clocks, the cheaper the guarantee. This is why the same design with ordinary NTP clocks, where uncertainty is hundreds of milliseconds, is not worth building.
No message is exchanged between the two transactions, and no global sequencer exists. People often assume strong global ordering needs a central authority handing out numbers, which would be a bottleneck and a single point of failure. The wait replaces the coordination, and it scales because every node waits independently.
Deep dive two: reads that never block
Every version of every row is kept with the timestamp that wrote it. A read at timestamp T sees exactly the versions committed at or before T, and takes no locks at all, so readers never block writers and writers never block readers.
| key | bytes | PK | Ordered, so ranges of keys form the partitions. |
| timestamp | bigint | PK | Descending, so the newest version is found first. Assigned at commit, after commit wait. |
| value | bytes | A tombstone marks a delete, since nothing is removed in place. | |
| lock | optional | Held only by writers, and only between prepare and commit. Readers never take one. |
This gives you the feature people find most surprising. A read only transaction can pick a timestamp a few seconds old and be served entirely by the nearest replica, in the same region, with no cross region round trip and no leader involvement, while still seeing a globally consistent snapshot. Most read traffic in these systems runs this way, which is what keeps the design usable despite the expensive writes.
Deep dive three: what a cross partition commit actually costs
Break it
The API
{
"mutations": [...],
"readSet": [{ "key": "user:4471", "version": 1756109981004 }]
}Trade-offs
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Bounded clock uncertainty plus commit wait | External consistency with no global sequencer and no messages between unrelated transactions. | Every commit waits about twice the uncertainty bound, and you must own the hardware to keep that bound small. | You control the datacentres. With ordinary NTP the wait dominates and the design stops being worth it. |
| Two phase commit over consensus groups | Real cross partition transactions, with participants that survive machine and region failure. | Roughly twice the round trips of a single partition commit, with locks held throughout. | Participants are fault tolerant and under one operator. Never across organisations. |
| Multi version storage | Readers never block writers, and any replica can serve a consistent snapshot at a past timestamp with no coordination. | Storing old versions and garbage collecting them, which bounds how far back you can read. | Any system where read traffic dominates and you want it served locally. |
| Stale reads as a first class option | Most reads become local and single digit milliseconds while remaining globally consistent. | The caller has to reason about how much staleness is acceptable for their case. | Always offer it. The alternative is every read paying for a guarantee most of them do not need. |
Interview replay
Checkpoint
1. What does commit wait actually achieve?
2. Why is two phase commit reasonable here but not across two banks?
3. Clock uncertainty widens from 7ms to 300ms. What happens to correctness?
The hard part is not consensus, it is time. Within a partition, a consensus group replicates across regions and elects a leader, and across partitions two phase commit coordinates those groups, which is acceptable here precisely because every participant is fault tolerant and under one operator, unlike two banks where a blocked participant has no recovery path. Ordering comes from timestamps, and since clocks disagree the time service returns an interval rather than an instant: a transaction picks the upper bound as its commit timestamp and then waits until that timestamp is definitely past everywhere before becoming visible. That gives external consistency with no global sequencer and no messages between unrelated transactions, at a cost of about twice the clock uncertainty per commit, which is why tightening the clock bound with dedicated hardware is what makes the whole thing affordable. Storage is multi version, so readers never take locks and a read at a slightly past timestamp is served by the nearest replica with no cross region traffic while still seeing a consistent global snapshot, which is how most read traffic should actually run. And when clock uncertainty widens, commit wait grows automatically, so the system gets slower rather than wrong.
