Distributed systems
The moment your program runs on two machines, three things you never had to think about start to matter. A request can vanish and you cannot tell whether it ran. Two clocks disagree about what time it is. Two machines can both believe they are in charge. This path is about those problems and the small set of ideas that tame them: majorities, logical clocks, leases with fencing tokens, idempotency, retry budgets, sagas, logs and CRDTs.
Why this path exists#
Every system design question quietly leans on these ideas. A design says "a consensus group per partition", "dedupe on the event id" or "take a lease", and moves on. In the interview, that is exactly where the follow-up questions start: what happens when the leader is cut off, why your Redis lock is not safe, what "strongly consistent" means exactly, why retries made the outage worse. Those questions separate a mid-level answer from a senior one, and at work they are where the outages come from.
The system design path uses these ideas in passing. This path explains them properly, one per chapter. Every chapter has something you can break (a cluster you can split, a retry storm you can start, a lock holder you can pause), an interview exchange or a "what would you do" round, a checkpoint, and the 60 second version to say out loud.
Read the first two chapters in order, because everything after them builds on partial failure and clocks. After that, each part stands on its own.
Why it is hard#
What a network actually does to your requests, and why "what time is it?" has no single answer.
| Chapter | What you learn | Why it matters |
|---|---|---|
| Partial failure | A timeout only means you stopped waiting. Five different server outcomes look the same, so retries must be made harmless. | The root of every "charged twice" bug, and the reason every other chapter exists. |
| Time and clocks | Wall vs monotonic clocks, skew, last write wins, Lamport and vector clocks, hybrid clocks, TrueTime. | "Why not just use timestamps?" is a standard follow-up, and last write wins silently loses data. |
Staying up#
How a small failure turns into a big outage, and the habits that stop it.
| Chapter | What you learn | Why it matters |
|---|---|---|
| Retries, backoff and circuit breakers | Retry storms, backoff with jitter, retry budgets, breakers, bulkheads, and failures that outlive their cause. | Real outages spread this way. Senior candidates are expected to see the storm coming. |
| Tail latency at scale | Why fanning out to 100 servers makes their rare slowness common, hedged requests, partial results, measuring honestly. | A favourite staff-level follow-up on search, feeds and any scatter-gather design. |
| Failure detection and gossip | Heartbeats, SWIM, phi accrual, and how gossip reaches every node in about log N rounds. | How Cassandra-style clusters know who is alive, and why that is not enough to pick a leader. |
Agreeing on things#
What a read is allowed to return, how machines agree on one history, and how exactly one of them gets to be in charge.
| Chapter | What you learn | Why it matters |
|---|---|---|
| Consistency models | Linearizable, sequential, causal and session guarantees, linearizable vs serializable, and what each database really offers. | "What does strongly consistent mean?" is asked constantly, and mixing up linearizable and serializable is a common slip. |
| Consensus with Raft | Majorities, terms, elections, log replication, reads, and what happens in a network split. | "How do the replicas agree?" comes up in every staff-level design. |
| Leader election, leases and locks | Leases, the paused lock holder, fencing tokens, and etcd vs Redis vs your own database. | "Why is a Redis lock not safe?" is a well known senior interview trap. |
Doing it once, everywhere#
Charging a card once when every hop can repeat itself, changing several databases as one action, keeping history, and letting replicas merge on their own.
| Chapter | What you learn | Why it matters |
|---|---|---|
| Exactly-once in practice | Idempotency keys, the outbox and dedupe tables, hop by hop, and where Kafka's guarantee stops. | Payments, wallets and orders depend on it, and interviewers probe "exactly-once" claims. |
| Distributed transactions | Two phase commit vs sagas, compensations, ordering steps by risk. | Every microservices design ends at "how do these services stay consistent?" |
| Event sourcing, CDC and the log | Events as the source of truth, projections and CQRS, optimistic appends, and change data capture. | Ledger and wallet designs, audit trails, and knowing when not to do it. |
| CRDTs | Counters, sets and documents that merge themselves, and the rules they can never keep. | Collaborative and offline apps, multi-region counters, and when you need consensus instead. |