Active-passive vs active-active vs cells
In December 2021 a large part of AWS's us-east-1 region degraded for most of a working day, and a long list of apps went down with it, including ones that had "multi-region" on their architecture diagrams. Many had a standby region. Few could actually move to it: the failover had never been run end to end, the runbook needed services that were themselves in the broken region, or the standby did not have the capacity to take the whole load.
Surviving a regional failure is a choice between three shapes. Active-passive runs everything in one region and keeps a copy ready elsewhere. Active-active serves real traffic from several regions at once. Cell-based architecture splits the system into many independent copies that each serve a slice of users, so any failure, a region, a bad deploy, a poisoned request, hits one slice. They differ less in their diagrams than in what they cost you in data, in money and in engineering.
Your payments API runs in Mumbai with a standby in Hyderabad, replicated asynchronously with about 2 seconds of lag. Mumbai goes down. How much data could you lose, how long until you are serving again, and what decides both?
Two numbers decide everything#
Before picking a shape, agree on two numbers with the business, because they decide which shapes are even possible:
- RPO, recovery point objective: how much recent data you can afford to lose. With asynchronous replication, your RPO is your replication lag at the moment of failure. Zero RPO needs every write acknowledged by more than one region before it is confirmed, which adds the round trip between regions to every write.
- RTO, recovery time objective: how long you can be down. It covers noticing the failure, deciding to fail over, promoting the standby, moving traffic, and warming caches. A human in the loop costs at least minutes. DNS records cached by resolvers cost their TTL, and some resolvers ignore short TTLs.
The physics sets a floor. Mumbai to Singapore is roughly 60 ms round trip, Mumbai to Frankfurt around 120 ms, and Mumbai to the US east coast over 200 ms. A write that must be confirmed in two regions pays at least one of those on every request. That is why "zero data loss and low latency everywhere" is a request to have a conversation, not a requirement.