Distributed transactions
A travel app sells a trip as one purchase: a sleeper berth, a hotel room, paid from the app's
wallet. Three services, three databases. The wallet debits ₹4,200. The berth is held. Then the
hotel says it has no rooms left. The customer has been charged for a trip that does not exist,
and there is no ROLLBACK that reaches across three databases.
Inside one database this problem was solved decades ago. Across several, you pick one of two very different answers, and each costs something the other does not.
Wallet, seats and hotel are separate services with separate databases. The wallet has debited and the seat is held when the hotel refuses. How would you get the customer's money back and free the seat? What could go wrong while you do?
One action, several databases#
A transaction gives you all or nothing, but only inside one database, because only that database's log decides what committed. Once an action spans several databases, or several services that each own one, you need a protocol on top.
Before picking a protocol, ask the cheaper question: does this data need to be in separate
databases? If the wallet and the bookings always change together, putting them in one
Postgres turns this whole chapter into BEGIN ... COMMIT. A lot of distributed transaction
pain is a service boundary drawn in the wrong place, which the
monolith vs microservices article covers.
When they really must be separate, you have two options.
Two phase commit#
Two phase commit (2PC) makes several databases commit together. A coordinator runs it, and every database involved is a participant.
It gives you real atomicity: every database commits or none does, and nobody ever sees half of it. The problem is the gap between the two phases. A participant that has voted yes holds its locks and has promised to obey. If the coordinator dies before sending the decision, the participant cannot commit (the decision might have been abort) and cannot abort (it might have been commit). It waits, locks held, until the coordinator returns. That is why 2PC is called a blocking protocol.
Java's XA standard and Postgres's PREPARE TRANSACTION let you run 2PC across ordinary
databases. Postgres ships with it switched off (max_prepared_transactions defaults to 0),
which says a lot about how often it is a good idea between services.
Where 2PC is everywhere is inside distributed databases. Spanner and CockroachDB run 2PC across shards, but every participant and the coordinator are themselves Raft or Paxos groups. A coordinator that is a replicated log does not stay down when one machine dies, so the blocking problem mostly disappears. That design is walked through in Design a global database.
Sagas#
A saga gives up on atomicity and replaces it with a plan for undoing. The idea comes from a 1987 paper by Hector Garcia-Molina and Kenneth Salem. A long action becomes a sequence of local transactions, each committed in its own database straight away. Each step has a compensating action that semantically undoes it. If step 3 fails, run the compensations for steps 2 and 1, in reverse.
| Step | Local transaction | Compensation |
|---|---|---|
| 1 | Debit ₹4,200 from the wallet | Refund ₹4,200 |
| 2 | Hold berth B-42 | Release berth B-42 |
| 3 | Book the room | Cancel the booking |
Compensation is not rollback. Step 1 really committed. Money left the wallet, an SMS may have gone out, a report may have counted it. The refund is a new, visible transaction. The customer can see the debit and then the refund, and someone else searching for that train saw berth B-42 as taken for a few seconds. A saga has atomicity in the end and no isolation on the way.
Try both#
Pick the failure, then switch between 2PC and a saga. Watch where each service ends up, and read what the customer went through.
- 1Step 1: Wallet debits ₹4,200 and commits. The money has really left.
- 2Step 2: Seats holds berth B-42 and commits.
- 3Step 3: Hotel has no room. The step fails.
- 4Compensate in reverse: Seats releases B-42, then Wallet refunds ₹4,200. Each is a new local transaction, not a rollback.
The two differ in what they make you pay for. 2PC pays in locks and availability: everyone waits for the slowest participant, and a dead coordinator freezes the rows involved. A saga pays in visible intermediate states and code: every step needs a compensation, and the world can see the half-done trip.
Making a saga safe#
Most of the work in a saga is in the details.
Make every step and every compensation idempotent. The orchestrator will retry after timeouts (the exactly-once problem again), so "refund for booking 9913" must refund once no matter how often it arrives.
Order the steps by risk. Put the step most likely to fail first, so failures need little undoing. Put the step that cannot be undone, such as sending money to a partner or issuing the ticket, last. Once it succeeds, everything after it must be retried until it succeeds rather than compensated.
Use pending states. Hold the berth as HELD, not BOOKED, until the saga finishes. Other
readers can treat HELD differently, and a sweeper can release holds that are too old. This is
how you buy back some of the isolation the saga gave up.
Persist the saga's progress. Whatever runs the saga must record each completed step durably, or a crash leaves a booking half done with nobody remembering to finish or undo it. Each step is "local transaction plus event", which is the outbox once more.
- TemporalDurable workflows: saga orchestration with retries built in
- AWAWS Step FunctionsManaged state machines, often used for sagas
- Apache KafkaThe event bus for choreographed sagas
- Spanner2PC across Paxos groups, inside the database
- CockroachDB2PC across Raft groups, inside the database
- PostgreSQLPREPARE TRANSACTION, off by default
A UPI payment debits one bank and credits another, and no transaction spans both banks. The switch runs it as a sequence of steps with a reversal when the credit fails, and reconciles everything afterwards. When your app shows "payment pending, money will be refunded if not completed", you are looking at a saga waiting to find out whether to compensate. The full design is in Design UPI payments.
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Put it in one database | A real ACID transaction, and none of the rest of this chapter. | The data owners share a database, so you lose some team and scaling independence. | Always worth asking first. Many sagas exist only because of a service boundary in the wrong place. |
| Two phase commit between services | True atomicity and isolation: nobody sees a half-done state. | Locks held across network round trips, and a crashed coordinator blocks every participant. | Rarely between services. Inside a database built on consensus, where the coordinator cannot stay dead. |
| Orchestrated saga | No cross-service locks, each service stays available, and the flow is readable in one place. | A compensation for every step, visible intermediate states, and a workflow component to run. | Most multi-service business flows: orders, bookings, onboarding. |
| Choreographed saga | No coordinator, loose coupling between services. | Hard to see or change the whole flow, and failure paths spread across every service. | Short flows of two or three steps with few failure paths. |
Interview replay#
Checkpoint#
1. In two phase commit, a participant has voted yes and the coordinator crashes. Why can the participant not just abort after a timeout?
2. In a saga, step 2 has committed and step 3 fails. What happens to step 2?
3. Your saga charges a card, then issues an airline ticket that cannot be cancelled, then sends a confirmation email. Which order is right?
- First ask: one database? A boundary in the right place removes the problem.
- 2PC blocks when the coordinator dies after a yes vote. Fine only when the coordinator is a consensus group.
- Saga = local transactions + compensations. Atomic in the end, no isolation on the way.
- Riskiest step first, irreversible step last, everything idempotent, progress persisted.
A database transaction only covers one database, so an action across several services needs a protocol. First I check whether the data could just live in one database. If not, there are two options. Two phase commit asks every participant to prepare and vote, then the coordinator logs the decision and tells everyone. It is truly atomic, but participants hold locks while they wait, and if the coordinator dies after they vote yes, they are stuck. That is why it is used inside databases like Spanner and CockroachDB, where the coordinator is a replicated consensus group, and rarely between services. Between services I use a saga: a sequence of local transactions, each with a compensating action run in reverse on failure. It is atomic in the end but not isolated, so I use pending states, put the riskiest step first and the irreversible one last, make every step idempotent, and run it on a durable orchestrator like Temporal.