Design a UPI style payment rail
Someone scans a QR code at a tea stall, taps a PIN, and twelve rupees moves between two accounts at two different banks in about two seconds. Both banks are separate companies with separate databases and separate outage schedules. Neither of them can roll back the other one.
That is the whole problem. Everything else in this design is a consequence of it.
This page uses UPI as the reference because it is the largest system of its kind and because most people reading this have used it three times today. The design generalises: any rail that moves money between institutions you do not control ends up in roughly the same shape.
Before you scroll: the payer's bank confirms the debit, then the payee's bank fails to credit. The money has left one account and arrived nowhere. What should the system do, and what must it never do?
What you are building
- A push payment from one bank account to another. Payer initiates, two different banks, real time.
- Resolve an address like alice@bank to an account. Nobody types account numbers, and the address is the product.
- Guarantee the money is either in one account or the other. Never in neither, never in both, and eventually never in limbo.
- Reversals when a leg fails. Compensation, not rollback. There is no rollback across two banks.
- Under 3 seconds end to end at p95, because the customer is standing at a counter.
- Correctness beats availability. A payment that cannot be completed safely must fail, not guess.
- Every transaction is auditable and reconcilable to the paisa.
- One slow bank must not degrade payments between the other banks.
- Mandates, autopay and recurring collections. Same rail, different state machine.
- Onboarding, KYC and device binding. Large, regulated, and its own design question.
- Fraud scoring. Sits alongside the flow and would take the whole session.
- Card networks. Different settlement model, and mixing the two just confuses the conversation.
The numbers
Published monthly volumes for UPI have been well north of 15 billion transactions a month, and the figure keeps climbing, so treat any specific number as a checkpoint rather than a constant. What matters is the shape of the arithmetic.
Four ledger writes per transaction, because each leg is a double entry pair. That turns 52,083 payments a second into 208,333 appends a second, and it is the number that decides the ledger design.
The multipliers are not decoration. Payment traffic in India is spiky in ways an average hides completely: salary day, the first of the month, festival evenings, the last hour before a bill deadline. Designing for the average is designing to fail on the days that matter most.
The pieces
Four parties, and it matters that three of them are run by different organisations.
The switch is the interesting box. It routes, it holds the state machine, and it decides what happens when something goes wrong. What it does not do is hold funds. Nobody’s money sits in the middle, which is why a failed payment cannot be undone with a rollback and has to be undone with a second, opposite payment.
The happy path
Step through it. Watch how many separate systems have to agree, and notice that no two of them are inside the same transaction.
The textbook answer to two databases is two phase commit. It is the wrong answer here and you should be able to say why in one sentence: two phase commit requires every participant to hold locks until the coordinator decides, and no bank is going to let an external switch hold a lock on its ledger while a second bank is slow or offline. The blocking property that makes 2PC correct is exactly what makes it unusable across organisations.
Deep dive one: the state machine is the design
Because there is no rollback, the transaction’s state is the only thing standing between a customer and a lost payment. Every transition is written down before the action it describes is attempted, never after.
DEBITED is the state that defines the system. A transaction sitting in DEBITED is money that has left one account and not arrived anywhere, and every design decision below exists to make sure nothing stays there.
| txn_id | char(35) | PK | Generated by the initiator and unique for all time. This is what makes retries safe. |
| idempotency_key | char(64) | UQ | A unique index here is the entire duplicate payment defence. |
| payer_vpa | varchar(255) | Resolved to a bank and account at initiation, then frozen for the life of the transaction. | |
| payee_vpa | varchar(255) | ||
| amount_paise | bigint | Integer paise. Never a float. Floating point money is how you get a rounding difference nobody can explain. | |
| state | enum | IDX | INITIATED, DEBITED, SUCCESS, REVERSING, REVERSED, FAILED. |
| state_changed_at | timestamp | IDX | Indexed with state, so a sweeper can find everything stuck in DEBITED for over 30 seconds. |
The ledger is separate, append only, and double entry. Every movement writes two rows that sum to zero, so any imbalance is a bug you can actually detect.
| entry_id | bigint | PK | Time ordered, so the log reads in the order things happened. |
| txn_id | char(35) | FKIDX | Every entry traces back to one transaction. |
| account_id | bigint | IDX | |
| direction | enum('DR','CR') | Entries for one movement always come in pairs that sum to zero. | |
| amount_paise | bigint | ||
| posted_at | timestamp |
Deep dive two: the timeout that is not a failure
Here is the case that separates people who have worked on payments from people who have read about them. The switch sends a debit and gets nothing back.
A timeout tells you the response did not arrive. It tells you nothing at all about whether the debit happened.
Three rules come out of that sequence, and they are worth saying in exactly these words:
Never retry a money movement. Retry the question about it. A retry can double debit. A status check cannot. Every request carries a transaction id the bank will recognise, and asking twice is free.
Write the intent before the action. The record that says a debit is being attempted has to be durable before the debit is sent. Otherwise a crash at the wrong moment leaves money moving with nothing on your side that knows about it.
No transaction gets to stay in limbo. A job sweeps for anything stuck in DEBITED or DEBIT_PENDING past a threshold, resolves it by status check, and reverses if it must. This job is not a nice to have. It is the component that turns “we could lose a payment” into “we take up to a few minutes to fix a payment”.
Deep dive three: reconciliation, because the ledger will not agree
Two organisations, two databases, millions of transactions. Their idea of what happened and yours will differ, in small numbers, every single day. Not because anyone is wrong, but because a message got lost at the boundary of a batch, or a status check landed between two commits.
So both sides exchange a settlement file for the day and compare, line by line. Three categories come out:
Matched, which is nearly everything. In our records as successful, in theirs as successful, same amount.
In ours but not theirs, which means we told a customer their payment succeeded and the bank never saw it. This is the dangerous one. It gets investigated by a human.
In theirs but not ours, which usually means a response we never received for a transaction that did complete. Our state machine already caught most of these through status checks, and the remainder get corrected.
Any time you write to two systems you do not control together, you need this same trio: an idempotency key so retries are safe, a state machine that never leaves work in an ambiguous state, and a reconciliation pass that assumes the two sides will disagree. Ordering systems, inventory systems and anything talking to a third party API all have a version of this problem, and most of them handle it worse than a payment rail does.
Break it
The failure that actually happens in production is not the switch falling over. It is one bank getting slow.
The general form of that fix is worth naming, because it comes up in every design with third party dependencies. A shared resource pool converts one dependency’s failure into everyone’s failure. Bounded per dependency pools, plus a breaker that fails fast, converts it back into a local failure.
The API
{
"txnId": "UPI250825143207891",
"payerVpa": "bob@bank",
"payeeVpa": "alice@bank",
"amountPaise": 50000,
"idempotencyKey": "9f3c1e..."
}{ "reason": "BENEFICIARY_CREDIT_FAILED" }Trade-offs
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Saga with compensation over two phase commit | No participant holds a lock waiting on another organisation, so one slow bank cannot freeze everyone else. | There is a real window where money has left one account and not arrived, and you have to build the machinery that closes it. | Any transaction crossing an organisational boundary. 2PC is for systems under one operator. |
| Status check over retry | Cannot double debit, no matter how many times the network eats a response. | Slower recovery, and every downstream party has to implement a status endpoint that is actually correct. | Always, for anything that moves money or has any other side effect you cannot take back. |
| Append only ledger over mutable balances | Complete history, reconciliation is possible at all, and every correction is itself auditable. | Balances become a computed value, so you need snapshots to avoid summing a billion rows. | Anything financial. The audit requirement makes this less of a choice than it looks. |
| Per bank pools and breakers over one shared pool | One failing dependency stays one failing dependency. | Lower total utilisation, since each pool has to be sized for its own peak with headroom. | Any fan out to third parties with independent failure modes, which is most integrations. |
Interview replay
Checkpoint
1. The debit request to the remitter bank times out. What is the correct next action?
2. Why is two phase commit the wrong tool across two banks?
3. Bank B starts timing out and payments to healthy Bank A begin failing too. What is the cause?
This is two transactions at two organisations that cannot share a transaction, so the design is a saga with compensation rather than two phase commit, because no bank will hold locks waiting on another bank. A central switch owns routing and the state machine: resolve the payee address, debit the remitter, credit the beneficiary, and write every state change durably before attempting the action it describes. The state that matters is DEBITED, meaning money has left and not arrived, and everything else exists to make sure nothing stays there. On a timeout I never retry the debit, I status check the same transaction id, because a retry can debit twice. If the credit genuinely fails I append a reversal rather than editing anything, since the ledger is double entry and append only. A sweeper resolves anything stuck, and a daily reconciliation against the bank's own file catches what is left. The failure I would design for hardest is one slow bank filling a shared connection pool and taking down payments to every other bank, which is a per bank pool and a circuit breaker.
