Design a UPI style payment rail
Read firstMessage queuesTransactions and ACID
Someone scans a QR code at a tea stall, taps a PIN, and twelve rupees moves between two accounts at two different banks in about two seconds. Both banks are separate companies with separate databases and separate outage schedules. Neither of them can roll back the other one.
That is the whole problem. Everything else in this design is a consequence of it.
This page uses UPI as the reference because it is the largest system of its kind and because most people reading this have used it three times today. The design generalises: any rail that moves money between institutions you do not control ends up in roughly the same shape.
Step 1: Understand the problem#
Payments interviews reward paranoia. Every question below is really asking "what happens when this half succeeds", and the answers are what stop you designing a distributed transaction that cannot exist.
| You ask | They say | What it settles |
|---|---|---|
| Who owns the accounts, us or someone else? | Banks own them. We move instructions, not money. | The single most important answer. You cannot roll back another company's database, so every failure is undone with a compensating payment rather than an abort. |
| What is the peak rate, and when does it happen? | Tens of thousands a second, spiking hard on salary day and festival evenings. | Capacity planned for the spike, not the average, and a queue in front of anything that talks to a bank, because banks do not scale on your schedule. |
| Is it acceptable to debit twice under retry? | Absolutely not. Ever. | An idempotency key on every leg, generated once by the payer app and carried end to end. Retries are certain, so exactly once has to come from the key rather than from hope. |
| What if a bank does not answer at all? | Treat it as unknown, not as failed. | A third outcome besides success and failure. Timeouts get resolved by asking the bank later, and never by guessing, which is the subject of a whole deep dive. |
| How fast does the payer need an answer? | Under a few seconds, or they pay in cash instead. | A synchronous request path with hard deadlines, and everything that can be done afterwards being done afterwards. |
| Who decides what actually happened, in a dispute? | The end of day settlement file. Not our database. | Reconciliation as a first class part of the design rather than an operations afterthought. Your ledger and theirs will disagree, daily, and the process for that is the product. |
What you are building#
- A push payment from one bank account to another. Payer initiates, two different banks, real time.
- Resolve an address like alice@bank to an account. Nobody types account numbers, and the address is the product.
- Guarantee the money is either in one account or the other. Never in neither, never in both, and eventually never in limbo.
- Reversals when a leg fails. Compensation, not rollback. There is no rollback across two banks.
- Under 3 seconds end to end at p95, because the customer is standing at a counter.
- Correctness beats availability. A payment that cannot be completed safely must fail, not guess.
- Every transaction is auditable and reconcilable to the paisa.
- One slow bank must not degrade payments between the other banks.
- Mandates, autopay and recurring collections. Same rail, different state machine.
- Onboarding, KYC and device binding. Large, regulated, and its own design question.
- Fraud scoring. Sits alongside the flow and would take the whole session.
- Card networks. Different settlement model, and mixing the two just confuses the conversation.
Back of the envelope#
Published monthly volumes for UPI have been well north of 15 billion transactions a month, and the figure keeps climbing, so treat any specific number as a checkpoint rather than a constant. What matters is the shape of the arithmetic.
Four ledger writes per transaction, because each leg is a double entry pair. That turns 52,083 payments a second into 208,333 appends a second, and it is the number that decides the ledger design.
The multipliers are not decoration. Payment traffic in India is spiky in ways an average hides completely: salary day, the first of the month, festival evenings, the last hour before a bill deadline. Designing for the average is designing to fail on the days that matter most.
Step 2: Propose the high level design#
The API#
{
"txnId": "UPI250825143207891",
"payerVpa": "bob@bank",
"payeeVpa": "alice@bank",
"amountPaise": 50000,
"idempotencyKey": "9f3c1e..."
}{ "reason": "BENEFICIARY_CREDIT_FAILED" }The whole system on one whiteboard#
Four parties, and it matters that three of them are run by different organisations.
Walking Figure 1:
- The payer's app sends one request carrying an idempotency key it generated itself. Everything downstream is keyed off that.
- The switch resolves the payee's UPI id to a real account at a real bank. This lookup is why you can pay a phone number instead of an account number.
- Risk and limit checks run inline. They have a deadline measured in tens of milliseconds, because the whole payment has about two seconds.
- The debit goes to the payer's bank and the credit to the payee's bank. These are two separate companies, two separate calls, and no shared transaction.
- Every transition is written before it is attempted, so the state machine is always ahead of reality rather than behind it. If the switch dies mid payment, the row says what it was trying to do.
- The ledger is append only and double entry, which is how a disputed payment three months later has an answer.
- A sweeper picks up anything stuck in an unknown state and resolves it by asking the bank, never by guessing.
- And every night, reconciliation compares the ledger with what the banks say happened. The two will disagree. Having a process for that disagreement is the difference between a payment system and a demo.
The switch is the interesting box. It routes, it holds the state machine, and it decides what happens when something goes wrong. What it does not do is hold funds. Nobody's money sits in the middle, which is why a failed payment cannot be undone with a rollback and has to be undone with a second, opposite payment.
Step 4 is two calls to two different companies. The debit succeeds and the credit fails. Money has left one account and arrived nowhere. What should the switch do, and what must it never do?
The happy path#
Step through it. Watch how many separate systems have to agree, and notice that no two of them are inside the same transaction.
The textbook answer to two databases is two phase commit. It is the wrong answer here and you should be able to say why in one sentence: two phase commit requires every participant to hold locks until the coordinator decides, and no bank is going to let an external switch hold a lock on its ledger while a second bank is slow or offline. The blocking property that makes 2PC correct is exactly what makes it unusable across organisations.
Step 3: Design deep dive#
The state machine is the design#
Because there is no rollback, the transaction's state is the only thing standing between a customer and a lost payment. Every transition is written down before the action it describes is attempted, never after.
DEBITED is the state that defines the system. A transaction sitting in DEBITED is money that has left one account and not arrived anywhere, and every design decision below exists to make sure nothing stays there.
| txn_id | char(35) | PK | Generated by the initiator and unique for all time. This is what makes retries safe. |
| idempotency_key | char(64) | UQ | A unique index here is the entire duplicate payment defence. |
| payer_vpa | varchar(255) | Resolved to a bank and account at initiation, then frozen for the life of the transaction. | |
| payee_vpa | varchar(255) | ||
| amount_paise | bigint | Integer paise. Never a float. Floating point money is how you get a rounding difference nobody can explain. | |
| state | enum | IDX | INITIATED, DEBITED, SUCCESS, REVERSING, REVERSED, FAILED. |
| state_changed_at | timestamp | IDX | Indexed with state, so a sweeper can find everything stuck in DEBITED for over 30 seconds. |
The ledger is separate, append only, and double entry. Every movement writes two rows that sum to zero, so any imbalance is a bug you can actually detect.
| entry_id | bigint | PK | Time ordered, so the log reads in the order things happened. |
| txn_id | char(35) | FKIDX | Every entry traces back to one transaction. |
| account_id | bigint | IDX | |
| direction | enum('DR','CR') | Entries for one movement always come in pairs that sum to zero. | |
| amount_paise | bigint | ||
| posted_at | timestamp |
The timeout that is not a failure#
Here is the case that separates people who have worked on payments from people who have read about them. The switch sends a debit and gets nothing back.
A timeout tells you the response did not arrive. It tells you nothing at all about whether the debit happened.
Three rules come out of that sequence, and they are worth saying in exactly these words:
Never retry a money movement. Retry the question about it. A retry can double debit. A status check cannot. Every request carries a transaction id the bank will recognise, and asking twice is free.
Write the intent before the action. The record that says a debit is being attempted has to be durable before the debit is sent. Otherwise a crash at the wrong moment leaves money moving with nothing on your side that knows about it.
No transaction gets to stay in limbo. A job sweeps for anything stuck in DEBITED or DEBIT_PENDING past a threshold, resolves it by status check, and reverses if it must. This job is not a nice to have. It is the component that turns "we could lose a payment" into "we take up to a few minutes to fix a payment".
Reconciliation, because the ledger will not agree#
Two organisations, two databases, millions of transactions. Their idea of what happened and yours will differ, in small numbers, every single day. Not because anyone is wrong, but because a message got lost at the boundary of a batch, or a status check landed between two commits.
So both sides exchange a settlement file for the day and compare, line by line. Three categories come out:
Matched, which is nearly everything. Successful in our records, successful in theirs, same amount.
In ours but not theirs. We told a customer their payment succeeded and the bank never saw it. This is the dangerous one, and it goes to a human.
In theirs but not ours. Usually a response we never received for a payment that did complete. The state machine catches most of these through status checks, and the rest get corrected here.
Any time you write to two systems you do not control together, you need this same trio: an idempotency key so retries are safe, a state machine that never leaves work in an ambiguous state, and a reconciliation pass that assumes the two sides will disagree. Ordering systems, inventory systems and anything talking to a third party API all have a version of this problem, and most of them handle it worse than a payment rail does.
Break it#
The failure that actually happens in production is not the switch falling over. It is one bank getting slow.
Read every step as text
- normal · Healthy. Both banks answering in about 300ms. The switch holds a connection for roughly as long as a payment takes and everything moves.
- B slow · Degraded. Bank B is now taking 8 seconds instead of 300ms. Payments to Bank B are slow, which is unavoidable. Nothing else is affected yet.
- B timing out · Down. Bank B is timing out at 30 seconds and requests are piling up against it. Because the switch uses one shared connection pool, those stuck requests occupy every worker. Payments to Bank A, which is perfectly healthy, now fail too. One bank has taken down the whole rail.
- bulkheaded · Degraded. Give every bank its own bounded pool. Bank B can fill its own pool and no other. Add a circuit breaker per bank: after a run of failures, fail fast for that destination instead of waiting 30 seconds to find out what you already know. Payments to Bank B still fail, which is honest, and payments to everyone else keep working.
The general form of that fix is worth naming, because it comes up in every design with third party dependencies. A shared resource pool converts one dependency's failure into everyone's failure. Bounded per dependency pools, plus a breaker that fails fast, converts it back into a local failure.
Trade-offs#
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Saga with compensation over two phase commit | No participant holds a lock waiting on another organisation, so one slow bank cannot freeze everyone else. | There is a real window where money has left one account and not arrived, and you have to build the machinery that closes it. | Any transaction crossing an organisational boundary. 2PC is for systems under one operator. |
| Status check over retry | Cannot double debit, no matter how many times the network eats a response. | Slower recovery, and every downstream party has to implement a status endpoint that is actually correct. | Always, for anything that moves money or has any other side effect you cannot take back. |
| Append only ledger over mutable balances | Complete history, reconciliation is possible at all, and every correction is itself auditable. | Balances become a computed value, so you need snapshots to avoid summing a billion rows. | Anything financial. The audit requirement makes this less of a choice than it looks. |
| Per bank pools and breakers over one shared pool | One failing dependency stays one failing dependency. | Lower total utilisation, since each pool has to be sized for its own peak with headroom. | Any fan out to third parties with independent failure modes, which is most integrations. |
Interview replay#
Checkpoint#
1. The debit request to the remitter bank times out. What is the correct next action?
2. Why is two phase commit the wrong tool across two banks?
3. Bank B starts timing out and payments to healthy Bank A begin failing too. What is the cause?
- Tens of thousands of payments a second, each one a two phase movement between two separate companies.
- Four ledger writes per payment, so the ledger runs at four times the payment rate.
- A timeout is a third outcome, not a failure. Resolve it by asking the bank, never by guessing.
- You cannot roll back another company's database. You undo a payment with an opposite payment.
This is two transactions at two organisations that cannot share a transaction, so the design is a saga with compensation rather than two phase commit, because no bank will hold locks waiting on another bank. A central switch owns routing and the state machine: resolve the payee address, debit the remitter, credit the beneficiary, and write every state change durably before attempting the action it describes. The state that matters is DEBITED, meaning money has left and not arrived, and everything else exists to make sure nothing stays there. On a timeout I never retry the debit, I status check the same transaction id, because a retry can debit twice. If the credit genuinely fails I append a reversal rather than editing anything, since the ledger is double entry and append only. A sweeper resolves anything stuck, and a daily reconciliation against the bank's own file catches what is left. The failure I would design for hardest is one slow bank filling a shared connection pool and taking down payments to every other bank, which is a per bank pool and a circuit breaker.