Exactly-once in practice

A food delivery app runs a campaign: ₹50 cashback on your first order. A month later finance notices that about 3% of customers received ₹100. Every team swears its service is idempotent, and every team is right. The app retried a payment with no key, so some customers had two orders. The cashback consumer deduplicated perfectly, by event id, and two orders have two event ids.

Exactly-once is not a feature you switch on in one place. It is a property of the whole path, and it fails at the weakest hop.

What exactly-once actually means#

The delivery guarantees article makes the core argument: a message cannot be delivered exactly once over a network that loses acknowledgements, because a sender that hears nothing cannot tell "not done" from "done, reply lost". What you can have is an exactly-once effect: deliver at least once, and make every repeat do nothing.

exactly-once effect = at-least-once delivery + idempotent processing, at every hop

The last three words are the part that gets dropped. A system has as many hops as it has network calls, and each one can repeat or lose work in its own way.

Follow one order#

Here is the path a single order takes, with the hops numbered where something can go wrong.

1. App to API. The reply is lost, the app retries, and the API sees the order twice.
2. API to database to broker. The API saves the order and dies before telling anyone. The event is lost.
3. Relay to Kafka. The broker stored the event but its acknowledgement was lost, so the relay sends it again.
4. Kafka to consumer. The consumer credits the wallet, dies before committing its offset, and gets the event again after restart.

Try to credit it once#

Turn failures on, then add guards until the cashback comes out right. Start with the defaults: the consumer's dedupe is already on. Notice what it cannot fix.

Three rules fall out of it.

A guard can only remove repeats that share an id. The consumer's dedupe table is perfect at catching the same event twice. It is useless against two orders, because by then they have different order ids and different event ids. The id that marks "this is the same request" must be created at the very start, in the app, and carried through every hop. That is why the client generates the idempotency key, not the server.

Loss is worse than duplication. A duplicate can be caught later. A lost event cannot be recovered by anything downstream, because downstream never hears of it. The outbox turns "save, then publish" into one local transaction plus an at-least-once relay, which trades the loss for a duplicate, the problem you can solve.

Guards overlap, and that is good. The idempotent producer and the consumer's dedupe both catch a resent event. Keep the one closest to the effect even if you have the earlier one, because something you have not thought of (a manual replay, a consumer group reset) will eventually send the same event twice.

Recognising a repeat, at the consumer#

The consumer's guard is the one people get wrong, because the record of "I processed this" and the effect itself must commit together. Separately, a crash between them reopens the gap.

async function onOrderPlaced(event: OrderPlaced) {
  const db = await pool.connect()
  try {
    await db.query('BEGIN')
    // Remember the event. If it is already there, this inserts nothing.
    const { rowCount } = await db.query(
      'INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING',
      [event.id],
    )
    if (rowCount === 1) {
      await db.query('UPDATE wallets SET balance = balance + 50 WHERE user_id = $1', [event.userId])
    }
    await db.query('COMMIT')
  } catch (err) {
    await db.query('ROLLBACK')
    throw err // not committed: the event will be redelivered and tried again
  } finally {
    db.release()
  }
  // Only now tell Kafka we are done. A crash before this line means a redelivery,
  // which the processed_events row turns into a no-op.
}

Better still, when you can: make the effect itself a fact keyed by its cause, instead of an increment. INSERT INTO cashback (order_id PRIMARY KEY, amount) cannot happen twice for one order, and the wallet balance is the sum of those rows. The database's own unique constraint becomes the guard, and there is no second table to keep in step.

The event id has to be deterministic: the same order must produce the same id every time it is published. Derive it from the business fact (order-8812:placed), or store it in the outbox row once. A fresh random id generated at publish time makes every resend look new.

How long to remember#

A dedupe record only needs to live as long as a repeat can still arrive. Work it out from the longest path back:

Kafka keeps messages for 7 days by default, and a consumer group reset can replay all of them, so a Kafka consumer's dedupe table should hold at least that long.

SQS FIFO queues deduplicate for 5 minutes themselves, which covers producer retries and nothing after.

Stripe keeps idempotency keys for at least 24 hours, which tells you what it thinks the longest sensible client retry is.

Store dedupe rows partitioned by day so expiring old ones is dropping a partition, not a huge DELETE.

Kafka's exactly-once, and where it stops#

Kafka offers two features under the exactly-once name. The idempotent producer (on by default since Kafka 3.0) gives each producer a sequence number per partition, so the broker drops a resend it already has. That is hop 3. Transactions let a stream processor read from one topic and write to another, committing its output and its input offsets together, and consumers with isolation.level=read_committed only see committed output. Flink offers the same end-to-end guarantee between its sources and sinks using checkpoints.

Both guarantees are true and both stop at the edge of the system that provides them. The moment the consumer writes to Postgres, charges a card or sends an SMS, Kafka's transaction cannot include that, and you are back to the dedupe table.

Effects you cannot take back#

Some effects leave your system: an SMS, an email, a bank transfer, a call to a partner's API. You cannot put them in a transaction and you cannot delete them.

Pass a key along. Most payment providers and many messaging APIs accept an idempotency key. Derive it from the business fact, so a retry sends the same one.

Do the irreversible thing last. Commit your own state first ("cashback credited, SMS pending"), then send, then record "SMS sent". A crash leaves a pending row a sweeper can retry, rather than an SMS with no record behind it.

When you cannot know, ask. If a transfer times out, query its status by your reference before retrying. UPI apps show "payment pending" for exactly this reason: the money may have moved, and the only safe move is to check with the bank, not to send again.

Or accept it. A duplicate "your order is on its way" notification is harmless. Spend the effort on money, stock and anything a customer would screenshot.

Interview replay#

Checkpoint#