15 min read

Message queues and delivery semantics

A queue between two services buys you three things: the sender stops waiting, a burst gets absorbed instead of dropped, and the receiver can fail without losing work.

It also costs you three things people discover later. Messages arrive more than once. They arrive out of order. And a failure that used to be a 500 the caller could see is now a silent backlog nobody is watching.

Your answer

Your consumer processes a message, then crashes before acknowledging it. What happens next, and what does that mean your handler must be?

Queue or log#

These are different data structures that both get called a queue, and the difference decides what you can build.

Shape

One message goes to exactly one worker, and once acknowledged it is gone. Add workers and throughput rises, with no coordination. There is no history, so a second consumer that wants the same messages cannot have them, and a bug that ate a message leaves nothing to replay. SQS and RabbitMQ work this way.

Amazon SQSQueue: one worker gets each message
RabbitMQQueue: one worker gets each message
Apache KafkaLog: kept, many groups read it all
Amazon KinesisLog: kept, many groups read it all

Partitions decide your parallelism#

In a log, ordering is guaranteed within a partition and nowhere else. That single fact drives most of the design decisions people get wrong.

Consumer group

part 0

lag 2consumer 0

part 1

lag 1consumer 1

part 2

lag 3consumer 2

part 3

lag 2consumer 0

part 4

lag 2consumer 1

part 5

lag 1consumer 2

Six partitions, three consumers, two partitions each. Lag is low and even. This is the state you design for, and adding a fourth consumer would help because there are still spare partitions.

Ordering costs you parallelism, always

Every requirement of the form "these events must be processed in order" is a requirement that they share a partition, and a partition is served by one consumer. Choose the narrowest key that satisfies the requirement. Per user is usually fine. Per account is often fine. Globally ordered means one partition and one consumer, which is a queue with extra steps.

Delivery semantics#

Three phrases get thrown around, and only two of them are real.

At most once. Acknowledge before processing. If you crash in between, the message is gone. Almost never what you want, but correct for something like a metrics sample where a duplicate is worse than a gap.

At least once. Acknowledge after processing. If you crash in between, the broker redelivers and you process it twice. This is what essentially every system gives you, and it is the one to design for.

Exactly once is not a delivery guarantee, because the network cannot provide one. What exists is at least once delivery plus idempotent processing, which produces exactly once effects. Some systems automate this within their own boundary with transactional offsets, but the moment your handler calls an external API, you are back to making the handler idempotent yourself. Saying this plainly in an interview is a strong signal.

1/6 The broker hands over message 41. It stays on the broker, unacknowledged, until the consumer confirms it is done.

The failures worth naming#

The poison message. One malformed message throws every time. At least once delivery means it is redelivered forever, and it blocks its partition. Count attempts, and after a few, move it to a dead letter queue and carry on. A dead letter queue nobody alerts on is a folder where data goes to be forgotten, so alert on it being non empty.

Backpressure. Producers are faster than consumers, the backlog grows, and nothing errors. The system looks healthy right up until retention expires and messages are dropped silently. Consumer lag is the metric that matters here, not queue depth and not error rate.

The retry storm. A downstream service is struggling, every message fails, every failure retries immediately, and the retries triple the load on the thing that was already struggling. Exponential backoff with jitter, and a circuit breaker so you stop calling a dependency you already know is down.

Rebalance thrash. A consumer takes too long between polls, the group decides it is dead, partitions move, the work is redone, and the extra load makes the next consumer slow too. Keep processing off the poll thread, or raise the timeout deliberately rather than by accident.

Where a queue belongs#

Figure 1. The write the user waits for is small and synchronous. Everything that can happen a second later happens behind the log.

Figure 1 hides a real problem. Steps 1 and 2 are two separate systems, so a crash between them either commits a row nobody hears about or publishes an event for a row that does not exist. The usual fix is the outbox pattern: write the event into a table in the same transaction as the row, and have a separate process ship rows from that table to the log. One commit, no gap.

Break it#

Consumer health

steady

steadydownstream slowretry stormbacklog past retentionprotected

Healthy. Consumers keep up, lag is a second or two, and the log is doing what it is for: absorbing jitter so nobody has to wait for anybody.

Read every step as text

steady · Healthy. Consumers keep up, lag is a second or two, and the log is doing what it is for: absorbing jitter so nobody has to wait for anybody.
downstream slow · Degraded. The downstream API goes from 50ms to 2 seconds. Consumers are now bound by it, throughput collapses, and lag starts climbing. Nothing has errored, so no alert on error rate will fire. Consumer lag is the only signal that anything is wrong.
retry storm · Down. The API starts failing. Every message retries immediately, so the failing service now receives several times its normal load, which guarantees it stays down. The consumers are busy doing work that cannot succeed.
backlog past retention · Down. Hours later the oldest messages pass the retention window and are deleted unread. This is silent, permanent data loss with no error anywhere, and it is the reason lag needs an alert with a threshold well below retention.
protected · Degraded. A circuit breaker stops calling a dependency already known to be down, backoff with jitter spreads the retries, and a dead letter queue takes messages that fail repeatedly so they stop blocking their partition. The backlog still grows while the API is down, but it drains cleanly afterwards instead of being lost.

Trade-offs#

Choice	What you gain	What you pay	Pick it when
Event log over work queue	Many independent consumers on one stream, and replay when a consumer had a bug.	Ordering and parallelism are tied to partition count, which you choose up front.	Several teams need the same events, or you want to reprocess history.
At least once with idempotent handlers	No message is ever lost, and duplicates are harmless.	Every handler needs a natural key and a uniqueness check, which is real work per handler.	The default. Assume duplicates, because you will get them.
Partition by user id	Ordering per user, which is the only ordering most products actually need, with full parallelism across users.	A single very active user becomes a hot partition that more consumers cannot fix.	Most event streams. Narrow the key until ordering still holds and no further.
Outbox pattern	Removes the gap where a row is committed but its event never published, or the reverse.	A table, a relay process, and events arriving a moment later than the commit.	Whenever a database write and a published event must both happen or neither.

Interview replay#

Interviewer

Your consumer crashes after doing the work but before acknowledging. What happens?

The question that decides whether you understand delivery semantics or have memorised the phrases.

You

The broker redelivers, because from its side an unacknowledged message is unprocessed. So the handler runs twice, which means it has to be idempotent. I would derive a key from the message, usually an event id, and make the write conditional on it, so the second attempt collides and returns success without repeating the side effect.

Goes straight from the mechanism to the requirement it places on the handler.

Interviewer

Could you use exactly once delivery instead?

A trap. The phrase is on every broker’s marketing page.

You

Not really. Exactly once delivery is not achievable over a network, because the acknowledgement can always be the thing that gets lost. What you can have is at least once delivery with idempotent processing, which gives exactly once effects. Some brokers automate that within their own boundary using transactional offsets, but as soon as my handler calls an external service, the guarantee ends and I am back to doing it myself.

Corrects the premise without being smug and names exactly where the boundary is.

Interviewer

How many partitions?

Checking whether you know what the number controls.

You

More than my expected consumer count, because a partition only ever goes to one consumer in a group, so partitions are a hard ceiling on parallelism. I would overprovision, since raising it later changes which key lands where and breaks per key ordering across the change. And I would pick the partition key as the narrowest thing that still gives the ordering we need, usually per user rather than global.

Three sentences, three separate consequences, all of them practical.

Interviewer

How would you know the queue is unhealthy?

Open. A chance to name the failure that produces no errors.

You

Consumer lag, alerted well below the retention window. That is the one that matters, because the dangerous failure here is silent: consumers falling behind produce no errors at all, and if the backlog passes retention the messages are deleted unread. I would also alert on the dead letter queue being non empty, since a DLQ nobody looks at is just a place data goes to disappear.

Picks the metric that catches the silent failure and explains why error rate would not.

Checkpoint#

Checkpoint

1. You have 6 partitions and 8 consumers in one group. What happens?

2. Why is "exactly once delivery" the wrong thing to promise?

3. Consumers are falling behind and lag has been growing for two hours. No errors are being reported. What is the risk?

Worth memorising

A queue deletes on acknowledgement, a log keeps the record and each consumer holds its own offset.
Parallelism is capped by partition count, so it is a capacity decision you make early and change painfully.
At least once is the default. Exactly once comes from an idempotent consumer, never from a better broker.
Ordering is per partition only. Global ordering means one partition, which means no parallelism at all.

Say this in 60 seconds

I would use an event log rather than a work queue when more than one consumer needs the same events or I might want to replay them, and a plain work queue when it really is just background jobs. Ordering only holds within a partition, so I would partition by the narrowest key that still gives the ordering we need, usually per user, and pick a partition count above my expected consumer count because partitions are a hard ceiling on parallelism. I would assume at least once delivery and make every handler idempotent with a key derived from the message, since exactly once delivery is not a thing you can have over a network, only exactly once effects. To publish an event and commit a row together I would use an outbox table so there is no gap between them. And the metric I would alert on is consumer lag, well below the retention window, because falling behind produces no errors at all until messages start being deleted unread.