Message queues and delivery semantics
A queue between two services buys you three things: the sender stops waiting, a burst gets absorbed instead of dropped, and the receiver can fail without losing work.
It also costs you three things that people discover later: messages arrive more than once, they arrive out of order, and a failure that used to be a 500 the caller could see is now a silent backlog nobody is looking at.
Your consumer processes a message, then crashes before acknowledging it. What happens next, and what does that mean your handler must be?
Queue or log
These are different data structures that both get called a queue, and the difference decides what you can build.
Partitions decide your parallelism
In a log, ordering is guaranteed within a partition and nowhere else. That single fact drives most of the design decisions people get wrong.
Every requirement of the form “these events must be processed in order” is a requirement that they share a partition, and a partition is served by one consumer. Choose the narrowest key that satisfies the requirement. Per user is usually fine. Per account is often fine. Globally ordered means one partition and one consumer, which is a queue with extra steps.
Delivery semantics
Three phrases get thrown around, and only two of them are real.
At most once. Acknowledge before processing. If you crash in between, the message is gone. Almost never what you want, but correct for something like a metrics sample where a duplicate is worse than a gap.
At least once. Acknowledge after processing. If you crash in between, the broker redelivers and you process it twice. This is what essentially every system gives you, and it is the one to design for.
Exactly once is not a delivery guarantee, because the network cannot provide one. What exists is at least once delivery plus idempotent processing, which produces exactly once effects. Some systems automate this within their own boundary with transactional offsets, but the moment your handler calls an external API, you are back to making the handler idempotent yourself. Saying this plainly in an interview is a strong signal.
The failures worth naming
The poison message. One malformed message throws every time. At least once delivery means it is redelivered forever, and it blocks its partition. Count attempts, and after a few, move it to a dead letter queue and carry on. A dead letter queue nobody alerts on is a folder where data goes to be forgotten, so alert on it being non empty.
Backpressure. Producers are faster than consumers, the backlog grows, and nothing errors. The system looks healthy right up until retention expires and messages are dropped silently. Consumer lag is the metric that matters here, not queue depth and not error rate.
The retry storm. A downstream service is struggling, every message fails, every failure retries immediately, and the retries triple the load on the thing that was already struggling. Exponential backoff with jitter, and a circuit breaker so you stop calling a dependency you already know is down.
Rebalance thrash. A consumer takes too long between polls, the group decides it is dead, partitions move, the work is redone, and the extra load makes the next consumer slow too. Keep processing off the poll thread, or raise the timeout deliberately rather than by accident.
Where a queue belongs
Figure 1 hides a real problem. Steps 1 and 2 are two separate systems, so a crash between them either commits a row nobody hears about or publishes an event for a row that does not exist. The usual fix is the outbox pattern: write the event into a table in the same transaction as the row, and have a separate process ship rows from that table to the log. One commit, no gap.
Break it
Trade-offs
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Event log over work queue | Many independent consumers on one stream, and replay when a consumer had a bug. | Ordering and parallelism are tied to partition count, which you choose up front. | Several teams need the same events, or you want to reprocess history. |
| At least once with idempotent handlers | No message is ever lost, and duplicates are harmless. | Every handler needs a natural key and a uniqueness check, which is real work per handler. | The default. Assume duplicates, because you will get them. |
| Partition by user id | Ordering per user, which is the only ordering most products actually need, with full parallelism across users. | A single very active user becomes a hot partition that more consumers cannot fix. | Most event streams. Narrow the key until ordering still holds and no further. |
| Outbox pattern | Removes the gap where a row is committed but its event never published, or the reverse. | A table, a relay process, and events arriving a moment later than the commit. | Whenever a database write and a published event must both happen or neither. |
Interview replay
Checkpoint
1. You have 6 partitions and 8 consumers in one group. What happens?
2. Why is "exactly once delivery" the wrong thing to promise?
3. Consumers are falling behind and lag has been growing for two hours. No errors are being reported. What is the risk?
I would use an event log rather than a work queue when more than one consumer needs the same events or I might want to replay them, and a plain work queue when it really is just background jobs. Ordering only holds within a partition, so I would partition by the narrowest key that still gives the ordering we need, usually per user, and pick a partition count above my expected consumer count because partitions are a hard ceiling on parallelism. I would assume at least once delivery and make every handler idempotent with a key derived from the message, since exactly once delivery is not a thing you can have over a network, only exactly once effects. To publish an event and commit a row together I would use an outbox table so there is no gap between them. And the metric I would alert on is consumer lag, well below the retention window, because falling behind produces no errors at all until messages start being deleted unread.
