Design a notification service

A marketing team schedules a campaign to 40 million users for 9am. At 9:00:02 the queue is full of campaign messages. At 9:00:03 someone's one time password is behind 40 million of them, and by the time it arrives the login screen has already timed out.

Nobody wrote a bug. Every service worked. The design just never said which message mattered more, and that single omission is what this question is about.

Step 1: Understand the problem#

"Send a notification" sounds like one API call. It is really a fan out problem, a scheduling problem and a third party integration problem wearing one name.

What you are building, and what you cut#

Back of the envelope#

Step 2: Propose the high level design#

The API#

One endpoint does almost all the work. The interesting decisions are all in the request body.

{
  "userId": 4471,
  "template": "otp_login",
  "params": { "code": "402913" },
  "priority": "urgent",
  "idempotencyKey": "otp:4471:1756112400"
}

{
  "template": "weekly_digest",
  "segmentId": "seg_9931",
  "priority": "bulk",
  "notBefore": "2026-09-11T09:00:00+05:30"
}

{
  "marketing": { "push": false, "email": true },
  "quietHours": { "start": "22:00", "end": "08:00", "tz": "Asia/Kolkata" },
  "dailyCap": 5
}

The data model#

Three tables, and the interesting one is the smallest.

The whole system on one whiteboard#

Walking Figure 1:

1. Any internal service posts a notification. It gets a 202 immediately, because nothing downstream is allowed to make a checkout page slow.
2. The API renders the template for each channel. Callers never send rendered text, which is what lets you fix a typo in one place for every language.
3. Preferences are checked before the message is enqueued. A suppressed notification is recorded as suppressed, not silently dropped, because "why didn't my user get it" is the most common support question you will get.
4. What survives is enqueued onto the lane matching its priority. Three lanes, three sets of workers, and nothing in the bulk lane can delay the urgent one.
5. Dedupe and rate capping happen just before sending, not at accept time, because the cap is about what a person actually received.
6. Workers pull at a rate the provider will tolerate, not at the rate the queue offers.
7. They call the providers, record whatever callbacks come back, and hand permanent failures to a dead letter queue with the provider error attached.

Step 3: Design deep dive#

Priority is a lane, not a number#

The instinct is to put a priority field on the message and sort the queue by it. This does not work, for a reason worth being able to say precisely. A queue you have to sort is a queue you have to read, and a 40 million message backlog is not something you read inside a few milliseconds.

So separate the storage, not just the label. Three topics, three consumer groups, three independent capacity pools. Urgent is provisioned for its peak, which is small, and it is never behind anything. Bulk is provisioned for cost and is expected to lag.

Delivering at least once without annoying anybody#

Every provider on the list can accept a request, fail to respond, and deliver anyway. That means retries are mandatory and duplicates are possible, so exactly once is not available. What you can build is at least once delivery with a deduplication window, which produces the same user experience for a fraction of the effort.

Three mechanisms, and you should name all three:

An idempotency key from the caller. For an OTP that key is naturally otp:{userId}:{minute}, so a retried login cannot produce a second message. Make it unique per user for 24 hours and enforce it in the database, not in a cache, because a cache eviction should never cost a duplicate.

A dedupe check in the worker, just before sending. This catches the notification that was enqueued twice by two different code paths, which happens more often than anyone expects once several teams are calling you.

The provider's own idempotency token, where one exists. APNs has a collapse id, most SMS gateways have nothing, and email has nothing at all. So this third mechanism is partial, and you should say so rather than implying it covers everything.

The cap that stops people uninstalling the app#

The most expensive failure in a notification system is not a dropped message. It is sending enough messages that somebody turns notifications off, because that user is now unreachable forever, including for the ones that matter.

So the service enforces limits the sending teams do not control:

A daily cap per user per category, counted in the same partition as the user's notifications so the check is one read. A quiet hours window in the user's own timezone, where a bulk notification is held until morning rather than dropped, and an urgent one goes anyway. And a collapse rule, where five notifications of the same category inside an hour become one that says five, which is the only mechanism here that reduces volume rather than moving it.

Say out loud that this belongs in the platform. Every team believes their notification is the important one, and a limit that each team implements for itself is not a limit.

Break it#

The lesson worth taking from the last two states: in a system whose whole job is calling other people's services, the circuit breaker is not a refinement. It is the difference between one channel degrading and everything stopping.

Trade-offs#

Interview replay#

Checkpoint#