Design a notification service

A marketing team schedules a campaign to 40 million users for 9am. At 9:00:02 the queue is full of campaign messages. At 9:00:03 someone's one time password is behind 40 million of them, and by the time it arrives the login screen has already timed out.

A notification is any message your app sends to a person without being asked at that moment: a push alert on the lock screen, an SMS with a login code, an email receipt. A notification service is the one place every team in the company sends them through. Watch one login code go through it.

A few words come up all through this page:

• A provider is the outside company that does the last step of delivery: Apple Push Notification service (APNs) for iPhones, Firebase Cloud Messaging (FCM) for Android, an SMS gateway for texts, an email provider for mail. You cannot see inside them.
• A campaign is a marketing send to a whole group of users at once, often millions.
• An idempotency key is a unique label the caller attaches to a request, so that if the same request arrives twice, the second one is recognised and ignored. (Idempotency keys shows one working.)

Nobody wrote a bug in the opening story. Every service worked. The design never said which message mattered more, and that one gap is what this question is about.

Step 1: Understand the problem#

"Send a notification" sounds like one API call. It is really three problems with one name: a fan out problem (one request becomes many messages), a scheduling problem (some messages should wait), and a problem of depending on outside companies that fail often.

What you are building, and what you cut#

Back of the envelope#

Three inputs. Watch the gap between the average and the peak, because the whole design is about that gap. (New to these estimates? Back of the envelope walks through the method.)

Step 2: Propose the high level design#

The API#

One endpoint does almost all the work. The interesting decisions are in the request body. Note the 202 Accepted replies: an HTTP status that means "I have your request, I have not finished it". The caller does not wait for the send. The work waits in a queue instead, which is what message queues are for.

{
  "userId": 4471,
  "template": "otp_login",
  "params": { "code": "402913" },
  "priority": "urgent",
  "idempotencyKey": "otp:4471:1756112400"
}

{
  "template": "weekly_digest",
  "segmentId": "seg_9931",
  "priority": "bulk",
  "notBefore": "2026-09-11T09:00:00+05:30"
}

{
  "marketing": { "push": false, "email": true },
  "quietHours": { "start": "22:00", "end": "08:00", "tz": "Asia/Kolkata" },
  "dailyCap": 5
}

The data model#

Three tables, and the interesting one is the smallest. A wide column store (think Cassandra) suits this: huge write volume, and every read is "everything for one user".

The whole system on one whiteboard#

Figure 1 is the picture at the top with every box added, including the ones the login code skipped past.

Walking Figure 1:

1. Any internal service posts a notification and gets a 202 at once. Nothing downstream is allowed to make a checkout page slow.
2. The API renders the template for each channel: a message with blanks, like Your code is {code}, filled in with the values. Fix a typo once and it is fixed everywhere, in every language.
3. Preferences are checked before the message is queued. A message we decide not to send is recorded as suppressed, not silently dropped, because "why didn't my user get it?" is the most common support question you will get.
4. What survives goes into the lane for its priority. A lane is its own queue with its own workers, three of each, so nothing in the bulk lane can delay the urgent one. If queues are new, message queues covers them from the start.
5. Dedupe (drop a repeat) and the daily cap run just before sending, not when the message is accepted, because the cap is about what a person actually received.
6. Channel workers pull messages at the speed the provider will take, not the speed the queue offers. Workers pulling, rather than the queue pushing, is what lets them set the pace (push vs pull).
7. Workers call the providers, record whatever delivery callbacks come back, and move permanent failures to a dead letter queue: a side queue where messages that keep failing are parked, with the error, for a person to look at.

Step 3: Design deep dive#

Priority is a lane, not a number#

Picture a single post office counter with 40 million parcels in line. A customer arrives with one urgent letter. Labelling the letter "urgent" does not help if the clerk still has to dig through the line to find it. What helps is a second counter that only takes urgent letters. That is the whole idea.

Figure 2 plays that out: a login code and a campaign arriving at the same moment.

Delivering at least once without annoying anybody#

Every provider can take your request, fail to answer, and deliver anyway. So you must retry, and a retry can create a duplicate. That rules out exactly once delivery. What you can build is at least once delivery (every message arrives, sometimes twice) plus a way to spot and drop the repeats. At most once vs at least once vs exactly once explains why exactly once cannot exist across a network.

For a login code the key writes itself: otp:{userId}:{minute}, so a retried login cannot produce a second message. Keep the key in the database, not in a cache, because a cache can evict it, and an eviction should never cost a duplicate. For a payment confirmation, build the key from the payment id, so the same payment can never notify twice.

The retries themselves need care too. Hammering a provider that is already struggling is how a slow provider becomes a dead one.

Retries, backoff and circuit breakers has a live retry storm if you want to watch the second rung fail.

The cap that stops people uninstalling the app#

The most expensive failure here is not a dropped message. It is sending so many that someone turns notifications off, because that person is now unreachable for ever, including for the messages that matter. Think of a shop that phones you three times a day: you block the number, and then you miss the call about your order.

Say out loud that this belongs in the platform. A limit that each team implements for itself is not a limit. (The same thinking, applied to requests rather than messages, is in rate limiting vs throttling vs backpressure.)

Break it#

Raise the pressure and watch which box goes red first.

The lesson from the last two states: in a system whose whole job is calling other people's services, the circuit breaker is not a nice extra. It is the difference between one channel slowing down and everything stopping.

Trade-offs#

The level bar#

One moment from this round, answered three ways. Pick the one you think is staff level before the labels appear. (What changes at senior and staff explains the gap.)

Then run the question cold, with a 45 minute timer and no notes, and score the run on the five things this question is really testing.

Interview replay#

Checkpoint#