An online shop ships an order. One line of code calls a push API: “Your order is on its way.” In testing it works every time.
Then the real world arrives. The push provider takes two seconds to answer, and the order service waits with it. The provider is down for ten minutes, and orders fail. A retry fires twice, and a customer’s phone buzzes twice. Imagine a marketing blast to millions of people, with a login code stuck behind it for half an hour.
None of that is a bug in the line of code. It is the missing system around it: decide once, store first, then nudge the device with an id.
- The interview framework and estimation primer.
- Message delivery guarantees, design a rate limiter and design a chat system, because this problem reuses all three.
Why does a push only wake a device instead of carrying the message? Because the push can miss, so the durable record is the truth. A notification is a nudge, not the delivery.
- A notification service is a pipeline: decide, store, queue, deliver, retry and expire, not a single API call.
- Never call the push provider inside the user’s request: store the notification and deliver it from a queue.
- Retries are unavoidable, so every notification needs an id and a duplicate check.
- Give notifications an expiry, because a late “your driver arrived” is worse than none.
- Keep urgent notifications in a separate lane from campaigns, so bulk sends cannot delay a login code.
Design a notification service in 45 minutes: rules and duplicate checks in one place, store before delivering, retry safely with expiry, remove dead tokens, and keep urgent messages ahead of campaigns.
Step 1: Pin Down the Problem#
Functional requirements:
- Services publish events (“order shipped”, “new message”, “login code”).
- The service decides who is notified, on which channel (push, email, SMS, in-app), and when.
- Users can set preferences, mute things, and see a notification history.
- A campaign can be sent to millions of users.
Out of scope (say these out loud):
- Writing the message copy (an event carries a template id, a language and data, and the worker renders it), email deliverability internals, SMS carrier setup, and open-rate analytics.
- The chat system’s own delivery path. Here the question is how a device is woken.
Ask first, then state your assumptions aloud: is this one company’s events or a platform for other teams? Which messages are transactional (a login code) and which are marketing? How fast must a code arrive? Which regions? I will assume at-least-once delivery plus a duplicate check, because no push provider offers more.
Non-functional requirements, each with a number. These are assumptions you announce:
- Fast for urgent messages: a login code should arrive in seconds.
- Reliable: an accepted notification is delivered, or expires on purpose, without duplicates in the common case.
- Scale: assume 10 million daily users. Step 4 turns that into numbers.
- Lawful: opt-outs and quiet hours are enforced in code, not by convention (see the table under step 5, first dive).
Step 2: Entities and API#
Event id, type, user, data
Preference user, type, channels
Notification id, user, type,
status, expires_at
Device user, token, os
POST /events
Idempotency-Key: <event id>
-> 202 accepted
-> 401/403 caller not allowed
-> 409 same key, other body
-> 429 caller over quota
GET /notifications?cursor=
-> items + next_cursor
PUT /preferences
POST /devices
-> register a tokenThe service takes events in and makes notifications out. The id of a notification is built from the event, so the same event always makes the same id. That one choice powers the duplicate check later, and it also makes POST /events safe for a caller to retry. History uses a cursor, not an offset, because new items keep arriving at the top of a newest-first list and an offset would repeat or skip rows.
A user has several devices, so Device is a list, and POST /devices runs at each app start because tokens change. status moves from pending to sent to delivered, or ends failed or expired.
Step 3: A Design That Works#
Start with the hook’s line of code: the order service calls the push provider inside the request.
It works in a demo, and many candidates stop here.
Assume 11,600 events a second at peak and a provider that answers in 2 seconds. How many requests are waiting on the provider at the same moment?
Call it 23,200 requests in flight (11,600 × 2 seconds), each holding a thread, with no retries, record or rules. The fix is a queue in front of the provider, which brings the rest of the design with it.
Step 4: Estimate What Matters#
Assume 10M daily users
20 notifications each a day
Events: 200M / 86,400
≈ 2,300/s
Peak (x5): ≈ 11,600/s
Inbox: 200M x 300 B
= 60 GB a day
≈ 1.8 TB for 30 daysThen the campaign:
Campaign: 10M users
Lane speed: 5,000/s (assumed)
10,000,000 / 5,000
= 2,000 s ≈ 33 minutesEach line decides something:
- 11,600 events a second at peak is modest for a queue. The hard part is correctness and isolation.
- 1.8 TB for 30 days makes the inbox a real data store with a retention rule. Partition it by user, because every read is one user’s history.
- A campaign takes about 33 minutes, and anything urgent in its lane waits that long.
- The 5,000 a second is an assumption: half of FCM’s default quota of 600,000 messages a minute per project (10,000 a second, as of October 2026), leaving room for urgent traffic. Quotas change.
Step 5: Deep Dives#
Four questions decide this design, and interviewers usually pick two or three: delivery and retries, lanes, and duplicates. Start with the first and let them steer. The production half (failures, regions, caller auth, compliance, follow-ups) has its own lesson: Failure Modes, Security and Follow-Ups.
Who Should Be Notified, and When?#
Caps per user are a rate limiter: a token bucket per user and type decides when one more notification is too many.
Every event produces a notification for every affected user, on every channel.
Check rules in one fixed order and stop at the first that decides. Matrix does this with push rules, which are checked by kind in a fixed order: override, content, room, sender, then underride. Mastodon’s notify service first drops a notification if the sender is blocked or muted or the conversation is muted.
Put the rules, the per-user caps, the quiet hours and the duplicate check in one decide step that every channel trusts. Caps use a token bucket per user and type. The duplicate check uses the notification id: Matrix says duplicate suppression using the event id is needed so that retries do not cause duplicate alerts, while updating an unread count is idempotent and needs none.
In Mastodon’s code, a notification that fails a policy is saved and marked filtered, not dropped: no push, but history stays honest. The order of the six steps is design judgement, apart from the sourced parts.
Quiet Hours and Scheduled Sends#
A notification that should wait is saved with a send-at time in the user’s time zone, and a scheduler queues it when due, so nothing sleeps inside a worker.
Opt-Out and Consent#
Keep transactional and marketing as separate types with separate opt-outs, so turning off promotions never blocks a login code, and log every opt-in and opt-out. The legal rules that shape this (send hours, unsubscribe deadlines) are in the follow-ups lesson.
Why Store Before You Deliver?#
A queue gives at-least-once delivery. Whatever consumes it must cope with seeing the same job twice, and must be able to find the facts again.
Put the whole message on a queue and let a worker send it.
Write the notification to the inbox first, then queue a job that carries only its id. Mastodon’s service saves the notification and only then pushes.
When the job runs, the worker loads the record and asks whether it should still go. Mastodon’s push worker returns early if the notification is older than its 48-hour TTL and checks that the activity it points to still exists, because it could have been deleted in the meantime.
As in the chat lesson, the inbox is the truth and the push says “go and look”.
How Do You Deliver Through an Unreliable Provider?#
Delivery is a consumer of events, and the provider is an external service that fails on its own schedule.
Your service does not reach the phone. Matrix’s push gateway design shows the path: the server calls a push gateway, the gateway submits to a push provider, and the provider reaches the device. The spec’s example names APNS and GCM; GCM is the old name for what Google replaced with Firebase Cloud Messaging (FCM).
What Push Providers Promise#
- Apple calls APNs a best-effort service that may reorder notifications. For an offline device it stores only one per app, usually the latest, for up to 30 days depending on
apns-expiration. - FCM keeps an undelivered message for four weeks by default unless you set a
ttl, so “your driver arrived” can land weeks late. - Payloads are small: 4 KB on APNs (5 KB for VoIP) and 4,096 bytes on FCM. So send an id and let the app fetch the rest.
- A collapse id (
apns-collapse-id, at most 64 bytes) merges repeats into one visible notification, so “3 new messages” replaces two older alerts instead of stacking.
The provider is a best-effort pipe, so the inbox, not the push, is the record of what the user should see.
Three Ways to Deliver#
Call the provider once, as in step 3.
Workers send from the queue and retry with growing waits. Matrix says that on an HTTP error the server should retry for a reasonable time with exponential backoff; Mastodon’s worker uses a dedicated queue with five retries. Five tries that double from 30 seconds wait 30 + 60 + 120 + 240 + 480 = 930 seconds, about 15 minutes.
When the tries run out, mark the record failed and park the job in a dead-letter queue.
Retry-After header, cap the tries, and stop when the notification expires. FCM asks for exactly this on a 503 UNAVAILABLE, and for a wait of at least a minute before retrying a quota error.Not every failure deserves a retry. Mastodon’s web push worker removes a subscription on a 4xx other than 408 and 429, and raises on other failures so the job is retried.
Mobile providers need a finer rule:
- Delete on APNs 410, token no longer active, and FCM 404 UNREGISTERED.
- Alert, do not delete, on FCM 400 INVALID_ARGUMENT (a bad token only when the payload is valid) and APNs 403 (your own credentials failing).
- Prune by age. FCM considers a registration stale after a month without connecting by default and expires an inactive Android one after 270 days, so save a last-seen time whenever the app uploads its token.
Set an expiry per type: the Web Push standard requires a TTL and forbids delivery after it. When a channel reports back later, as Twilio’s status callbacks do, store the result on the same record.
The iOS Nudge-With-an-Id Pattern#
iOS limits background work, so ntfy’s documentation says instant notifications on iOS need a central server. A self-hosted server forwards a “poll request” holding only the message id through Firebase and APNS, and the phone fetches the message from the original server. Without that forward, delivery can take hours, or 20 to 30 minutes if the phone is in use.
How Do You Keep Campaigns From Blocking Urgent Messages?#
A queue is first in, first out. Without priorities, a pile of promotions in front of a login code means the code waits for all of them.
All notifications share a queue and are sent in order.
Give urgent types, such as login codes and security alerts, their own queue and workers, and keep campaigns in a bulk lane. ntfy has five priorities, from min to max, and Matrix lets a notification be marked high or low, so a gateway can send less urgent ones in a way that preserves battery power.
Send the bulk lane at a steady rate below the provider’s limit, spread across time zones, with a short TTL so a message that missed its moment is dropped. ntfy lets a publisher delay delivery by 10 seconds up to 3 days.
Decide once, store first, nudge with an id. The inbox record makes every retry safe; the device fetches the rest.
The Final Design in One Picture#
One notification, start to finish:
- The order service publishes “order shipped” and returns at once.
- The decide step checks blocks, preferences, quiet hours, the cap and the id, then drops, filters or passes it.
- The notification is saved in the inbox, then a job carrying only its id joins the urgent or bulk lane.
- A worker loads the record, checks it has not expired, and sends it to the provider. Success marks it sent; transient errors retry until expiry; a dead token is removed.
- The device gets a small push with an id and fetches the content, or reads the inbox on opening the app.
Production Depth Lives in the Follow-Ups Lesson#
The working design is complete for the round. These three topics are in Failure Modes, Security and Follow-Ups.
| Topic | The question it answers |
|---|---|
| Failure table and monitoring | If this part dies, what happens, and what do you watch? |
| Two regions and caller auth | Where does it run, and who may publish? |
| Compliance and follow-ups | Which rules limit sends, and what comes after the main design? |
What Each Level Should Show#
| Level | Expected in the room |
|---|---|
| Mid-level | A queue and workers, channels, a provider. May need a hint about retries and duplicates. |
| Senior | An inbox before delivery, an id-based duplicate check, retries with backoff and expiry, dead-token cleanup, a separate lane for urgent messages. |
| Staff+ | A central decide step with caps and quiet hours, filtered versus dropped, sweeps for stuck records, campaign throttling, provider limits and which errors delete a token, a home region per user with declared failover, caller auth and quotas, opt-out and quiet-hour rules, a failure table, and what to build first. |
Red Flags and Green Flags#
“On each event, call FCM and APNs.” No queue, no duplicate handling, nothing about a provider being slow or a campaign sharing the path.
“I’ll decide once in a central step, store the notification before sending, and deliver from per-lane queues with retries, an expiry and dead-token cleanup. Urgent messages get their own lane, so a campaign cannot delay a login code.”
Describing the happy path and stopping. The interviewer is waiting for “what if it is down, retries twice, or there are millions of them?”.
Try It: A Flash Sale Alert#
Change one variable. A flash sale sends an alert to 2 million users. Assume the bulk lane sends 4,000 a second and a login code arrives while it runs.
- How long does the alert take to finish?
- What happens to the login code if they share a lane?
- How do you change the design?
Hint
Divide users by the send rate. Then ask what a first-in, first-out queue does to something that arrives late.
One reasonable answer
2,000,000 / 4,000
= 500 s (8.3 min)The alert takes about eight minutes, and a code arriving mid-way waits behind the rest if they share a lane.
Give codes their own lane and short expiry. Keep the sale in the throttled bulk lane with a TTL shorter than the sale.
Check Yourself#
1.Why save the notification before queueing the delivery?
The record is the truth. A lost job can be found and retried from it; a self-contained job disappears with its message.
2.A retry fires after the provider already accepted the first try. How do you avoid a double buzz?
Derive an id from the event, check it before sending, and send it so the device can drop a repeat. Matrix calls for this duplicate suppression because retries are expected.
3.APNs answers 410 for one token, 429 for another, and 403 for a third. What do you do with each?
The 410 means the token is no longer active, so delete it. The 429 means too many requests to that token, so wait and retry. The 403 is a problem with your certificate or signing token, so keep every device token and alert a human. All three are on Apple’s response list.
In One Picture#
Requirements
events in, notifications out
Decide
rules, caps, duplicate id
Store
inbox before any push
Deliver
queue, retry, expire, clean
Lanes
urgent apart from campaigns
Rule
decide once, store first,
nudge with an idConclusion#
A notification service is a pipeline wrapped around an unreliable edge. A durable record with a small nudge on top also runs through chat and feeds.
Which would you tackle first: duplicates, expiry or lanes? Tell us in the comments.
- The delivery path to a device: design a chat system and message delivery guarantees.
- Caps per user: design a rate limiter.
- Next in Module 4: design a file storage service, where large uploads must survive a dropped connection.
