The interviewer nods at your diagram and says: “Good. Now the gateway in the middle dies at peak. And someone sends a 200 MB video. And a regulator asks where the messages live.”
That is the second half of a chat round. The core design is the part you draw. These follow-ups show whether you have run something like it, and every answer can lean on one fact from the main lesson: the durable log is the truth, and everything faster is a bonus.
- The main lesson, Design a Chat System: Store First, Then Push. This one assumes its gateways, route table, per-conversation sequence numbers and device cursors.
- Message delivery guarantees, for why retries are safe.
Why store a message before pushing it? Because the push can miss, and a message already in the durable log is picked up from the device’s cursor later. Every fallback below is a version of that sentence.
- Every component except the log can die without losing a message, because devices resume from a cursor.
- Big files skip the chat path: upload to blob storage with a presigned URL and send a reference.
- A rolling restart is a small reconnect storm, so drain gateways a few at a time.
- Encryption, regions and retention are product trade-offs. Name the cost of each choice.
- Say “judgement” out loud when an answer is design opinion rather than a vendor’s documented behaviour.
Answer the follow-ups that come after a chat design: walk each failure to its fallback, design media uploads in three tiers, and give a short trade-off answer on encryption, abuse, monitoring, deploys, cost and retention.
What Breaks, and What Saves You#
Start with the failures, because the main design was built around them. “Judgement” below means design opinion, not a vendor’s documented behaviour.
| If this dies | What happens | Why it is safe |
|---|---|---|
| A gateway | Its devices reconnect, with jitter | Cursors resume from the log |
| A conversation’s owner | Sends to it pause until a standby takes over | The log holds the last seq; the standby continues from it |
| A log node | Reads and writes move to its copies | Ack only after the durable write (assume a second zone) |
| The route table | Pushes miss | Devices still find messages by gap detection and /sync |
| Pub/Sub | Fast path stops | It was never the source of truth |
| The push provider | Offline devices are not woken | They sync on next open |
Which row of the table is the only one where users notice, even though nothing is lost? Think about what the user sees, then check the first two rows.
Which Failures Do Users Feel?#
Two rows cost something visible. A dead gateway makes its devices wait through a reconnect, and a dead conversation owner pauses sends for that one conversation until the standby takes over. The other rows are invisible, because the log still holds every message. Say that distinction aloud: lost nothing, paused briefly is the honest claim, not “no impact”.
A failover is the standby taking over. The second zone in the log row is an assumption: one copy of the log in another zone is what lets an ack mean “survives a zone loss”.
How Do Photos and Videos Work?#
Text messages are assumed at 200 bytes; a video can be megabytes. Keep big bytes off the chat path.
- Works: Send the file through the chat connection. The client sends the bytes over its connection and the chat service stores them. Trade-off: Gateways are sized for many small messages. Large files tie up connections, and every reader pulls them back through the same servers.
- Scales: Upload to blob storage, send a reference. The client asks the chat service for an upload slot and gets a presigned URL. It uploads straight to blob storage, then sends an ordinary message whose body is a reference: object key, type, size and a checksum. On Amazon S3, a presigned URL allows an upload without the client holding any storage credentials, and one made with the SDK can last up to 7 days. Trade-off: Two steps can half-fail: the upload finishes but no message goes out. Clean up unreferenced objects on a schedule.
- Survives: Resumable uploads, thumbnails and a CDN. Phones lose signal mid-upload, so upload big files in parts and retry only the failed one. S3’s multipart upload retries just the interrupted parts and is recommended from 100 MB. A worker then makes a thumbnail, so a chat list never downloads the full file. Readers fetch media through a CDN. Trade-off: More moving parts. S3 keeps parts, and bills for them, until you complete or abort the upload, and suggests a lifecycle rule that aborts old incomplete uploads.
Why a Reference Message?#
The reference message is the point of the design. It goes through the same idempotent, numbered, stored path as any text, so media gets ordering, retries and catch-up for free. The file itself never touches a gateway.
Follow-ups Interviewers Add#
Short answers. “Judgement” marks design opinion, not a vendor’s behaviour.
| Question | Answer in one breath |
|---|---|
| What if traffic grows 10x? | 200 gateways instead of 20 at the assumed 100,000 each; the log shards by conversation. The hot spot is one huge conversation: see the exercise in the main lesson. |
| Multi-region and data residency? | Judgement: a home region and one owner per conversation, with the log copied asynchronously to a second region. State the cost: a region loss can drop the last acknowledged messages (diagram below). Residency rules decide which region may hold a conversation. |
| End-to-end encryption? | The server routes ciphertext, and sequence numbers and cursors still work on opaque bytes. WhatsApp’s engineering post says the sender encrypts separately for each recipient device and groups use the Signal Protocol’s Sender Key scheme. MLS (RFC 9420) targets group key agreement for groups of two to thousands. Cost: no server-side search or moderation, and a new device needs history from an old one. |
| Spam and abuse? | Judgement: rate limit per user and device (the 429 in the main lesson’s API), limit new accounts and group joins, check block lists before delivery, scan media in a worker, and let users report. |
| What do you monitor? | Send-to-ack and send-to-push latency at the p99, ack success rate against an SLO, connections per gateway, reconnect rate, and how often gap repair fires. Alert on what users feel, not on CPU. |
| How do you deploy? | A rolling gateway restart is a small reconnect storm, so drain a few at a time. To change storage, dual-write, backfill, compare reads, then switch. |
| What dominates cost? | Judgement: idle connections (gateway count) and media downloads, not text storage. Move old messages to cold storage and serve media through the CDN. |
| How long do you keep data? | A product and legal call. Delete-after-delivery (as WhatsApp describes) shrinks the log; keep-forever adds about 29 TB a year at the assumed load. |
What Happens When a Whole Region Goes Dark?#
This is the multi-region row at full size. Assume each conversation has a home region (judgement), and the log is copied to a second region a moment after the ack. If the home region fails, the second region can take over from its copy, but any acknowledged message still inside the copy lag is gone from that copy.
Say the choice plainly: fail over now and accept losing the tail, or wait for the home region and accept an outage. Synchronous copies close the gap and add the cross-region round trip to every send.
Media has the same shape. A reference message can reach region B before its file does. S3 Replication Time Control copies most objects in seconds and 99.9 percent within 15 minutes, so a client should show a placeholder and retry the download instead of treating a missing object as an error. Residency rules add one more constraint: a conversation’s copy may only live in a region its members’ rules allow (judgement, not legal advice).
How Do You Answer a Follow-Up?#
Read the table for its pattern. Four rows (regions, encryption, retention, abuse) are trade-offs with a price, so answer with the choice and its cost. The other four (growth, monitoring, deploys, cost) are about operating what you drew, so answer with what you would watch or change first.
Listing every technology you know instead of naming one choice and its cost. “Encryption: the server cannot search messages, so search moves to the device” beats a tour of key-exchange schemes.
A 45-Minute Plan#
Use this when the interviewer lets you pace the round. Follow-ups are the last slice, not the first.
5 min requirements, assumptions
5 min numbers and API
10 min simple design and limits
20 min two or three deep dives
5 min failures and follow-upsIf time is short, spend the last five minutes on the failure table, then offer one follow-up from the list above for the interviewer to pick.
Try It: A 250 MB Video on a Bad Connection#
A user sends a 250 MB video from a phone with a weak signal. Assume the app uploads in 10 MB parts and the signal drops after 20 parts are done.
- How many parts are there, and how much must be re-sent if the upload restarts from scratch?
- How much must be re-sent with a resumable upload?
- The last part finishes, then the app crashes before the message is sent. What state is left, and how do you clean it up?
Hint
Divide the size by the part size. Then ask which object exists in storage and whether any message points at it.
One reasonable answer
250 MB / 10 MB = 25 parts
20 done, 5 left
restart: resend 250 MB
resume: resend 50 MBRestarting re-sends all 25 parts. A resumable upload re-sends only the 5 missing parts, a fifth of the data. If the app crashes after the upload, the object exists but no message points at it. Retry sends the message with the same client id, so a second attempt is safe. For users who never return, a scheduled sweep deletes objects that nothing references after a set age. Part sizes here are assumptions, and S3 recommends multipart from 100 MB.
After the core design, expect one of: “What if the gateway dies?”, “How do you send a video?”, “Is it end-to-end encrypted?”. Start each answer with the one choice, then its cost, then a number if you have one.
Check Yourself#
1.A log node dies. Why is no acknowledged message lost?
The ack is sent only after the durable write, and the log has a copy in a second zone. Reads and writes move to the copy.
2.Why send a reference in the message instead of the video itself?
Gateways are sized for many small messages. The file goes straight to blob storage, and the message that points at it is stored, ordered and retried like any text message.
3.A conversation's owner dies. What do users see, and what does not break?
Sends to that one conversation pause until a standby takes over and continues from the last sequence number in the log. Other conversations and all stored messages are unaffected.
4.Region A fails with its copy two messages behind. What are the two choices, and what does each cost?
Fail over to region B and lose the two acknowledged messages the copy never received, or wait for region A and accept the outage. Synchronous copying avoids the loss but adds a cross-region round trip to every send.
5.What changes if the product adds end-to-end encryption?
The server routes ciphertext only, so it cannot search or moderate message text, and a new device needs history from an old one. Search moves to the device.
Tomorrow, cover the right-hand column of the failure table and say what covers each failure. Then name the one failure that users can feel.
In One Picture#
Failures
all end at the durable log
Media
presigned upload + reference
Follow-ups
one choice, then its cost
Deploys
drain a few gateways at a time
Plan
failures get the last 5 minutes
Rule
the log is the truthConclusion#
The 60-second follow-up answer: nothing but the log is the source of truth, so any gateway, route table, Pub/Sub or push failure costs a reconnect at most. Media goes to blob storage with a presigned URL and travels as a reference message. For encryption, regions and retention, name the choice and its price.
Which follow-up has caught you out before? Tell us in the comments.
- The core design: Design a Chat System: Store First, Then Push.
- Where the object bytes live: CDN and object storage.
- Pushing to a sleeping device: design a notification service, the next problem in Module 4.
