Design a Rate Limiter: Failure Modes, Security and Follow-Ups (2026)

The limiter in the diagram works. Then the follow-ups start. A client invents an X-Forwarded-For header and gets a fresh allowance on every call. A new rule ships on Monday and blocks real customers by nine. A second region opens and every check now crosses an ocean. The monthly cap arrives and every retry fails for weeks.

None of that changes the core design, yet all of it decides whether the design survives production. Interviewers often spend the last fifteen minutes here. This lesson is the second half of Design a rate limiter, which builds the design itself.

Intermediate30 min

Before you start

Design a rate limiter: the gateway, the shared counters and the fail-open, local and fail-closed options this lesson stress-tests.
Rate limiting explained for the algorithms, and API gateways for where the check runs.

Remember from the core lesson

When the shared counter store cannot be reached, what does a gateway do for ordinary reads and for login? Reads fall back to the gateway’s local bucket or fail open. Login fails closed, because unlimited attempts are the risk the limiter exists to stop.

Key takeaways

Ask “what if it dies?” for every part. The limit is approximate by design, so most failures cost a few extra requests.
Trust a client address only from a header your own proxy sets, and do not mistake a limiter for DDoS protection.
Rejecting, delaying and shedding differ: a limit caps one caller, shedding protects the service.
Count cost where requests differ: estimate on arrival, correct afterwards. The same 429 can mean “wait a second” or “wait until next month”.
Across regions, pick regional shares, a home region or replicated counters, and say which accuracy you gave up.
Roll out a new rule in shadow mode and watch four signals before enforcing it.

After this lesson you can

Stress-test a rate limiter design: say what breaks when each part fails, close the common abuse holes, choose between rejecting, delaying and shedding, charge by cost, place counters across regions, roll out a rule safely, and answer the usual follow-ups in one line each.

What Breaks When Each Part Dies?#

Take the final design from the core lesson and kill its parts one at a time. An interviewer asks this to see whether you design for failure or only for the happy path.

Part	What happens	Why it is fine
One gateway	Its local buckets vanish	Shared counters are untouched
A shard primary	Failover promotes a replica, a few counts lost	The limit is approximate
Whole counter store	Checks time out	Per-route open, local or closed
Rules store	Gateways keep the last good rules	Alert on rules age

In CAP terms this is an availability choice: cut off from its counters, a gateway keeps answering from a local, possibly stale limit. In PACELC terms, even with no failure you trade a little accuracy (replicas, local buckets) for lower latency. Both are design judgement.

The pattern behind the table

Nothing on the request path is allowed to be a single point of failure, and every failure has a named, tested fallback. A row that reads “we would find out from users” is the one to fix.

Which Client Address Do You Trust?#

A limiter keyed by IP address is only as honest as the address. Trust a client address only from a header your own proxy sets: X-Forwarded-For is forgeable unless a trusted proxy overwrites it, which is why AWS WAF lets a rule pick the header it reads the IP from.

Two more production limits on what a limiter does:

A limiter is not DDoS protection. Floods are absorbed at the CDN or edge, or the counter store becomes the target.
Keys hold IPs and user IDs, personal data in many places. Keep them short-lived and out of logs.

The first limit is why the core design lists network-level attack protection as out of scope. The second is why a key’s TTL matters beyond memory: it is also a retention policy.

Reject, Delay or Shed?#

A hard limit refuses at once; a soft limit delays or treats the number as a target. Real systems differ:

NGINX limit_req is a leaky bucket. Per its docs, burst defaults to 0, excess within the burst is delayed unless nodelay, and the rest gets 503 unless you set limit_req_status. Ask for 429.
Amazon API Gateway throttles with a token bucket and calls its limits best-effort, “targets rather than guaranteed request ceilings”. Your design can be approximate too.
AWS WAF rate-based rules count over a 60 to 600 second window and do not guarantee an exact match. That is the edge layer, before your gateway.
Stripe in 2017 ran four limiters: a request rate limiter, a concurrent requests limiter (“20 API requests in progress at the same time”), and two load shedders that drop non-critical requests (with 503) to protect critical ones such as creating charges. Limiting in-flight work protects slow endpoints that a per-second rate does not.

Rate limiting is not load shedding: a limit stops one caller taking more than its share; shedding protects the service when total load is too high, dropping by importance.

Is One Request Always One Unit?#

Predict first

One request carries 50 input tokens and the next carries 200,000, the size gap the Claude docs use. Both count as “one request” against a limit of 60 a minute. Which one should the limiter charge more, and when does it find out the real size?

The bigger one should cost more, and often the limiter only learns the real size after the work is done. The Claude API limits requests, input tokens and output tokens per minute with a continuously refilled token bucket, so 60 a minute can be enforced as 1 a second. Input tokens are estimated at the start and adjusted to the actual count, and for most models cached input does not count.

An AI gateway’s limiter can copy that:

Spend the estimated tokens on arrival.
Refund or charge the difference when real usage is known.

The AI gateway lesson shows where that fits.

Same 429, different meaning

At its monthly spend cap, a Claude organization also gets 429 with error type rate_limit_error, but no retry-after, and retries fail until 00:00 UTC on the first of the next month. Only error_code: enforced_spend_limit_reached tells the two apart. A client that retries every 429 will hammer the API for weeks. Send Retry-After only when waiting helps, and say which limit was hit.

One Region or Many?#

One global store puts a cross-region round trip on every check and breaks the latency budget. Three answers; the first two are design judgement, not measured results:

Regional shares: each region gets a share (a third each for three) and a slow background sync rebalances, accepting drift for a client who moves.
Home region: for login, every check for a key goes to one region and pays the latency, because the exact limit matters more.
Replicated counters: Redis documents Active-Active replication (CRDT-based) for limits enforced across regions. Regions converge, so each can briefly overspend.

Pick by route, not by system

Ordinary reads can take regional shares, where a little drift costs nothing. Login takes a home region, where exactness is the point and the extra delay is acceptable.

Can Checks Stay Local?#

A third shape avoids a call per request. Envoy’s rate limit quota filter has each proxy report its load every reporting_interval and receive a quota assignment from a central service, which rebalances quota between proxies. Checks stay local, so the cost is accuracy within one interval (judgement). Check the API’s maturity first.

It is the same trade as the regional shares above, applied between gateways in one region: spend locally, settle centrally, accept drift for one interval.

How Do You Know the Limits Are Right?#

Watch four signals:

The 429 rate per rule. A jump right after a rule change usually means the rule is wrong, not the clients.
Check latency (p99), with an SLO such as 99.9% of checks under 5 ms (an assumption to tune).
Limiter errors, and any sustained fail-open, which should alert.
Store memory and evictions.

So roll a new rule out in shadow mode first: count what it would reject, but let the request through. Stripe’s write-up gives the same advice (“dark launch”), and Envoy’s local rate limit filter supports this: a request can be checked without being enforced, and stats count rate_limited apart from enforced.

Practice: Ship a Tighter Limit#

Exercise · Roll out a lower limit on a busy route15 min

An API route allows 100 requests a minute per key. Abuse is growing, so the team wants 60. Assume 5 percent of keys send more than 60 a minute at peak. Write the rollout:

How do you learn who the change would hit?
What do you watch, and what would make you stop?
How do clients find out?

Hint

The 5 percent is an assumption, not a measurement. The shadow run replaces it with the real figure.

One reasonable answer

text

Day 1  shadow at 60, allow all
       count would-reject per key
Day 2  read the list: abuse or
       real customers?
Day 3  enforce for 10% of keys
       watch 429 rate, latency
Day 4  enforce for all, if clean

Shadow mode turns the guess into a count and a list of keys. Tell the heavy legitimate keys before enforcing, and move them to a bigger plan instead of making exceptions. Stop and fall back to shadow if the 429 rate for ordinary keys jumps, since that means the rule is wrong, not the clients. Send Retry-After on every 429 so clients wait instead of retrying at once.

The 45-Minute Plan#

The core lesson fills the first 35 minutes. This is the full round:

Minutes	Do
0 to 5	Ask the questions, state requirements and out of scope
5 to 10	Entities, API, key layout, estimate
10 to 18	One gateway, then break it with twenty
18 to 35	Deep dives: algorithm (5), shared atomic counters and sharding (7), failure mode (5)
35 to 42	Identification, heavy clients, regions, cost
42 to 45	Recap and invite follow-ups

This pacing is a suggestion, not a rule. Skip the algorithm tour before skipping the distributed part.

For the closing 60-second answer, add one sentence from this lesson: how you would roll the limit out (shadow first) and which signal would tell you it is wrong.

In an interview

Expect follow-ups. Answer each with a trade-off, not a tool name:

They ask	Answer in one line
Twenty gateways?	Shared, atomic counters
Two requests at once?	One Lua script, store clock
Redis dies?	Open, local or closed, per route
Ten times the traffic?	More shards by key, local bucket first
One client is half the load?	Local reject, blocklist, bigger plan
Change a limit live?	Rules store; clamp tokens to the new capacity
LLM API?	Count tokens: estimate, then correct
Client forges its IP header?	Read it only from your own proxy
A flood from many sources?	Absorb at the edge, not the store
Second region?	Shares, home region or replicas, by route
New limit blocks real users?	Shadow first, then enforce in steps
Monthly cap hit?	Same 429, no Retry-After; say which limit

Check Yourself#

Check yourself3 questions · answer, then open

1.Why is `X-Forwarded-For` unsafe to key a limiter on, and when is it safe?

Any client can send it, so a forged value gets a fresh allowance. It is safe only when a trusted proxy you run overwrites it, so the value comes from your own infrastructure.

2.A request is estimated at 2,000 tokens and the work uses 1,200. What does the limiter do on arrival and afterwards?

On arrival it spends the 2,000 estimate. When the real usage of 1,200 is known it refunds the 800 difference, so the next request sees the true remaining allowance.

3.What changes if the same limiter must also run in a second region?

One global store would put a cross-region trip on every check. Choose by route: regional shares for ordinary traffic, a home region for login, or replicated counters that converge and can briefly overspend. Say which accuracy you gave up.

In One Picture#

text

What breaks
  nothing is a single point of
  failure; each has a fallback
Who to trust
  headers from your own proxy
Reject / delay / shed
  limit one caller; shed to save
  the service
Cost
  estimate, then correct
Regions
  shares, home region, replicas
Rollout
  shadow, watch 429s, enforce

Conclusion#

The core design is a small data structure. The production answer is everything around it: a fallback for every part, addresses you can trust, the right tool for rejecting versus shedding, cost where requests differ, a plan per region, and a rollout that proves a rule before it bites. Say which accuracy you gave up each time and the follow-ups become easy.

Which follow-up would catch you out in a live round? Tell us in the comments.

Where to go from here

Back to the core design: design a rate limiter for the token bucket, atomic counters, sharding and failure modes.
Where the limiter sits: API gateways and AI gateway design.
Coming next in the course: the news feed.