# Design a Rate Limiter: Failure Modes, Security and Follow-Ups (2026)
> The production half of the rate limiter problem: what breaks when each part dies, which client addresses to trust, rejecting versus delaying versus shedding, cost-based limits, several regions, safe rollout, and the follow-up questions interviewers ask next.
*Source: https://www.infowok.com/design-a-rate-limiter-follow-ups/ · Sukhveer Kaur · Published October 24, 2026*
---
The limiter in the diagram works. Then the follow-ups start. A client invents an `X-Forwarded-For` header and gets a fresh allowance on every call. A new rule ships on Monday and blocks real customers by nine. A second region opens and every check now crosses an ocean. The monthly cap arrives and every retry fails for weeks.
None of that changes the core design, yet all of it decides whether the design survives production. Interviewers often spend the last fifteen minutes here. This lesson is the second half of [Design a rate limiter](/design-a-rate-limiter/), which builds the design itself.
- [Design a rate limiter](/design-a-rate-limiter/): the gateway, the shared counters and the fail-open, local and fail-closed options this lesson stress-tests.
- [Rate limiting explained](/rate-limiting-explained/) for the algorithms, and [API gateways](/api-gateway-explained/) for where the check runs.
When the shared counter store cannot be reached, what does a gateway do for ordinary reads and for login? Reads fall back to the gateway's local bucket or fail open. Login fails closed, because unlimited attempts are the risk the limiter exists to stop.
- Ask "what if it dies?" for every part. The limit is approximate by design, so most failures cost a few extra requests.
- Trust a client address only from a header your own proxy sets, and do not mistake a limiter for DDoS protection.
- Rejecting, delaying and shedding differ: a limit caps one caller, shedding protects the service.
- Count cost where requests differ: estimate on arrival, correct afterwards. The same `429` can mean "wait a second" or "wait until next month".
- Across regions, pick regional shares, a home region or replicated counters, and say which accuracy you gave up.
- Roll out a new rule in shadow mode and watch four signals before enforcing it.
Stress-test a rate limiter design: say what breaks when each part fails, close the common abuse holes, choose between rejecting, delaying and shedding, charge by cost, place counters across regions, roll out a rule safely, and answer the usual follow-ups in one line each.
## What Breaks When Each Part Dies?
Take the final design from the core lesson and kill its parts one at a time. An interviewer asks this to see whether you design for failure or only for the happy path.
| Part | What happens | Why it is fine |
|---|---|---|
| One gateway | Its local buckets vanish | Shared counters are untouched |
| A shard primary | Failover promotes a replica, a few counts lost | The limit is approximate |
| Whole counter store | Checks time out | Per-route open, local or closed |
| Rules store | Gateways keep the last good rules | Alert on rules age |
In CAP terms this is an availability choice: cut off from its counters, a gateway keeps answering from a local, possibly stale limit. In PACELC terms, even with no failure you trade a little accuracy (replicas, local buckets) for lower latency. Both are design judgement.
Nothing on the request path is allowed to be a single point of failure, and every failure has a named, tested fallback. A row that reads "we would find out from users" is the one to fix.
## Which Client Address Do You Trust?
A limiter keyed by IP address is only as honest as the address. Trust a client address only from a header your own proxy sets: `X-Forwarded-For` is forgeable unless a trusted proxy overwrites it, which is why [AWS WAF lets a rule pick the header](https://docs.aws.amazon.com/waf/latest/developerguide/waf-rule-statement-type-rate-based-high-level-settings.html) it reads the IP from.
Two more production limits on what a limiter does:
- A limiter is not DDoS protection. Floods are absorbed at the CDN or edge, or the counter store becomes the target.
- Keys hold IPs and user IDs, personal data in many places. Keep them short-lived and out of logs.
The first limit is why the core design lists network-level attack protection as out of scope. The second is why a key's TTL matters beyond memory: it is also a retention policy.
## Reject, Delay or Shed?
A hard limit refuses at once; a soft limit delays or treats the number as a target. Real systems differ:
- **NGINX `limit_req`** is a leaky bucket. Per its [docs](https://nginx.org/en/docs/http/ngx_http_limit_req_module.html), `burst` defaults to 0, excess within the burst is delayed unless `nodelay`, and the rest gets 503 unless you set `limit_req_status`. Ask for `429`.
- **Amazon API Gateway** throttles with a token bucket and calls its limits [best-effort](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-request-throttling.html), "targets rather than guaranteed request ceilings". Your design can be approximate too.
- **AWS WAF** rate-based rules count over a [60 to 600 second window](https://docs.aws.amazon.com/waf/latest/developerguide/waf-rule-statement-type-rate-based-high-level-settings.html) and do not guarantee an exact match. That is the edge layer, before your gateway.
- **Stripe** in 2017 [ran four limiters](https://stripe.com/blog/rate-limiters): a request rate limiter, a concurrent requests limiter ("20 API requests in progress at the same time"), and two load shedders that drop non-critical requests (with `503`) to protect critical ones such as creating charges. Limiting in-flight work protects slow endpoints that a per-second rate does not.
Rate limiting is not load shedding: a limit stops one caller taking more than its share; shedding protects the service when total load is too high, dropping by importance.
## Is One Request Always One Unit?
One request carries 50 input tokens and the next carries 200,000, [the size gap the Claude docs use](https://platform.claude.com/docs/en/api/rate-limits). Both count as "one request" against a limit of 60 a minute. Which one should the limiter charge more, and when does it find out the real size?
The bigger one should cost more, and often the limiter only learns the real size after the work is done. The Claude API [limits](https://platform.claude.com/docs/en/api/rate-limits) requests, input tokens and output tokens per minute with a continuously refilled token bucket, so 60 a minute can be enforced as 1 a second. Input tokens are estimated at the start and adjusted to the actual count, and for most models cached input does not count.
An AI gateway's limiter can copy that:
1. Spend the estimated tokens on arrival.
2. Refund or charge the difference when real usage is known.
The [AI gateway lesson](/ai-gateway-design/) shows where that fits.
At its monthly spend cap, a Claude organization also gets `429` with error type `rate_limit_error`, but no `retry-after`, and retries fail until 00:00 UTC on the first of the next month. Only `error_code: enforced_spend_limit_reached` tells the two apart. A client that retries every `429` will hammer the API for weeks. Send `Retry-After` only when waiting helps, and say which limit was hit.
## One Region or Many?
One global store puts a cross-region round trip on every check and breaks the latency budget. Three answers; the first two are design judgement, not measured results:
- **Regional shares:** each region gets a share (a third each for three) and a slow background sync rebalances, accepting drift for a client who moves.
- **Home region:** for login, every check for a key goes to one region and pays the latency, because the exact limit matters more.
- **Replicated counters:** Redis [documents Active-Active replication](https://redis.io/docs/latest/develop/use-cases/rate-limiter/) (CRDT-based) for limits enforced across regions. Regions converge, so each can briefly overspend.
Ordinary reads can take regional shares, where a little drift costs nothing. Login takes a home region, where exactness is the point and the extra delay is acceptable.
## Can Checks Stay Local?
A third shape avoids a call per request. Envoy's [rate limit quota filter](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/rate_limit_quota_filter) has each proxy report its load every `reporting_interval` and receive a quota assignment from a central service, which rebalances quota between proxies. Checks stay local, so the cost is accuracy within one interval (judgement). Check the API's maturity first.
It is the same trade as the regional shares above, applied between gateways in one region: spend locally, settle centrally, accept drift for one interval.
## How Do You Know the Limits Are Right?
Watch four signals:
- The `429` rate per rule. A jump right after a rule change usually means the rule is wrong, not the clients.
- Check latency (p99), with an SLO such as 99.9% of checks under 5 ms (an assumption to tune).
- Limiter errors, and any sustained fail-open, which should alert.
- Store memory and evictions.
So roll a new rule out in shadow mode first: count what it would reject, but let the request through. Stripe's write-up gives the same advice ("dark launch"), and Envoy's local rate limit filter [supports this](https://github.com/envoyproxy/envoy/blob/main/docs/root/configuration/http/http_filters/local_rate_limit_filter.rst): a request can be checked without being enforced, and stats count `rate_limited` apart from `enforced`.
## Practice: Ship a Tighter Limit
An API route allows **100 requests a minute per key**. Abuse is growing, so the team wants **60**. Assume 5 percent of keys send more than 60 a minute at peak. Write the rollout:
1. How do you learn who the change would hit?
2. What do you watch, and what would make you stop?
3. How do clients find out?
The 5 percent is an assumption, not a measurement. The shadow run replaces it with the real figure.
```text
Day 1 shadow at 60, allow all
count would-reject per key
Day 2 read the list: abuse or
real customers?
Day 3 enforce for 10% of keys
watch 429 rate, latency
Day 4 enforce for all, if clean
```
Shadow mode turns the guess into a count and a list of keys. Tell the heavy legitimate keys before enforcing, and move them to a bigger plan instead of making exceptions. Stop and fall back to shadow if the 429 rate for ordinary keys jumps, since that means the rule is wrong, not the clients. Send `Retry-After` on every `429` so clients wait instead of retrying at once.
## The 45-Minute Plan
The core lesson fills the first 35 minutes. This is the full round:
| Minutes | Do |
|---|---|
| 0 to 5 | Ask the questions, state requirements and out of scope |
| 5 to 10 | Entities, API, key layout, estimate |
| 10 to 18 | One gateway, then break it with twenty |
| 18 to 35 | Deep dives: algorithm (5), shared atomic counters and sharding (7), failure mode (5) |
| 35 to 42 | Identification, heavy clients, regions, cost |
| 42 to 45 | Recap and invite follow-ups |
This pacing is a suggestion, not a rule. Skip the algorithm tour before skipping the distributed part.
For the closing 60-second answer, add one sentence from this lesson: how you would roll the limit out (shadow first) and which signal would tell you it is wrong.
Expect follow-ups. Answer each with a trade-off, not a tool name:
| They ask | Answer in one line |
|---|---|
| Twenty gateways? | Shared, atomic counters |
| Two requests at once? | One Lua script, store clock |
| Redis dies? | Open, local or closed, per route |
| Ten times the traffic? | More shards by key, local bucket first |
| One client is half the load? | Local reject, blocklist, bigger plan |
| Change a limit live? | Rules store; clamp tokens to the new capacity |
| LLM API? | Count tokens: estimate, then correct |
| Client forges its IP header? | Read it only from your own proxy |
| A flood from many sources? | Absorb at the edge, not the store |
| Second region? | Shares, home region or replicas, by route |
| New limit blocks real users? | Shadow first, then enforce in steps |
| Monthly cap hit? | Same 429, no Retry-After; say which limit |
## Check Yourself
Any client can send it, so a forged value gets a fresh allowance. It is safe only when a trusted proxy you run overwrites it, so the value comes from your own infrastructure.
On arrival it spends the 2,000 estimate. When the real usage of 1,200 is known it refunds the 800 difference, so the next request sees the true remaining allowance.
One global store would put a cross-region trip on every check. Choose by route: regional shares for ordinary traffic, a home region for login, or replicated counters that converge and can briefly overspend. Say which accuracy you gave up.
## In One Picture
```text
What breaks
nothing is a single point of
failure; each has a fallback
Who to trust
headers from your own proxy
Reject / delay / shed
limit one caller; shed to save
the service
Cost
estimate, then correct
Regions
shares, home region, replicas
Rollout
shadow, watch 429s, enforce
```
## Conclusion
The core design is a small data structure. The production answer is everything around it: a fallback for every part, addresses you can trust, the right tool for rejecting versus shedding, cost where requests differ, a plan per region, and a rollout that proves a rule before it bites. Say which accuracy you gave up each time and the follow-ups become easy.
**Which follow-up would catch you out in a live round?** Tell us in the comments.
- **Back to the core design:** [design a rate limiter](/design-a-rate-limiter/) for the token bucket, atomic counters, sharding and failure modes.
- **Where the limiter sits:** [API gateways](/api-gateway-explained/) and [AI gateway design](/ai-gateway-design/).
- **Coming next in the course:** the news feed.