Rate limiting sounds like a bouncer — a way to keep bad actors out. That framing is why so many teams add it too late, thinking of it as an anti-abuse feature they’ll get to eventually. Then one afternoon a single client’s retry loop, or one script someone left running, buries an unprotected endpoint and takes the whole service down for everyone else.
That’s the reframe: rate limiting is self-defense, not punishment. Its main job is protecting your system from traffic it can’t handle — abusive, buggy, or just unexpectedly popular — before that traffic protects itself right into an outage. This post is part of the System Design Foundations series, anchored by the System Design Building Blocks overview, and it covers the algorithms and the one you should reach for by default.
- Rate limiting protects the system from itself — a retry storm or one runaway client is the real threat, not just malicious abuse.
- Token bucket is the best general default: it enforces a steady average rate while allowing short, controlled bursts.
- For LLMs, you limit by tokens, not requests — and you have to respect the provider’s limits, not just impose your own.
What Rate Limiting Is Really For#
A rate limiter caps how much any one client can send in a window, and returns 429 Too Many Requests when they go over. The obvious use is stopping abuse — but the more common save is protecting you from accidents. A client with a buggy retry loop can hammer you thousands of times a second without meaning any harm; a viral moment sends ten times your normal traffic; one internal job misfires. Cloudflare’s rate limiting overview frames it exactly this way: it’s about keeping a service available under load, not just blocking attackers. Stripe, in its classic rate limiters post, goes further — they run several limiters, including ones that shed load to protect the platform during incidents. Rate limiting is a reliability tool that happens to also stop abuse.
The Four Algorithms#
Almost every limiter is one of these four. The difference is how they count.
| Algorithm | How it works | Trade-off |
|---|---|---|
| Fixed window | Count requests per fixed interval, reset each window | Dead simple; allows a 2× burst at the window boundary |
| Sliding window | Weight the count across the moving window | Smooths the boundary burst; the practical default at scale |
| Token bucket | Tokens refill at a steady rate; each request spends one | Allows controlled bursts and a steady average — best general default |
| Leaky bucket | Requests queue and drain at a constant rate | Smooths bursts into steady output; made for traffic shaping |
The fixed window is tempting because it’s trivial, but it has a nasty edge case: a client can fire a full window’s worth of requests at the end of one window and another full batch at the start of the next, sneaking through 2× the limit in an instant. The token bucket avoids that while allowing the bursts real clients actually make.
Picture the token bucket above: tokens drip in at a fixed rate up to a cap, and every request spends one. When traffic is quiet, tokens build up, so a short burst can be served immediately. When the bucket empties, requests get a 429 until it refills. That single mechanism gives you a firm average rate and tolerance for the spiky-but-legitimate traffic real users generate.
Token Bucket in Code#
The whole algorithm is a few lines — refill based on elapsed time, then try to spend a token:
# Token bucket: refill at a steady rate, spend one token per request.
def allow(bucket):
now = time.monotonic()
elapsed = now - bucket.last
bucket.tokens = min(bucket.capacity,
bucket.tokens + elapsed * bucket.refill_rate)
bucket.last = now
if bucket.tokens >= 1:
bucket.tokens -= 1
return True # allowed
return False # empty bucket -> 429 Too Many RequestsTwo numbers define the policy: capacity (how big a burst you tolerate) and refill_rate (the steady requests-per-second you allow). That’s the whole reason token bucket is the default — the behaviour you want is just two tunable knobs.
Which One to Use#
Default to token bucket for public APIs — it matches real usage and gives you burst-plus-average in two parameters. At large distributed scale, where every limiter check costs coordination, a sliding window counter is usually the better compromise, as Kong’s scalable rate limiting write-up lays out. Reach for a leaky bucket when you specifically need to protect a fragile downstream by smoothing bursts into a steady stream. Keep fixed window only for simple internal limits where the boundary burst doesn’t matter.
The default that’s right most of the time: a token bucket, sized so capacity is the burst you can absorb and refill_rate is the sustained load you can serve. Everything fancier is for a specific problem — reach for it only when you have that problem.
Always return the limit in your response headers (RateLimit-Remaining, Retry-After). A limiter that rejects silently just turns into a mystery outage on the client side — tell callers exactly when to try again.
Rate Limiting LLM Traffic#
Rate limiting for AI systems keeps the same algorithms but changes the unit — and that change matters more than it looks.
You limit by tokens, not requests. A request-per-minute cap is meaningless when one call is a 50-token question and the next is a 50,000-token document summary. LLM providers meter you in tokens per minute, so your own limiter has to think in tokens and cost, not request count — the same shift the AI gateway makes when it budgets traffic by dollars.
The provider rate-limits you, so backoff is mandatory. Your app isn’t just imposing limits; it’s living under the model provider’s. Exceed their TPM and you get 429s that fail user requests, so production LLM apps need client-side throttling and exponential backoff to stay under the cap. And because inference is expensive and bursty, the better move is often to queue rather than reject — a leaky-bucket-style buffer in front of the model, which is the queue-in-front-of-inference pattern again, plus semantic caching to cut the number of calls you make in the first place.
AI-era rule of thumb: rate-limit by tokens per minute and cost, not request count; add exponential backoff so the provider’s limits don’t crash your app; and prefer queuing expensive inference over rejecting it, since a dropped LLM call is a lost user.
Quick Recap#
- Rate limiting is reliability, not just security — it protects you from retry storms and spikes.
- Four algorithms: fixed window (simple, edge-burst), sliding window (scale default), token bucket (general default), leaky bucket (traffic shaping).
- Token bucket wins because burst-plus-average is just two knobs,
capacityandrefill_rate. - Always tell clients the limit via headers and
Retry-After. - For LLMs, limit by tokens and cost, respect the provider’s limits with backoff, and queue rather than drop.
Conclusion#
The reason rate limiting gets added too late is that it’s mislabeled as an anti-abuse feature — something for when you have attackers. But the outage that finally makes a team take it seriously is almost never an attack; it’s a retry loop, a viral spike, or a script nobody remembered was running. Put a token bucket in front of anything that can be overwhelmed, size it to the load you can actually serve, and tell callers when to come back. It’s a small amount of code standing between one misbehaving client and everyone else’s bad afternoon.
What took you down first — a client’s retry storm, a viral spike, or one script hammering an endpoint nobody thought to protect? Tell me what melted.
- API Gateway Explained — where rate limiting usually lives, alongside auth and routing.
- Caching Strategies — the other half of protecting a service under load: serve less, compute less.
- System Design Building Blocks — where rate limiting sits among the cross-cutting concerns.
