# Rate Limiting Explained: Token Bucket, Leaky Bucket, and When Each Wins (2026)
> Rate limiting looks like a way to punish abusers. It's really self-defense — one client's retry storm or a runaway script can take down a service for everyone. The four algorithms, the one that's the right default, and how it all changes for token-metered LLM traffic.
*Source: https://www.infowok.com/rate-limiting-explained/ · Navmeet Kaur · Published July 6, 2026*
---
Rate limiting sounds like a bouncer — a way to keep bad actors out. That framing is why so many teams add it too late, thinking of it as an anti-abuse feature they'll get to eventually. Then one afternoon a single client's retry loop, or one script someone left running, buries an unprotected endpoint and takes the whole service down for everyone else.
That's the reframe: **rate limiting is self-defense, not punishment.** Its main job is protecting your system from traffic it can't handle — abusive, buggy, or just unexpectedly popular — before that traffic protects itself right into an outage. This post is part of the **System Design Foundations** series, anchored by the [System Design Building Blocks](/system-design-building-blocks/) overview, and it covers the algorithms and the one you should reach for by default.
- **Rate limiting protects the system from itself** — a retry storm or one runaway client is the real threat, not just malicious abuse.
- **Token bucket is the best general default:** it enforces a steady average rate while allowing short, controlled bursts.
- **For LLMs, you limit by tokens, not requests** — and you have to respect the provider's limits, not just impose your own.
## What Rate Limiting Is Really For
A rate limiter caps how much any one client can send in a window, and returns `429 Too Many Requests` when they go over. The obvious use is stopping abuse — but the more common save is protecting you from *accidents*. A client with a buggy retry loop can hammer you thousands of times a second without meaning any harm; a viral moment sends ten times your normal traffic; one internal job misfires. Cloudflare's [rate limiting overview](https://www.cloudflare.com/learning/bots/what-is-rate-limiting/) frames it exactly this way: it's about keeping a service available under load, not just blocking attackers. Stripe, in its classic [rate limiters post](https://stripe.com/blog/rate-limiters), goes further — they run *several* limiters, including ones that shed load to protect the platform during incidents. Rate limiting is a reliability tool that happens to also stop abuse.
## The Four Algorithms
Almost every limiter is one of these four. The difference is how they count.
| Algorithm | How it works | Trade-off |
|---|---|---|
| **Fixed window** | Count requests per fixed interval, reset each window | Dead simple; allows a 2× burst at the window boundary |
| **Sliding window** | Weight the count across the moving window | Smooths the boundary burst; the practical default at scale |
| **Token bucket** | Tokens refill at a steady rate; each request spends one | Allows controlled bursts *and* a steady average — best general default |
| **Leaky bucket** | Requests queue and drain at a constant rate | Smooths bursts into steady output; made for traffic shaping |
The **fixed window** is tempting because it's trivial, but it has a nasty edge case: a client can fire a full window's worth of requests at the end of one window and another full batch at the start of the next, sneaking through 2× the limit in an instant. The **token bucket** avoids that while allowing the bursts real clients actually make.
Picture the token bucket above: tokens drip in at a fixed rate up to a cap, and every request spends one. When traffic is quiet, tokens build up, so a short burst can be served immediately. When the bucket empties, requests get a `429` until it refills. That single mechanism gives you a firm average rate *and* tolerance for the spiky-but-legitimate traffic real users generate.
## Token Bucket in Code
The whole algorithm is a few lines — refill based on elapsed time, then try to spend a token:
```python
# Token bucket: refill at a steady rate, spend one token per request.
def allow(bucket):
now = time.monotonic()
elapsed = now - bucket.last
bucket.tokens = min(bucket.capacity,
bucket.tokens + elapsed * bucket.refill_rate)
bucket.last = now
if bucket.tokens >= 1:
bucket.tokens -= 1
return True # allowed
return False # empty bucket -> 429 Too Many Requests
```
Two numbers define the policy: `capacity` (how big a burst you tolerate) and `refill_rate` (the steady requests-per-second you allow). That's the whole reason token bucket is the default — the behaviour you want is just two tunable knobs.
## Which One to Use
Default to **token bucket** for public APIs — it matches real usage and gives you burst-plus-average in two parameters. At large **distributed** scale, where every limiter check costs coordination, a **sliding window counter** is usually the better compromise, as Kong's [scalable rate limiting write-up](https://konghq.com/blog/engineering/how-to-design-a-scalable-rate-limiting-algorithm) lays out. Reach for a **leaky bucket** when you specifically need to protect a fragile downstream by smoothing bursts into a steady stream. Keep **fixed window** only for simple internal limits where the boundary burst doesn't matter.
The default that's right most of the time: a token bucket, sized so `capacity` is the burst you can absorb and `refill_rate` is the sustained load you can serve. Everything fancier is for a specific problem — reach for it only when you have that problem.
Always return the limit in your response headers (`RateLimit-Remaining`, `Retry-After`). A limiter that rejects silently just turns into a mystery outage on the client side — tell callers exactly when to try again.
## Rate Limiting LLM Traffic
Rate limiting for AI systems keeps the same algorithms but changes the unit — and that change matters more than it looks.
**You limit by tokens, not requests.** A request-per-minute cap is meaningless when one call is a 50-token question and the next is a 50,000-token document summary. LLM providers meter you in *tokens per minute*, so your own limiter has to think in tokens and cost, not request count — the same shift the [AI gateway](/api-gateway-explained/) makes when it budgets traffic by dollars.
**The provider rate-limits *you*, so backoff is mandatory.** Your app isn't just imposing limits; it's living under the model provider's. Exceed their TPM and you get `429`s that fail user requests, so production LLM apps need client-side throttling and exponential backoff to stay under the cap. And because inference is expensive and bursty, the better move is often to *queue* rather than reject — a leaky-bucket-style buffer in front of the model, which is the [queue-in-front-of-inference](/kafka-vs-rabbitmq/) pattern again, plus [semantic caching](/caching-strategies/) to cut the number of calls you make in the first place.
AI-era rule of thumb: rate-limit by tokens per minute and cost, not request count; add exponential backoff so the provider's limits don't crash your app; and prefer queuing expensive inference over rejecting it, since a dropped LLM call is a lost user.
## Quick Recap
- **Rate limiting is reliability, not just security** — it protects you from retry storms and spikes.
- **Four algorithms:** fixed window (simple, edge-burst), sliding window (scale default), token bucket (general default), leaky bucket (traffic shaping).
- **Token bucket wins** because burst-plus-average is just two knobs, `capacity` and `refill_rate`.
- **Always tell clients the limit** via headers and `Retry-After`.
- **For LLMs,** limit by tokens and cost, respect the provider's limits with backoff, and queue rather than drop.
## Conclusion
The reason **rate limiting** gets added too late is that it's mislabeled as an anti-abuse feature — something for when you have attackers. But the outage that finally makes a team take it seriously is almost never an attack; it's a retry loop, a viral spike, or a script nobody remembered was running. Put a token bucket in front of anything that can be overwhelmed, size it to the load you can actually serve, and tell callers when to come back. It's a small amount of code standing between one misbehaving client and everyone else's bad afternoon.
**What took you down first — a client's retry storm, a viral spike, or one script hammering an endpoint nobody thought to protect?** Tell me what melted.
- **[API Gateway Explained](/api-gateway-explained/)** — where rate limiting usually lives, alongside auth and routing.
- **[Caching Strategies](/caching-strategies/)** — the other half of protecting a service under load: serve less, compute less.
- **[System Design Building Blocks](/system-design-building-blocks/)** — where rate limiting sits among the cross-cutting concerns.