“Isn’t an API gateway just a fancy load balancer?” It’s one of the most common mix-ups in system design, and it’s understandable — both sit in front of your backend, both take in requests and send them onward. But they answer completely different questions, and treating one as the other leads to real mistakes: auth logic scattered across services, or a gateway wheezing under work a load balancer should be doing.
A load balancer asks “which server should handle this?” An API gateway asks “should this request be allowed at all, and where does it belong?” One is about distribution; the other is about control. This post is part of the System Design Foundations series, anchored by the System Design Building Blocks overview, and it draws the line clearly — right after Load Balancers, its most-confused neighbour.
- Load balancer = distribution, API gateway = control. One picks a healthy server; the other decides if a request is allowed and where it goes.
- The gateway is the single home for cross-cutting concerns — auth, rate limiting, routing, aggregation — so services don’t each reimplement them.
- They work together: a gateway routes the request, and a load balancer behind it picks the instance.
API Gateway vs Load Balancer: The Confusion#
Start with the distinction, because everything else follows from it. A load balancer treats its backends as interchangeable copies. It doesn’t care what the request says; it cares which server is healthy and least busy. An API gateway cares deeply about what the request says — is this token valid, has this client blown its rate limit, does this path go to the orders service or the users service? IBM’s comparison of the two puts it well: the load balancer optimizes distribution and availability, while the gateway provides application-aware control.
They’re not competitors — they’re neighbours in the request path. As Kong’s breakdown describes the common shape: the gateway authenticates, applies limits, and routes to the right service, and a load balancer in front of that service picks the right instance. Gateway first for the “should this and where,” load balancer second for the “which one.”
What an API Gateway Actually Does#
The gateway earns its place by being the one spot where you handle everything that would otherwise be copy-pasted into every service.
| Job | What it means |
|---|---|
| Authentication / authorization | Verify the API key or token once, at the door, before anything downstream |
| Rate limiting | Cap how much each client can send, protecting services from abuse and spikes |
| Routing | Send the request to the right service by path, header, or version |
| Aggregation | Fan one client call out to several services and compose the response |
| Protocol translation | Bridge REST, gRPC, WebSocket, GraphQL so clients need one style |
The theme is cross-cutting concerns. Without a gateway, every service has to check auth, enforce limits, and log consistently on its own — five services, five copies of the same fiddly, security-critical code. AWS describes Amazon API Gateway as handling exactly this: accepting and processing concurrent calls, managing access control and throttling, in one managed layer. Centralize it once and every service behind it gets to focus on its actual job.
The Trap: Don’t Turn It Into a Monolith#
The gateway’s strength is also its danger. Because everything flows through it, it’s tempting to keep adding logic — a little request rewriting here, a business rule there — until the gateway quietly becomes the most critical and most tangled component you own. Now it’s a single point of failure and a bottleneck for every team, since any change means touching the one box all traffic depends on.
Keep the gateway boring. It should handle concerns that are truly cross-cutting — auth, limits, routing — and nothing that belongs to a single service. The moment business logic leaks into the gateway, you’ve rebuilt the monolith you split up, only now it’s on the critical path for every request.
A quick test: if a change to one feature forces a change to the gateway, that logic is in the wrong place. The gateway should rarely need to know what any individual service actually does.
Enter the AI Gateway#
The gateway pattern didn’t just survive the move to AI — it became one of the hottest pieces of LLM infrastructure, under a new name: the AI gateway (or LLM gateway).
It routes across model providers, not microservices. The same front-door idea, but the backends are models — GPT, Claude, an open-weight model you host. The gateway handles fallback when a provider is down, routes cheap requests to a small model and hard ones to a big one, and gives you one endpoint instead of app code littered with provider SDKs. That’s the same “clients shouldn’t know the backends” logic the classic gateway sells.
Its limits and caching are measured differently. Rate limiting by request count is meaningless when one request can be 50 tokens or 50,000; an AI gateway limits and budgets by tokens and cost. And because model calls are slow and pricey, the gateway is the natural place to put semantic caching — serve a stored answer for an equivalent question and skip the model entirely. Add guardrails (PII filtering, prompt-injection checks) and token-level observability, and the AI gateway starts to look a lot like the control plane of an AI system — which is exactly what it’s becoming.
AI-era rule of thumb: an AI gateway is the same pattern with the units changed — route across models not services, limit by tokens and dollars not requests, and put semantic caching and guardrails at the door where every call already passes.
Quick Recap#
- Load balancer picks a server; API gateway decides and routes a request. Distribution vs control.
- The gateway centralizes cross-cutting concerns — auth, rate limiting, routing, aggregation, protocol translation.
- Keep it boring — no business logic, or you’ve rebuilt a monolith on the critical path.
- They pair up: gateway routes to a service, a load balancer behind it picks the instance.
- In AI systems, the AI gateway routes across models, limits by tokens, and caches semantically.
Conclusion#
The reason API gateway and load balancer get confused is that they share a seat in the request path — but they’re doing opposite kinds of work. The load balancer is indifferent to your request and obsessed with your servers; the gateway is indifferent to your servers and obsessed with your request. Put the gateway in when you’re tired of every service reimplementing auth and rate limiting, keep it ruthlessly free of business logic, and let a load balancer handle the “which instance” underneath. Get that division right and each layer stays simple — which is the whole point of splitting them up.
Have you seen an API gateway quietly turn into a monolith — the one box every team dreads changing? Or a load balancer asked to do a gateway’s job? Share the horror story in the comments.
- Load Balancers Explained — the neighbour it’s most confused with, and why health checks are its real job.
- Kafka vs RabbitMQ — how services behind the gateway talk to each other asynchronously.
- System Design Building Blocks — where the gateway sits in the wider decision map.
