The core design works on a whiteboard. Then the follow-ups start. A provider fails and the fallback provider falls over too. One team’s batch job drinks the shared token quota. Halfway through an answer the client hangs up, and the bill no longer matches the provider’s. A crafted prompt gets another tenant’s cached reply. None of this changes the picture in AI gateway design, yet all of it decides whether the gateway survives a bad day.
This lesson takes the gateway sized in the core lesson and breaks it. Its assumed load is 116 calls a second at peak, 928 open streams and about 13.2 million tokens a minute.
- AI gateway design: the request path, the sizing arithmetic and the semantic cache.
- Design a rate limiter: token buckets and approximate shared counters.
Why is a gateway sized by open streams and not CPU? Each call waits seconds for the model, so open streams equal calls per second times seconds per answer.
- Retries multiply across layers. Retry at one layer, with a budget, and check the failover target has room before sending traffic to it.
- Reserve a worst-case cost, stream through, then settle. Budgets are approximate by design, and the overshoot has a computable bound.
- Fairness needs weighted shares and concurrency caps, not only a total.
- A semantic cache is fuzzy matching, so it can be attacked. Partition it and keep it off for risky routes.
- Say what you would roll out first and which signal would make you stop.
Stress-test an AI gateway: stop retry storms, set tenant budgets and fair shares, meter streamed answers correctly, defend the cache, and answer the usual follow-ups in one line each.
Why Can Failover Make an Outage Worse?#
Provider A starts failing. Each app SDK retries 3 times and the gateway retries 3 times, so one user call becomes 9 calls to a provider that is already down.
The Google SRE book makes the same point: with 3 retries at each of four layers, “a single user action may create 64 attempts (4^3)” on the database. The Amazon Builders’ Library reports a 243x load increase when every layer retries, and says its best practice is to retry at a single point in the stack.
The fixes, in order of value:
- One retry layer. The gateway retries, the SDK does not.
- A retry budget. The SRE book suggests a server-wide cap, for example 60 retries a minute in a process. Past it, fail the request.
- Backoff with jitter. Add random delay so retries do not arrive together.
- Honest triggers. Kong’s plugin does not fail over on client errors, and you list the codes (such as
http_429) that should.
Now the second storm: the failover itself. Peak is 116 calls a second. Assume provider B’s quota allows 60 calls a second.
116 sent to B, 60 allowed
116 - 60 = 56 a second get 429
each retries 3x: up to 168 moreProvider B is now overloaded by your own gateway. So check the target’s headroom before failing over, and shed low-priority calls first, meaning refuse them early to protect the rest.
Build a failover ladder in this order (judgement):
- A smaller model at the same provider.
- A second provider.
- A self-hosted model.
- A stored answer with a notice that it is degraded.
OpenRouter bills at the model that finally answered, so a silent fallback to a dearer model changes your cost. Log which target served each call.
One more rule: never fail over after the first byte. If the stream is already flowing, the user has seen text, and a second provider would start a different answer. Fail over before the first chunk, and after it return an error event.
How Do You Share Budgets Between Tenants?#
A tenant needs two limits, because they stop different harms. A budget caps money over a period. A rate share caps tokens a minute, so one tenant cannot starve the rest. LiteLLM’s docs show the shape: max_budget and budget_duration per key, team and user, plus tpm_limit and rpm_limit.
Assume three gateway instances each let up to 20 requests run per tenant, a request’s worst-case cost is $0.05 and the tenant’s daily budget is $400. How far past the budget can the tenant spend if every instance checks a slightly stale counter?
Reveal it with arithmetic, an upper bound in the worst case.
3 instances x 20 in flight
x $0.05 = $3.00 overshoot
$3 / $400 = 0.75 percentSo an approximate counter is fine when in-flight work is small against the budget, and the bound shrinks with a lower concurrency cap. This is the same trade as a rate limiter, and a hard cap can still fail closed.
Fairness needs weights. Assume the provider gives you 8 million tokens a minute and four tenants have weights 4, 2, 1 and 1. Assume each share is then 4, 2, 1 and 1 million. If a tenant is idle, others may borrow its share and give it back when it returns (judgement). Add a per-tenant concurrency cap so one slow tenant cannot hold every connection.
How Do You Meter a Streamed Answer?#
Streaming makes the usual bill-after-the-call model fragile, because the total is known only at the end. Use reserve, stream, settle.
Reserve input tokens plus max_tokens. Assume 1,500 input tokens and a cap of 4,000 output tokens, so the reserve is 5,500. Assume the answer used 1,900, so settling refunds 3,600.
Where does the real total come from? The OpenAI API only sends usage in a final chunk when stream_options.include_usage is set, with an empty choices list, and warns the chunk may be missing if the stream is interrupted.
Claude’s stream reports cumulative usage in the message_delta event, and can also send ping events and an error after a 200 status. So a gateway should:
- Ask for usage on every streamed call, and handle the empty-choices chunk.
- Count what it forwarded, as a fallback when no final figure arrives.
- Cancel the provider call when the client disconnects, or the spend continues unseen.
- Mark partial records, so billing can tell them apart.
The latency rule from the core lesson still holds: forward chunks at once. Any check that needs the whole answer runs out of band.
Can the Cache Be Poisoned?#
A semantic cache treats nearby embeddings as the same question, so it is a fuzzy match. A 2026 paper (ICML 2026) argues that this locality conflicts with collision resistance, and reports an 86 percent hit rate for response hijacking with a crafted input that transfers across embedding models. The OWASP LLM list names this area as LLM08, Vector and Embedding Weaknesses.
Defences are judgement, ranked by effect:
- Partition the key by tenant, model and system-prompt version, so one tenant’s entries never answer another’s.
- Exact match only on routes with money, safety or identity at stake.
- Do not cache answers that contain personal data or tool results.
- Short lifetimes and an invalidate button for bad entries.
- Sample hits each week and compare them with fresh answers, so threshold drift shows up before users report it.
Contrast the default of one hosted gateway: its cache key is a SHA-256 hash of the provider, endpoint, model, auth header and full body, an exact match with no fuzziness to exploit and a lower hit rate.
What About Security, Cost and Rollout?#
- Security. Keep provider keys in the gateway and hand callers their own scoped keys, so a leak is revocable per tenant. OWASP lists LLM01 Prompt Injection and LLM10 Unbounded Consumption: cap
max_tokensand budgets per tenant, and treat logged prompts as personal data with a retention limit. - Cost. Levers in order: cache, route simple calls to a smaller model, cap output length. If 30 percent of spend moves to a model that costs a fifth (an assumption), the $6,200 a day from the core lesson falls by 0.3 x 6,200 x 0.8 = $1,488.
- Rollout. Mirror traffic to a new route in shadow mode, canary by tenant, keep configuration versioned with a kill switch, and remap aliases to migrate models. The signal to stop is a rise in errors or in wrong-hit reports.
Practice: Provider Outage Drill#
Provider A fails at peak (116 calls a second). Provider B allows 60 calls a second. Calls are 70 percent interactive and 30 percent batch. Write the gateway’s response.
Hint
Interactive calls are about 81 a second. Compare that with B’s room.
One reasonable answer
116 x 0.7 = 81 interactive
81 > 60: even B cannot take itPause batch first (35 a second), since it can wait. That still leaves 81 interactive against 60 of room, so shed the excess with a clear 503 and Retry-After, or route the simple ones to a smaller model on B. Open the circuit on A for a cooldown, and keep retrying A with a probe, not full traffic. All figures are assumptions to replace with your own.
Answer each follow-up with a trade-off, not a tool name.
| They ask | Answer in one line |
|---|---|
| Provider is down? | Fail over before first byte, check headroom |
| Retries at every layer? | One layer, a budget, jitter |
| One tenant is greedy? | Weighted shares, concurrency cap, budget |
| Stream metering? | Reserve, count, settle; handle no final usage |
| Client hangs up? | Cancel upstream, bill your count |
| Cache returns wrong answer? | Threshold, partition, exact on risky routes |
| Cache poisoning? | Tenant-partitioned key, short TTL |
| Budget store down? | Last good config, local cap, hard caps fail closed |
| Model retired? | Remap the alias, canary it |
| Cut cost 30 percent? | Cache, smaller model, output caps |
| 10x traffic? | More instances, second provider, raise tier |
| Audit and privacy? | Per-tenant logs, retention limit |
A strong answer at senior level names failure modes unprompted. A staff answer adds fairness, security and a rollout plan, and says when not to build.
Check Yourself#
1.Why should a failover check the target's headroom first?
Moving all traffic at once can exceed the second provider’s quota, so it answers 429 and the retries make both providers worse. Check headroom, shed low-priority calls and consider a smaller model.
2.A stream ends without a final usage chunk. What does the gateway charge?
Its own count of the tokens it forwarded, marked as partial. It also cancels the provider call so spend does not continue.
3.What changes if the cache is shared by two tenants?
A crafted prompt could return the other tenant’s cached answer. Partition the cache key by tenant and model, and use exact match on risky routes.
In One Picture#
Retries one layer, a budget
Failover before first byte,
check headroom
Budgets reserve, stream, settle
Fairness weights + concurrency
Cache partition, exact on riskConclusion#
The core gateway is a proxy with a few shared stores. The production answer is everything around it: retries that do not multiply, failover that checks room, budgets with a known overshoot, honest streaming accounting and a cache that cannot be talked into lying. Name the accuracy you gave up each time.
Which of these probes would you stumble on first? Tell us in the comments.
- Back to the core design: AI gateway design for the request path, sizing and semantic cache.
- The limiter underneath: Design a rate limiter and its follow-ups.
- Governance above it: AI control plane architecture.
