I Never Let an Agent Call an LLM Provider Directly
· Bilal Tahseen

The failure pattern I see in agent systems that grew fast: every service imports its own provider SDK, hard-codes a model name, holds its own retry logic and API key, and fails independently when the provider has a bad minute.
The agent that summarizes tickets stops working. The one that drafts replies keeps going, but it's calling a different model so it gives different answers and fails your evals. The one that routes inquiries doesn't know either is broken, so it keeps routing to both. Nobody knows which services are affected until you search the codebase for model strings. The provider bills you for every retry. The on-call bills you for every page.
I stopped building systems like that. Every agent I ship now calls one internal interface: a gateway that knows all the providers, all the models, all the fallbacks, and all the policy. The agent asks for a capability, not a vendor. The gateway decides which model answers, retries when it should, fails over when it can, and fails loud when it can't. This is the architecture I wish someone had handed me before I learned it the hard way.

Why direct SDK calls break in production
Provider outages and regional incidents. Every provider has bad days. An API that works in one region might not work in another. If twenty services are calling the provider directly, all twenty are down. If you want to fail over to a different provider, you're deploying twenty services under an outage, and nobody tests the fallback path until it's already too late.
429s and rate limits. One service hits the rate limit and blocks. Another service is well under the limit but shares the same account, so it gets throttled too. You can't coordinate retries across services. You can't enforce a budget. You can't shed load to a secondary provider because every service makes its own decision.
Model version renames and deprecations. Providers ship dated snapshots, retire older ones on a deprecation schedule, and aliases can start pointing at a new snapshot. A hard-coded model string either breaks when the version is deprecated or silently changes behavior when the alias shifts. If the model name is a string literal across many services, you're changing every one of them when you need to migrate.
Per-tenant cost blowouts. One customer runs a prompt loop. One agent retries a 500 error fifty times. Without a central budget enforcer, the first time you know there's a problem is when the invoice arrives. By then you've paid for millions of tokens you didn't need to serve.
No single place for traces. Every service logs its own calls in its own format. When you need to debug why a specific user's agent behaved unexpectedly, you're stitching together spans from six services, each one with different trace IDs, different timestamps, and different ideas of what "logging the prompt" means. You can't replay. You can't attribute cost. You can't find out which model version actually answered.
Keys scattered across services. API keys in environment variables, in secret managers, in CI, spread across many repositories. Rotation is a multi-day project. A leaked key means every service that used it is compromised. You can't enforce key-per-service or key-per-capability, so one breach is total.
Data residency rules ignored. A regulated customer can't have their data leave the EU. One agent service calls the US endpoint. Another calls the EU one. A third doesn't know which to use, so it guesses. Nobody wrote it down. The contract says data stays in the EU, but several services are violating that, and you won't know until the audit.
The common thread is no coordination. Every service makes local decisions that should be global. A gateway is how you make them global.
The gateway: one interface, logical capabilities, not vendor models
The rule I build into every gateway: agents request a capability, not a model.
An agent doesn't ask for claude-3-5-sonnet-20241022. It asks for reasoning-large or extract-small or draft-fast. The gateway owns the mapping from capability to provider and model. When you need to migrate off a model, you change one routing table, not every service that calls it.
Here's the interface:
from pydantic import BaseModel
from typing import Literal, Optional
class GatewayRequest(BaseModel):
capability: str
messages: list[dict]
tenant_id: str
user_id: Optional[str] = None
max_tokens: int = 4096
temperature: float = 0.7
trace_parent: Optional[str] = None
class GatewayResponse(BaseModel):
content: str
model: str
provider: str
input_tokens: int
output_tokens: int
latency_ms: int
cached: bool
trace_id: str
The gateway is a FastAPI service. Every agent in the system calls it. The flow:
-
Auth and tenant budget check. Verify the request is from a known service. Look up the tenant's budget: if they've exceeded their cap for this billing period, return 429 with a clear message. Link to budget caps.
-
Policy router. The capability name maps to a list of candidate models. The policy decides which one to use based on: task type, cost tier, latency requirement, data residency rules, and tenant allowlist. A tenant in the EU only gets routed to EU endpoints. A tenant on the cheap plan only gets small models. A task marked
high-accuracygets routed to the best model in the tier, regardless of cost. -
Prompt and response cache. For deterministic, idempotent reads (e.g., temperature 0), check the cache. Key is
hash(tenant_id, capability, resolved_model, routing_policy_version, messages, max_tokens, temperature). If a hit exists and it's fresh, return it. If the request isn't safe to cache — it has side effects, temperature is nonzero and a sampled response being replayed matters to your product, or the tenant opted out — skip this. -
Retry with timeout. Try the selected provider. If it fails with a transient error, retry with exponential backoff and jitter. If it exceeds the retry budget or the error is non-retriable, move on.
-
Circuit breaker. Before each call, check the circuit breaker for that provider and model. If it's open, skip that candidate and try the fallback. After the call, record success or failure and update the breaker state.
-
Trace and cost ledger. Write every request, response, token count, latency, model used, and cost estimate to the trace store. Tag it with the tenant, user, capability, and trace parent so you can query it later. This is the source of truth for billing, debugging, and replaying traces.
-
Return with provenance. The response tells the caller which model actually answered, not which one they asked for. This matters: if the primary was down and the fallback answered, the caller needs to know, because the fallback might have different tool support or output format.
The gateway is not smart. It doesn't know what the agents are doing. It just enforces the policy, handles the transient failures, and records everything.
Retry vs fallback vs fail loud
Not every failure is retriable. Not every retriable failure should fall back. The decision tree matters.

Retry only transient errors. 429, 503, 500, timeouts, connection errors. These are "try again and it might work." Retry with bounded exponential backoff: start at 1 second, double each time, cap at 32 seconds. Add jitter so concurrent retries don't thundering-herd. Respect Retry-After headers: if the provider tells you to wait 60 seconds, wait 60 seconds, but cap how long you'll honor it against a max wait or deadline, and if Retry-After exceeds the remaining budget, raise so the caller can fall back rather than sleeping indefinitely.
Here's the retry classifier:
import time
import random
from typing import Optional
def is_transient(status_code: Optional[int], error: Optional[Exception]) -> bool:
if status_code in {429, 500, 502, 503, 504}:
return True
if error and isinstance(error, (TimeoutError, ConnectionError)):
return True
return False
def backoff_with_jitter(attempt: int, base: float = 1.0, cap: float = 32.0) -> float:
wait = min(base * (2 ** attempt), cap)
jitter = random.uniform(0, wait * 0.1)
return wait + jitter
def retry_with_budget(fn, max_attempts: int = 3, deadline_seconds: float = 60.0):
start_time = time.time()
for attempt in range(max_attempts):
try:
return fn()
except Exception as e:
status = getattr(e, "status_code", None)
if not is_transient(status, e) or attempt == max_attempts - 1:
raise
retry_after = getattr(e, "retry_after", None)
if retry_after is None and hasattr(e, "response"):
retry_after_header = e.response.headers.get("retry-after")
if retry_after_header:
retry_after = int(retry_after_header)
elapsed = time.time() - start_time
remaining = deadline_seconds - elapsed
if retry_after and retry_after > remaining:
raise
wait = retry_after if retry_after else backoff_with_jitter(attempt)
wait = min(wait, remaining)
if wait <= 0:
raise
time.sleep(wait)
In an async FastAPI gateway, use asyncio.sleep instead of time.sleep. Disable the provider SDK's own built-in retries (max_retries=0) so you don't multiply retry attempts.
Never retry 400, schema errors, or content policy violations. A 400 means the request is malformed. Retrying won't fix it. A schema error means the model didn't return valid JSON when you asked for structured output. Retrying the same prompt with the same model will likely fail again. A content policy violation means the input or output hit the provider's filter. Retrying is pointless and might get you rate-limited or banned.
Fallback only within the same capability tier and region. If the primary model for reasoning-large is down, fall back to another large reasoning model, not to a small one. The small one might not be able to handle the task. If the tenant requires EU data residency, the fallback must also be EU. If no eligible fallback exists, fail loud.
Fail loud. If all candidates fail, or the retry budget is exhausted, or no fallback is allowed, return a clear error with the reason. Don't return a half-answer from a model that wasn't suitable. Don't silently degrade to a cheaper model and hope nobody notices. Queue the request for later, degrade the feature, and alert. The caller decides what to do. The gateway's job is to tell the truth.
Circuit breaker per provider and model
A circuit breaker prevents cascading failures. When a provider or model is consistently failing, the breaker opens and stops sending traffic to it, giving it time to recover while the system tries fallbacks.
The states:
- Closed: Everything is working. All requests go through.
- Open: The error rate exceeded the threshold. No requests go through. Return an error or fall back immediately.
- Half-open: After a cooldown, allow a few probe requests. If they succeed, close the breaker. If they fail, reopen it.
The breaker is per (provider, model) pair, not per gateway instance. If you have three gateway replicas, they share the same breaker state, because independent breakers mean replica A learns a model is down, replica B doesn't, and half your traffic still hits the dead endpoint.
Store the state in Redis:
import redis
import time
from typing import Literal
CircuitState = Literal["closed", "open", "half_open"]
class CircuitBreaker:
def __init__(self, redis_client: redis.Redis, key_prefix: str = "circuit"):
self.redis = redis_client
self.key_prefix = key_prefix
self.error_threshold = 0.5
self.window_seconds = 60
self.cooldown_seconds = 30
self.half_open_probes = 3
self.min_requests = 20
def _key(self, provider: str, model: str) -> str:
return f"{self.key_prefix}:{provider}:{model}"
def state(self, provider: str, model: str) -> CircuitState:
key = self._key(provider, model)
state = self.redis.get(f"{key}:state")
if not state:
return "closed"
state = state.decode() if isinstance(state, bytes) else state
opened_at = self.redis.get(f"{key}:opened_at")
if state == "open" and opened_at:
opened_at = float(opened_at.decode() if isinstance(opened_at, bytes) else opened_at)
if time.time() - opened_at > self.cooldown_seconds:
self.redis.set(f"{key}:state", "half_open")
self.redis.delete(f"{key}:probes")
return "half_open"
return state
def allow_request(self, provider: str, model: str) -> bool:
current_state = self.state(provider, model)
if current_state == "closed":
return True
if current_state == "open":
return False
if current_state == "half_open":
key = self._key(provider, model)
probes = self.redis.incr(f"{key}:probes")
return probes <= self.half_open_probes
return False
def record_success(self, provider: str, model: str):
key = self._key(provider, model)
current_state = self.state(provider, model)
bucket = int(time.time() // self.window_seconds)
total_key = f"{key}:total:{bucket}"
self.redis.incr(total_key)
if self.redis.ttl(total_key) == -1:
self.redis.expire(total_key, self.window_seconds)
if current_state == "half_open":
probes_val = self.redis.get(f"{key}:probes")
probes = int(probes_val.decode() if isinstance(probes_val, bytes) else probes_val or 0)
if probes >= self.half_open_probes:
self.redis.delete(f"{key}:state", f"{key}:opened_at", f"{key}:probes")
def record_failure(self, provider: str, model: str):
key = self._key(provider, model)
now = time.time()
bucket = int(now // self.window_seconds)
errors_key = f"{key}:errors:{bucket}"
total_key = f"{key}:total:{bucket}"
self.redis.incr(errors_key)
if self.redis.ttl(errors_key) == -1:
self.redis.expire(errors_key, self.window_seconds)
self.redis.incr(total_key)
if self.redis.ttl(total_key) == -1:
self.redis.expire(total_key, self.window_seconds)
errors_val = self.redis.get(errors_key)
total_val = self.redis.get(total_key)
errors = int(errors_val.decode() if isinstance(errors_val, bytes) else errors_val or 0)
total = int(total_val.decode() if isinstance(total_val, bytes) else total_val or 0)
current_state = self.state(provider, model)
if current_state == "half_open":
self.redis.set(f"{key}:state", "open")
self.redis.set(f"{key}:opened_at", now)
elif total >= self.min_requests and (errors / total) > self.error_threshold:
self.redis.set(f"{key}:state", "open")
self.redis.set(f"{key}:opened_at", now)
This is a simplified version. The read-then-write sequence isn't atomic across replicas; production versions use a Lua script or Redis MULTI/pipeline for atomicity. Production breakers also track P99 latency, success rate per tenant, and per-endpoint health. The pattern is the same: shared state, rolling windows, automatic recovery.
Fallbacks are not free
The demo slides show a fallback chain: if OpenAI is down, use Anthropic, if Anthropic is down, use Google. It sounds great. Implementing it correctly is harder than it looks.
Fallback models must be evaluated. You can't add a model to the fallback chain just because it has an API. It has to pass the same golden set as the primary. If it doesn't, you're trading uptime for quality, and the first time the fallback activates, your users notice.
Prompts may need per-model variants. Not all models interpret system messages the same way. Not all models handle tool definitions identically. Some need more explicit instructions. Some do better with examples. If your primary is Anthropic and your fallback is OpenAI, and the prompt was tuned for Anthropic, the fallback might produce worse results, even if both models are technically capable.
Structured output and tool calling differ. One model supports native JSON schema enforcement. Another doesn't, so you have to parse the output yourself and retry if it's invalid. One model supports parallel tool calls. Another only does one at a time. The gateway needs to know this and adapt, or the fallback will fail in ways the primary didn't.
Tag every trace with the model that actually answered. If the primary was down and the fallback ran, your replayable traces need to record that. When you debug an unexpected answer, the trace tells you which model produced it, so you can replay it with the same model, not the one you thought was handling that capability.
Fallbacks count against the budget. If a tenant's budget is $100 and they've used $95, and the primary fails so the gateway falls back to a more expensive model, does that fallback push them over the cap? It depends on your policy. I tie fallbacks into the budget enforcement: if the fallback would exceed the cap, fail loud instead. Surprise overages are worse than downtime.
The point is not that fallbacks are bad. The point is that a fallback is a different execution path, and like all execution paths, it needs testing, monitoring, and policy.
When NOT to build this
Not every system needs a gateway.
Single-provider prototype. If you're building a demo, or an internal tool, or a proof-of-concept that calls OpenAI and nothing else, a gateway is premature. Import the SDK. Call it directly. Ship the thing. Build the gateway when you have a reason to.
Low-volume internal tool. If your agent runs a handful of times a day and downtime just means someone reruns it, there's no outage to prevent. The complexity of the gateway, the cache, the circuit breakers, and the fallback chains costs more than the problem.
Managed gateway already covers your needs. LiteLLM, Portkey, and other hosted gateways exist. If they handle your providers, your policies, and your budget enforcement, and you're okay with the vendor lock-in and the latency of another hop, use them. Don't build it yourself just because you can. I build gateways when the managed ones don't support the policy I need, or when data residency rules mean the gateway must run in my own VPC, or when the cost of the managed service is more than running my own.
Be honest about where you are. A small system that calls one model and has no SLA does not need this. A system with multiple providers, SLAs, regulated data, per-tenant budgets, and agents that must not go down when a provider has a bad hour does.
Production checklist
If you're building this, here's what I make sure works before it ships:
- The gateway enforces capability-to-model routing, not hard-coded model names in agents.
- Auth and tenant budget checks happen before any provider call.
- Retry logic respects
Retry-After, uses exponential backoff with jitter, and has a retry budget. - Non-retriable errors (400, schema failures, policy violations) fail immediately without retry.
- Circuit breakers are shared across gateway replicas via Redis or equivalent.
- Fallback chains only include models that have passed the golden set for that capability.
- Every trace records which model actually answered, not which one was requested.
- The cache is keyed on tenant_id, resolved model, and routing policy version, so one tenant can never be served another tenant's cached response.
- Per-tenant budget caps are enforced at the gateway, not after the call.
- Data residency policies are enforced: EU tenants only get EU endpoints.
- There's a runbook for "a provider is down, how do I fail over," and it's tested.
- Alerts fire when circuit breakers open, when retry budgets are exhausted, and when fallbacks activate.
And one more: the gateway's own uptime matters more than any single agent. If the gateway goes down, every agent goes down. That means the gateway needs its own redundancy, its own health checks, and its own on-call. It's not optional infrastructure. It's the infrastructure.
Close
Direct SDK calls are fine for prototypes. They don't survive contact with production. A provider has a bad hour, or a model gets renamed, or a tenant runs up a bill, or a regulation says data can't leave the region, and suddenly many services are down or leaking or costing more than you budgeted.
The gateway is the layer that absorbs those failures. It routes by capability, not vendor. It retries transient errors with a budget. It fails over to models that have been evaluated. It enforces budgets and policies in one place. It logs and traces every call so you can replay and debug. And it fails loud when it can't fulfill a request, instead of silently degrading and hoping nobody notices.
I build gateways for production agent systems where uptime, cost control, and data residency are not optional. If you're running agents that can't tolerate a provider outage, that span multiple models or providers, or that operate under budget or compliance constraints, and you need this built properly, let's talk: https://www.bilaltahseen.com/hire-ai-engineer.
Building something with AI?
I help teams ship production AI agents, retrieval systems, and document intelligence. Let's talk about yours.