Route across providers with fallback, budgets, and unified interfaces.
Calling many LLM providers directly spreads provider-specific code across your app. A model gateway like LiteLLM owns that translation behind one interface. This module frames the problem before we trace a call.
What a gateway owns turns Model Gateways with LiteLLM into a usable mental model.
Why this matters: It matters because you need to know which job LiteLLM does and which job stays in your app.
When an app calls just one model provider, life is simple. The moment you add a second — say a cheaper model for drafts and a stronger one for final answers — your code fills with provider-specific clients, auth, retry logic, and response shapes that don't match.
A solves this. LiteLLM is one such gateway: it exposes a single OpenAI-style interface and translates each call into whatever the chosen provider expects. Your app sends one shape of request; the gateway handles OpenAI, Anthropic, Google, Bedrock, or a self-hosted model behind it.
gpt-4o comes back tagged with the provider that actually served it.A support tool drafts replies with a small, cheap model and escalates hard tickets to a frontier model. Without a gateway, that's two SDKs, two auth flows, and two response parsers in the same codebase.
With LiteLLM, the app calls completion(model="gpt-4o-mini", ...) for drafts and completion(model="claude-3-5-sonnet", ...) for escalations. Same call shape, same parsing — only the model name changes.
The smallest useful path a request takes through a model gateway.
Write one sentence: the gateway owns provider translation, routing, and reliability; it does not own your prompt or your business logic.
Confirm the model name you pass is one LiteLLM recognizes; check the provider docs for the exact identifier.
Send the same prompt to two model names and verify each response reports the provider that served it — that proves routing works.
The technical habit is to make the gateway's hidden choices visible: log the requested model name, the provider it resolved to, token counts, cost, and latency for every call.
Now that you know what a gateway owns, trace one request through it: request in, route, translate, call, normalize, log. Seeing the path is what lets you debug it later.
How a call flows turns Model Gateways with LiteLLM into a usable mental model.
Why this matters: It matters because understanding the request path is what lets you debug latency and errors later.
In module 1 the gateway was a black box that owns translation. Open it up. A single call passes through four stages, and each one is a place you can inspect when something goes wrong.
Because every model call goes through the same path, you measure and latency in one place instead of per provider.
You send the same prompt twice — once as model="gpt-4o" and once as model="claude-3-5-sonnet". At Resolve, the first maps to OpenAI, the second to Anthropic.
At Translate the two requests diverge into different native schemas, yet at Normalize both come back as the same response object your code reads identically. The only thing your app saw differ was the model string.
from litellm import completion resp = completion( model="claude-3-5-sonnet", messages=[{"role": "user", "content": "Summarize this ticket."}], ) print(resp.choices[0].message.content) print(resp.model) # which provider model actually served it
completion(...)messages=[...]resp.modelThis is the smallest runnable shape: one call, one response object. Note the response is OpenAI-shaped even though Anthropic served it — that normalization is the gateway's whole value.
It fails at Resolve. The gateway can't map an unknown model name to a provider, so it errors before any translation or network call happens — which is exactly where you want it to fail: early and cheap.
Pin down where the app stops and the gateway starts: the app builds the request body; the gateway resolves the model and forwards it.
Read the gateway's log line for one call: requested model, resolved provider, status, latency, tokens.
Send a request for an unknown model name and confirm it fails at Resolve with a clear error, not silently downstream.
A gateway adds a hop and an operator to maintain. This module forces the choice: when does LiteLLM earn its keep, and when is a single provider SDK simpler?
When to reach for it turns Model Gateways with LiteLLM into a usable mental model.
Why this matters: It matters because a gateway adds a hop and an operator burden you only want when it pays off.
Decision this forces: Choose how to call your models: direct provider SDK, LiteLLM SDK in-process, or a LiteLLM proxy server.
A model gateway is not free. It adds a network hop, a config to maintain, and one more thing that can go down. The honest question is when that cost buys you enough.
Write the user-visible need in one line: e.g. "answers must keep flowing even if one provider is down."
Count how many providers and reliability guarantees the requirement actually implies — not how many you might want someday.
Pick the lightest option that satisfies the requirement and can be tested; add the gateway only when the evidence demands it.
| Option | Fit | Operational burden | Risk | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Direct provider SDK | Best for a single provider, simple needs. | Fewest moving parts. | No fallback; provider outage stops you. | Use when you commit to one provider and need no failover. | Low | Low |
| LiteLLM SDK (in-process) | Best for multi-provider apps wanting one call shape. | A library, no extra service to run. | No central control across services. | Use when one app needs provider flexibility and fallbacks. | Low | Medium |
| LiteLLM proxy server | Best when many services share routing, keys, and cost rules. | A service to deploy and monitor. | A single choke point if not made reliable. | Use when central cost tracking and key management matter across teams. | Medium | High |
A proxy gateway adds one network hop to every call. Usually it is small next to model latency, but it is real — and it concentrates risk: if the proxy is down, every model call is down.
You decided a gateway fits. Now wire the smallest version that proves it: a router config with two models and a fallback, plus one call. Earlier modules showed worked examples; here a key line is left for you to complete.
Wire it up turns Model Gateways with LiteLLM into a usable mental model.
Why this matters: It matters because the smallest working slice is what you extend into a real deployment.
The answer is Resolve. It reads the router's model list to map a model name to a provider and credentials. The config below is exactly what Resolve consumes.
A router config is just a list: each entry names a model and the provider that serves it. Add a fallback and the router retries a backup model when the primary fails.
from litellm import Router router = Router( model_list=[ {"model_name": "primary", "litellm_params": {"model": "gpt-4o"}}, {"model_name": "backup", "litellm_params": {"model": "claude-3-5-sonnet"}}, ], # TODO: make 'primary' fall back to 'backup' on failure fallbacks=[...], ) resp = router.completion( model="primary", messages=[{"role": "user", "content": "Draft a reply."}], )
Router(model_list=[...])litellm_paramsfallbacks=[...]The router knows two model names. The missing piece is the fallback wiring: when 'primary' errors, the router should automatically retry 'backup' before giving up.
fallbacks=[{"primary": ["backup"]}] — a list of maps from a failing model name to an ordered list of backups to try. When a call to 'primary' raises, the router transparently re-issues it as 'backup'.
If you let an AI assistant generate this router config, verify these before trusting it:
A gateway concentrates every model call, so its failure modes hit everything at once. This late module covers how it breaks under load and the guards — fallbacks, rate limits, cost caps, versioning — that keep it reliable.
Run it in production turns Model Gateways with LiteLLM into a usable mental model.
Why this matters: It matters because a gateway concentrates traffic, so its failure modes hit every model call at once.
Decision this forces: Choose the right level of operational rigor for your gateway: bare router, router with fallbacks and limits, or a full proxy with budgets and version pinning.
It is the single choke point: every model call flows through the gateway, so when it degrades, all of them degrade together. That is exactly why the guards below matter.
Knowing the happy path isn't enough. Because the gateway sits in front of every call, these are the failures that bite at scale — and the guard for each.
Traffic spikes past a provider's quota and calls start returning 429s. Guard: per-model limits in the router plus fallbacks to a second provider.
A misconfigured primary fails constantly and the backup quietly absorbs it all — your bill and climb while dashboards look green. Guard: alert on fallback rate, not just error rate.
A loop or a leaked key drives spend through the roof. Guard: per-key budget caps enforced at the gateway, which is the one place that sees every call.
A provider updates the model behind an alias and behavior shifts with no code change. Guard: pin exact model in the router and log the served model on every call.
Replace floating aliases with exact model ids so behavior can't drift under you.
Add per-model throughput limits and per-key budget caps at the gateway before traffic grows.
Treat a rising fallback rate as a primary-provider incident, not a success — it hides cost and latency growth.
| Option | Fit | Operational burden | Risk | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Bare router | Fine for a prototype or internal tool. | Almost none. | No caps; cost and outages can surprise you. | Use while traffic is low and failures are cheap. | Low | Low |
| Router + fallbacks + limits | Best when users depend on the result. | Needs config, alerts, and an owner. | Lower once fallback rate is monitored. | Use when uptime and predictable cost matter. | Medium | Medium |
| Full proxy + budgets + version pinning | Best for many teams sharing models. | A monitored service with key management. | Lowest, if the proxy itself is made reliable. | Use when central cost control and governance are required. | High | High |
From memory, reconstruct the whole path: name what a model gateway owns versus what your app owns, list the four stages one request passes through, state the condition that justifies a gateway over a direct SDK, and recall two production failure modes plus the guard that prevents each.
Build a one-page LiteLLM plan: a router config with a primary model, a fallback, and a per-key budget cap; the single call your app makes; and the test that forces the primary to fail so you can confirm the backup serves it. Then point yourself at the next rung: add a provider-specific rate limit and a dashboard alert on fallback rate.
Your app calls three LLM providers directly. You add a fourth. What maintenance problem does this create that a model gateway solves?
A gateway centralises provider-specific auth, SDKs, and response normalisation so your app code stays unchanged when you add or swap providers.
Trace a LiteLLM request: your app calls litellm.completion(model='openai/gpt-4o', ...). Which step is where most latency and errors actually appear?
The provider network hop is where timeouts, rate-limit errors, and model-side failures concentrate — everything else in the path is local and fast.
A solo developer is building a weekend prototype that calls one provider. Which option is the best fit, and why?
The gateway pays off when you have multiple providers, need fallbacks, or must centralise keys — a single-provider prototype has none of those pressures, so the direct SDK is the smallest option that meets the requirement.
Write the two LiteLLM router config fields that together map a model alias to a specific provider and authenticate the call. (Name the fields; you don't need to write full YAML.)
model_name is the alias the router matches on; litellm_params.model tells LiteLLM which provider+model to call, and litellm_params.api_key supplies the credential — together they form one complete routing entry.
In production, your gateway starts dropping requests during a traffic spike. Which pairing of failure mode → guard is correct?
Fallbacks are the direct guard against rate-limit (429) errors — when the primary model is throttled, the router retries on a backup provider instead of surfacing the error to the caller.