Choose hosted APIs, self-hosted models, routers, and async jobs deliberately.
A model that returns one prediction in a notebook is not a deployed app. This module frames what changes when real users, concurrency, and uptime enter the picture — and why that demands deliberate deployment patterns.
Frames deployment as turning a working model into a service that survives real traffic.
Why this matters: Every later choice (serving engine, gateway, scaling) only makes sense once you see what production load demands.
In a notebook you call the model once and read the result. Nothing competes for the , nothing fails halfway, and there is no "how many requests per second" to worry about. A deployed app has to answer for all of that. Deployment patterns exist precisely because the gap between "it works for me" and "it works for everyone" is where AI apps break.
Production traffic adds three things a single notebook call never tests:
The first tradeoff you will meet over and over: making each answer faster usually means keeping expensive GPUs warm, while keeping cost low usually means accepting more . You cannot escape it — you can only place yourself on it deliberately.
Team A serves a chat assistant: users wait on every token, so they keep two GPUs warm and optimize tail . The GPUs sit half-idle, but answers feel instant.
Team B runs the same model to summarize a million documents overnight. Nobody is waiting, so they pack requests into large batches for maximum per dollar and let any single job take seconds. Same weights, opposite deployment — because the load is different.
What happens to a single inference request once your model is deployed — not just 'call model, get answer'.
Write one sentence: how many requests per second do you expect at peak, and how fast must each one come back? This single number shapes every later choice.
Decide which matters more for THIS app: tail (a user is waiting) or per GPU-dollar (a batch job at 3am). You rarely get both.
Pick the one outage you most fear — a cold-start spike, a saturated queue, a bad model version — and make sure your design has an answer for it before you ship.
In production you never report "average latency" alone — you track percentiles (p50, p95, p99). A model with a great average can still have a p99 that times out for one user in a hundred, and that user is the one who complains.
Now that you know production load is the problem, this module traces the request path and names the three layers — gateway, serving engine, GPU — that every AI deployment is built from.
Traces the request path and names the three layers that every AI deployment has.
Why this matters: You cannot reason about latency, cost, or failures without knowing which layer owns each.
(Concurrency, traffic variability, and failure.) The three layers below exist to absorb exactly those three pressures — keep that mapping in mind as you read.
Almost every AI deployment, however fancy, is three layers stacked in front of the model weights.
The key insight: batching lives in the serving engine, not the gateway. The engine waits a few milliseconds to gather several requests, runs them through the GPU in one pass, and so trades a little for much higher . That single decision is the heart of AI serving.
A user reports a 4-second response. Knowing the three layers tells you exactly where to look: was the request queued at the because every replica was busy (a scaling problem), did it wait in the engine's batch window (a batching-config problem), or was the genuinely slow on a long prompt (a model/hardware problem)?
Without the layered model you guess. With it, you instrument each boundary and the 4 seconds resolves into 'queued for 3.5s' — and now you know to add capacity, not a faster GPU.
The request hits the first: it verifies the API key, checks the caller is under their rate limit, and picks a serving replica that is alive and not overloaded.
The engine groups your request with other in-flight ones and schedules them onto the GPU together, so the hardware processes many sequences per pass.
The runs the forward passes; tokens stream back up through the gateway to the client. Time spent here is most of your .
GPUs are massively parallel but have high fixed cost per kernel launch. Running one request at a time leaves most of the chip idle. Continuous batching (as in vLLM) keeps slotting new requests into the running batch as others finish, pushing several-fold higher at the cost of a small, bounded wait.
With the request path understood, this module is the central decision: which deployment pattern fits your app. Each pattern places you differently on the latency, throughput, cost, and operational-burden tradeoff.
Compares the main deployment patterns and the latency/throughput/cost tradeoffs each forces.
Why this matters: This is the decision that determines your GPU bill and whether users wait seconds or milliseconds.
Decision this forces: Choose the deployment pattern for your AI app: real-time serving, batch/offline, or serverless scale-to-zero.
Once you know the layers, deployment becomes a small set of recurring patterns. Three cover most cases.
Cutting across all three is a second axis: self-hosted versus managed. Self-hosting your own engine gives full control and lower per-token cost at scale, but you own the GPUs, patches, and pager. A managed API trades that control for someone else carrying the operational weight.
If a human is staring at a spinner, you need real-time serving and must optimize . If it is an offline job, batch wins on cost.
Steady high volume justifies always-warm GPUs. Spiky or rare traffic makes scale-to-zero attractive — you stop paying for idle time, accepting a cold start.
Before choosing self-hosted, ask honestly: can your team run a engine on GPUs at 3am? If not, a managed endpoint is the smaller, safer choice until volume forces the move.
| Option | Latency | Cost efficiency | Operational burden | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Real-time serving | Lowest — endpoint stays warm, no cold start. | Poor at low traffic — you pay for idle GPUs. | Needs autoscaling, health checks, and monitoring. | Use when a human is waiting on the response (chat, search, assistants). | High (always-on GPUs) | Medium |
| Batch / offline | Irrelevant — jobs finish minutes to hours later. | Best — large batches keep the GPU fully utilized. | Simple — a queue and a worker, no uptime SLA. | Use for bulk processing where no user is waiting (nightly summaries, embeddings). | Low per item | Low |
| Serverless / scale-to-zero | Good when warm, but cold starts add seconds. | Excellent for spiky traffic — zero idle cost. | Platform handles scaling; you tune cold-start behavior. | Use for low-volume or bursty traffic where idle cost would dominate. | Pay-per-use | Medium |
A serverless GPU endpoint must load multi-gigabyte model weights into VRAM before it can answer. That cold start can take 10–60 seconds. The fix is not 'never use serverless' — it is keeping a small warm pool, or accepting cold starts only on traffic that tolerates them.
With a pattern chosen, this module sketches the smallest real-time serving endpoint and the autoscaling rule that protects the GPU. It is the worked-to-completion ramp: read the endpoint, then predict the missing scaling logic.
Sketches the smallest runnable serving endpoint plus the autoscaling logic that protects the GPU.
Why this matters: Connecting the pattern to a concrete artifact is what turns the concept into something you can ship.
A minimal real-time deployment is two pieces. First, a endpoint: a process that loads the model once, then answers requests on a port. Second, an autoscaling rule: logic that watches load and adds or removes replicas so the is neither idle nor drowning.
The scaling signal that matters for AI serving is usually queue depth (requests waiting), not CPU. A GPU can sit at modest CPU while its request queue — and your — climbs.
Picture one GPU pod running a vLLM engine behind a . At 10 requests/second it keeps p95 under target. Traffic doubles at lunchtime; the queue grows, p95 creeps up, and the autoscaler spins a second pod — bringing latency back down without anyone paging an engineer.
That self-healing behavior is the whole point of pairing an endpoint with an autoscaling rule. The endpoint serves; the rule keeps it from falling over.
# A minimal real-time serving endpoint + the autoscaling decision. model = load_model_to_gpu() # done ONCE at startup, not per request def handle_request(prompt): return model.generate(prompt) def should_scale(queue_depth, replicas, high=20, low=5): # TODO: return "up", "down", or "hold" based on queue_depth if queue_depth > high: return "up" if queue_depth < low and replicas > 1: return "down" return "hold"
load_model_to_gpu()queue_depthhigh / lowThe endpoint loads the model once and answers requests; the autoscaler scales on queue depth, not CPU. Treat this as the smallest runnable shape — real deployments add health checks, timeouts, and metrics.
It returns "up": 25 exceeds the high-water mark of 20, so a replica is added. Queue depth is the right signal because a GPU can show low CPU while requests pile up and latency climbs — CPU would never trigger the scale-up that latency actually needs.
Load weights into the at startup, never per request — loading is the slow part, so do it once and reuse.
Accept a prompt, run inference, return tokens. This is the unit your will route to and your autoscaler will replicate.
Watch queue depth: above a high-water mark, add a replica; below a low-water mark for a while, drop one. The code below leaves that rule for you to predict.
If you let an AI assistant generate serving or autoscaling config, check these four things before trusting it — they are the ones models get wrong most often:
The final module is the WHEN-IT-BREAKS step: the production failure modes that take AI deployments down, and the checks that prove a setup is ready. It ends by deciding how much rigor your app actually needs.
Names the production failure modes and the checks that prove a deployment is actually ready.
Why this matters: Most outages come from cold starts, queue overflow, and version skew — this is where you defend against them.
Decision this forces: Choose the right level of deployment rigor: prototype, production, or optimized — sized to the app's real uptime and latency needs.
(Queue depth or latency — a GPU can be saturated while CPU stays low.) Two of the failure modes below are exactly what happens when that defense is missing or mis-tuned.
Knowing the happy path isn't enough. Three failure modes cause most AI deployment outages — recognize them and you defend against them.
A scaled-to-zero or freshly-added replica must load weights into the before it can answer. The first requests hit a 10–60 second wall. Defense: a small warm pool, or pre-warming before predicted spikes.
Traffic outruns capacity, the request queue grows unbounded, and climbs until requests time out. Defense: autoscale on queue depth (module 4), cap the queue, and shed load gracefully rather than hanging.
A redeploy swaps the model weights but a cached prompt template, tokenizer, or downstream parser still expects the old version. Outputs look plausible but are subtly wrong. Defense: pin and log the model so every response is traceable to exact weights, and roll out behind a canary.
Drive traffic at your stated peak and confirm p95 holds. If it doesn't, you found a queue-overflow risk before users did.
Scale a replica from zero and time the first response. Decide whether that delay is acceptable for this app, or whether you need a warm pool.
Confirm every response can be tied to an exact model tag, so a bad rollout can be caught and reverted fast.
| Option | Fit | Operational burden | Outage risk | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Prototype (single endpoint) | Great for a demo or to prove value fast. | Minimal — one replica, no autoscaling. | High under real load; no defense against spikes. | Use while traffic is tiny and an outage costs nothing. | Low | Low |
| Production (autoscaled + monitored) | Right when real users depend on uptime. | Needs autoscaling, version pinning, dashboards. | Low — defends against queue overflow and skew. | Use once the app is on the critical path for users. | Medium | Medium |
| Optimized (multi-region, tuned batching) | Best only after a measured cost or latency bottleneck. | High — specialist tuning and more infra to run. | Low, but added complexity is its own risk source. | Use only after metrics prove the production tier is insufficient. | High | High |
The single habit that prevents most of these failures is observability: log the model tag, queue depth, and per-percentile on every request. You cannot catch version skew or queue overflow you cannot see.
From memory, reconstruct the path: name the three layers a request passes through, then the three deployment patterns and the one tradeoff that separates them. Finally, recall the three failure modes that take AI deployments down and the defense for each.
Solo build: take one AI feature you want to ship and write its one-page deployment plan — peak load and p95 target, the chosen pattern (real-time/batch/serverless) with its tradeoff, a minimal endpoint plus an autoscaling signal, and the one failure mode you'll defend against first. NEXT rung: implement the thin slice on a managed GPU endpoint, load-test it to your target, then study autoscaling and canary rollouts to graduate from prototype to the production tier.
A data scientist runs a model in a Jupyter notebook with no issues, then deploys it to production — and latency spikes, errors appear under load, and costs balloon. Which of the following best explains why deployment is its own problem, separate from running the model?
Production traffic introduces concurrent requests, unpredictable load spikes, and uptime pressure — none of which a single-user notebook ever tests.
An inference request arrives at your deployed AI app. Tracing it through the stack: which layer is responsible for batching multiple requests together before they hit the GPU?
The serving engine sits between the gateway and the GPU and owns inference-level optimizations like request batching to maximize GPU utilization.
Your app scores millions of customer records overnight and results are needed by 6 AM — but no user is waiting in real time. Your GPU budget is tight. Which deployment pattern is the best fit?
Batch/offline serving is purpose-built for high-volume, latency-tolerant workloads — it processes records in bulk on a schedule and avoids the cost of always-on infrastructure.
Look at a minimal real-time serving config that defines a container image, a replica count, and an autoscaling rule (e.g., scale up when CPU/GPU utilization exceeds 70%, scale down when it drops below 30%). In one or two sentences, explain what the autoscaling rule is actually deciding and why both the scale-up AND scale-down thresholds matter.
Autoscaling balances two opposing risks: under-provisioning (too few replicas → slow responses) and over-provisioning (too many idle replicas → wasted cost), so both thresholds need deliberate tuning.
The three failure modes that cause most AI deployment outages are: GPU/memory exhaustion, a cascade of slow requests overwhelming the queue (request pile-up), and cold-start latency spikes. You're building an internal analytics tool used by ~10 people, with no SLA and results needed within a few minutes. What is the RIGHT level of deployment rigor?
Deployment rigor should be sized to real uptime and latency needs — a low-traffic internal tool with no SLA warrants prototype-level setup, not the overhead of full production hardening.