Serve an open model with batching, streaming, and production checks.
Stand up the vLLM process with `vllm serve`, choosing the right installation path (pip vs. Docker) and setting the flags that govern host, port, API key, and tensor-parallel degree. You'll trace how a single CLI invocation maps to the OpenAI-compatible endpoint your clients will hit.
Stand up a vLLM inference server with vllm serve, choosing pip or Docker and setting the flags that govern host, port, auth, and tensor parallelism.
Why this matters: This is the foundation of every subsequent module — nothing else (model loading, GPU tuning, client integration) works until the server process is correctly configured and accepting connections.
Decision this forces: pip install vs. Docker image — which installation path fits your deployment target?
You have a model checkpoint and a GPU. How does one CLI command turn that into a production endpoint your clients can hit with standard HTTP?
vllm serve HTTP server — all before the first request arrives.
Every flag maps directly to one subsystem. Host/port govern the socket. --api-key gates the HTTP layer. --tensor-parallel-size
The non-obvious part: the process is single-entry but multi-process under the hood. Tensor-parallel workers are forked at startup. A misconfigured flag can silently produce a server that binds but never accepts model traffic.
# pip path pip install vllm # Docker path (swap image tag to pin a release) docker run --gpus all --rm \ -p 8000:8000 \ vllm/vllm-openai:latest \ --model meta-llama/Llama-3-8B-Instruct # Verify the process accepted connections curl http://localhost:8000/health
--gpus all-p 8000:8000vllm/vllm-openai:latest--model/healthTwo launch paths, one verification step — the /health endpoint returns 200 only after weights are loaded and the scheduler is ready, so it's the canonical readiness probe.
Note that --model is passed as a flag inside the Docker command, not as a docker run argument — it goes after the image name so vLLM's entrypoint receives it.
Connection refused until the socket binds (a few seconds), then 503 Service Unavailable while weights load, then 200 OK once the scheduler is ready. The 503 window can last minutes for large models — your readiness probe must retry, not fail fast.
vllm serve meta-llama/Llama-3-8B-Instruct \ --host 0.0.0.0 \ --port 9000 \ --api-key "sk-internal-prod" \ --tensor-parallel-size 2 # Confirm auth is enforced curl -s http://localhost:9000/v1/models | head -c 80 curl -s -H "Authorization: Bearer sk-internal-prod" \ http://localhost:9000/v1/models
--host 0.0.0.0--api-key--tensor-parallel-size 2The first curl returns a 401 with no header; the second returns the model list — confirming the Bearer gate is active.
--tensor-parallel-size 2 forks a second GPU worker at startup; if only one GPU is visible, the process exits immediately with a CUDA device count mismatch error.
vLLM forks 4 worker processes, each tries to claim a CUDA device by rank. Workers with rank ≥ 2 get a CUDA error (invalid device ordinal) and the main process aborts before the HTTP socket ever binds. You'll see 'RuntimeError: CUDA error: invalid device ordinal' in the log — the server never reaches the weight-loading phase.
Your server is up, auth is enforced, and /health returns 200. But the default launch makes no promises about GPU memory headroom, context length, or throughput under load.
Consider: you're serving Llama-3-8B on a single A10G (24 GB). Model weights consume ~16 GB in bf16. That leaves ~8 GB for KV cache. A single long-context request can exhaust it and trigger OOM mid-generation.
— the levers that determine whether your server survives real traffic or OOMs on the third request.
| Option | CUDA driver coupling | Reproducibility | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| pip install vllm | Tight — vLLM wheel must match your installed CUDA version exactly | Fragile across machines unless you pin the full requirements file | Bare-metal or VM where you control the CUDA/driver stack and need fast iteration or custom kernel patches. | No image pull overhead; driver mismatches are your problem | Low setup; high env-management burden over time |
| Docker (vllm/vllm-openai) | Bundled CUDA runtime — host driver only needs to meet the minimum version floor | Pinned image digest guarantees identical runtime across nodes | Kubernetes, cloud VMs, or any environment where image immutability and CUDA bundling matter more than iteration speed. | Image storage + registry egress; driver must still satisfy minimum CUDA version | Higher initial pull (~20 GB image); near-zero env debugging |
Each flag targets exactly one layer; understanding the mapping lets you debug startup failures without guessing.
If port 8000 is taken, vLLM logs 'OSError: [Errno 98] Address already in use' and exits. A process manager like systemd may restart it in a loop, making it look like a model-loading crash. Always check ss -tlnp | grep 8000 first.
A pip-installed vLLM wheel compiled for CUDA 12.1 on a host running CUDA 11.8 raises 'CUDA error: no kernel image is available for execution on the device' at the first CUDA call. The process starts and the socket binds, so health checks may briefly return 200 before the worker crashes.
Setting --tensor-parallel-size to a value that doesn't evenly divide the model's attention-head count raises a hard assertion during weight sharding. This happens before any request is served. The error message names the head count, so it's diagnosable, but it only surfaces at startup.
echo the key value before passing it.When an LLM generates your vllm serve command, check these specific things before trusting it:
--tensor-parallel-size, not --tp or --tensor_parallel_size. Wrong spellings are silently ignored, leaving defaults in place.--host and --port are present and explicit. Generated commands often omit them, leaving defaults that may conflict with other services.curl /health and curl /v1/models (with the key) after startup. A 200 from /health but 401 from /v1/models confirms auth is wired. A 200 from both confirms the full stack.nvidia-smi or CUDA_VISIBLE_DEVICES. Don't assume the node's total GPU count.Load an open-weight model (e.g., `meta-llama/Llama-3-8B-Instruct`) and tune `--gpu-memory-utilization`, `--max-model-len`, quantization (`--quantization awq`), and tensor parallelism so weights, KV cache, and activations fit within your VRAM budget. You'll reason through the memory partition and identify where the budget breaks.
How vLLM partitions GPU memory across weights, KV cache, and activations — and how quantization and tensor parallelism reshape that budget.
Why this matters: Getting the memory split right is the difference between a server that handles real concurrency and one that OOMs on the first long request.
Decision this forces: Full-precision vs. quantized weights — what quality/throughput tradeoff is acceptable for your workload?
vllm serve with --tensor-parallel-size 2, which process owns the weight shards — the CLI, the worker processes, or the HTTP server? And what flag controls how much GPU memory vLLM is allowed to claim?Answer: vllm serve HTTP frontend is a thin router. The --gpu-memory-utilization flag (default 0.90) caps GPU allocation. Module 1 set those flags — this module explains what happens inside the VRAM budget they carve out.
pool (dynamic, grows with sequences), and activation scratch (transient, proportional to batch size × hidden dim).
Weight size is the dominant fixed cost. Llama-3-8B in BF16 occupies ~16 GB (8B params × 2 bytes). Remaining VRAM after weights and activations becomes the KV cache pool.
KV cache per token per layer = 2 × num_heads × head_dim × bytes_per_element. For Llama-3-8B (32 layers, 8 KV heads, head_dim 128, BF16): 2 × 8 × 128 × 2 = 4,096 bytes per token per layer, or ~131 KB per token total.
On a 24 GB GPU with 16 GB for weights and ~1 GB for activations, you have ~7 GB for KV cache. That supports roughly 53 K concurrent tokens before requests queue.
You're serving Llama-3-8B-Instruct on a single A10G (24 GB). BF16 weights consume 16 GB, leaving only ~7 GB for KV cache — fine for short sessions, but a 32 K context workload needs far more. You need to cut weight memory.
AWQ (Activation-aware Weight Quantization) at W4A16 drops weights to ~4 GB, freeing ~12 GB for KV cache. It preserves quality better than naive 4-bit because it protects salient weights identified via calibration data.
GPTQ at 4-bit lands in the same weight-size ballpark but uses a different calibration approach (layer-wise OBQ). In practice, AWQ tends to outperform GPTQ on instruction-following tasks at the same bit-width, but GPTQ has broader pre-quantized model availability on Hugging Face.
--kv-cache-dtype fp8. Conflating weight quantization with KV cache quantization is a common sizing mistake.vllm serve meta-llama/Llama-3-8B-Instruct \ --quantization awq \ --gpu-memory-utilization 0.90 \ --max-model-len 32768 \ --tensor-parallel-size 2 \ --dtype auto \ --port 8000
--quantization awq--gpu-memory-utilization 0.90--max-model-len 32768--tensor-parallel-size 2--dtype autoThis command loads an AWQ-quantized Llama-3-8B-Instruct across two GPUs, capping VRAM at 90% per device and extending the context window to 32 K tokens.
--dtype auto lets vLLM pick the compute dtype from the checkpoint's metadata — critical for AWQ, which stores dequantization scales alongside weights.
vLLM fails at worker initialization, before any weights load. It tries to spawn 2 worker processes but finds only 1 CUDA device, raising: 'ValueError: ... tensor_parallel_size (2) > number of GPUs (1)'. The process exits immediately — no partial load occurs.
# Scenario: two A100 40 GB GPUs, Llama-3-8B-Instruct AWQ checkpoint. # Target: 64 K context, maximise KV cache, leave 5% VRAM headroom. # Complete the two missing flags. vllm serve meta-llama/Llama-3-8B-Instruct \ --quantization awq \ --tensor-parallel-size ___ \ --max-model-len ___ \ --gpu-memory-utilization 0.95 \ --dtype auto
--tensor-parallel-size ___--max-model-len ___Stop — fill in the two blanks before revealing. Hints: (1) how many GPUs are available? (2) what integer represents 64 K tokens?
Changed lines: --tensor-parallel-size 2 (two GPUs available) and --max-model-len 65536 (64 K = 65536 tokens). Memory estimate per GPU: 40 GB × 0.95 = 38 GB claimed; AWQ shard ≈ 2 GB (4 GB total ÷ 2); activations ≈ 1 GB → ~35 GB per GPU for KV cache. That is a large pool — enough to hold tens of thousands of concurrent tokens per device. Note: --gpu-memory-utilization 0.95 is aggressive; monitor for OOM under bursty long-completion traffic.
Llama-3-8B-Instruct on a single 24 GB GPU (BF16). Weights consume ~16 GB; activations ~1 GB. Slide to see how much VRAM remains for the KV cache pool.
| Option | Weight VRAM (8B model) | Throughput impact | Quality vs. BF16 | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| BF16 (full precision) | ~16 GB | Baseline; no overhead | Reference — no degradation | When VRAM is ample, quality is non-negotiable, or you need a clean baseline before quantizing. | Highest VRAM (~16 GB for 8B) | Low — no extra flags |
| AWQ W4A16 | ~4–5 GB | Higher token/s due to smaller weight footprint; dequant overhead is minor on modern GPUs | Minimal degradation on most benchmarks; salient-weight protection helps | When VRAM is the binding constraint and you need the best quality-per-bit at 4-bit weight precision. | ~4–5 GB for 8B | Medium — requires a pre-quantized AWQ checkpoint |
| GPTQ 4-bit | ~4–5 GB | Slightly lower than AWQ on some kernels; depends on GPU arch | Slightly more degradation than AWQ on instruction tasks; task-dependent | When a pre-quantized GPTQ checkpoint already exists for your model and AWQ is unavailable. | ~4–5 GB for 8B | Medium — requires a pre-quantized GPTQ checkpoint |
--gpu-memory-utilization 0.95 and startup succeeds, but the first long batch triggers torch.cuda.OutOfMemoryError. Cause: vLLM pre-allocates KV cache blocks at startup; if profiling underestimates peak activation memory, the forward pass exceeds budget. Fix: lower --gpu-memory-utilization or reduce --max-model-len.--tensor-parallel-size 3 raises a shape mismatch error because 32 ÷ 3 is not an integer. Use a power-of-two or clean divisor of the model's head count.--quantization awq against a GPTQ checkpoint may not error — vLLM loads weights with the wrong kernel, producing degraded outputs with no warning. Always verify the checkpoint's quantization_config.quant_type in config.json before setting the flag.--quantization matches config.json:quantization_config.quant_type in the checkpoint directory.--tensor-parallel-size divides num_attention_heads evenly (check config.json:num_attention_heads).nvidia-smi after startup. Each GPU should show roughly equal memory; lopsided split signals TP misconfiguration.Trace how PagedAttention enables continuous batching — new requests join mid-flight without waiting for the current batch to drain — and tune `--max-num-seqs`, `--max-num-batched-tokens`, and scheduler policy to hit your throughput target. You'll revisit the KV-cache budget from Module 2 and see how batch size and context length compete for the same pages.
How PagedAttention enables continuous batching and how to tune --max-num-seqs, --max-num-batched-tokens, and scheduler policy to hit a throughput or latency target.
Why this matters: Getting these settings wrong is the most common reason a vLLM deployment hits its SLA in load tests but fails in production under variable request arrival rates.
Answer: activations and the compete for post-weight VRAM. The flag is --gpu-memory-utilization, which Module 2 used to carve out the KV-cache budget.
That budget is the hard ceiling this module works inside. Continuous batching and scheduler tuning are both strategies for spending that budget as efficiently as possible. They must never exceed it. They must also never leave it idle.
Static batching holds all in-flight requests until the slowest sequence finishes. Then it flushes the batch and admits new work. Under variable arrival rates, GPU utilization collapses between batches.
— enabled by 's block-granular KV-cache allocation — lets the scheduler insert a new sequence into the next decode step the moment a KV-cache page is free. There is no drain-and-refill cycle.
The non-obvious cost is that a long-running sequence holds its pages for its entire lifetime. So a burst of long-context requests can starve short ones. This can happen even when the GPU has spare compute. Batch size and context length compete for the same page pool.
The two knobs that govern this tradeoff are --max-num-seqs (concurrent sequence cap) and --max-num-batched-tokens (token budget per scheduler step). Together they bound both memory pressure and per-step compute.
You're serving a coding assistant. 80% of requests are short completions (≤256 tokens prompt + output); 20% are long document-refactor tasks (≤8 k tokens). Your SLA is P99 TTFT < 800 ms.
With --max-num-seqs 128 and --max-num-batched-tokens 32768, the 20% long tasks each hold ~8 k pages for seconds. Short completions queue behind them. P99 TTFT blows past 800 ms even though GPU utilization looks healthy at 85%.
The fix is to tighten --max-num-seqs to 32 and set --max-num-batched-tokens to 16384. Long tasks still run, but they can't monopolize the page pool. Short completions get scheduled within 1–2 decode steps of arrival.
The signal that confirms the fix worked: queue depth drops from a steady 40–60 pending requests to single digits, and GPU utilization stays above 70%. If queue depth is low but GPU utilization is also low, the bottleneck shifted to the network or client — not the scheduler.
vllm serve meta-llama/Llama-3-8B-Instruct \ --gpu-memory-utilization 0.90 \ --max-model-len 8192 \ --max-num-seqs 128 \ --max-num-batched-tokens 65536 \ --port 8000
--max-num-seqs 128--max-num-batched-tokens 65536--max-model-len 8192This launch looks reasonable on paper — high concurrency, large token budget — but it's a trap for mixed-length workloads.
Predict: with 128 concurrent sequences each potentially holding 8 k context, what happens to the KV-cache page pool, and which metric surfaces the problem first?
The KV-cache page pool exhausts before 128 sequences can all hold 8 k tokens. vLLM begins preempting (swapping or aborting) lower-priority sequences. The operator sees 'KV cache is full' warnings in logs, rising queue depth in Prometheus, and P99 TTFT spiking — even though GPU compute utilization may still read high.
vllm serve meta-llama/Llama-3-8B-Instruct \ --gpu-memory-utilization 0.90 \ --max-model-len 8192 \ --max-num-seqs 32 \ --max-num-batched-tokens 16384 \ --scheduler-policy fcfs \ --port 8000
--max-num-seqs 32--max-num-batched-tokens 16384--scheduler-policy fcfsDelta from Stage 1: --max-num-seqs drops from 128 → 32; --max-num-batched-tokens drops from 65536 → 16384; --scheduler-policy fcfs is made explicit.
At 32 sequences × 8192 max tokens, worst-case KV demand is 262 k tokens — well within the page pool for an 80 GB A100 at 0.90 utilization after Llama-3-8B weights (~16 GB).
Change --scheduler-policy fcfs to --scheduler-policy priority. Then tag short requests with a higher priority value in the API call (the 'priority' field in the OpenAI-compatible request body). The changed line: --scheduler-policy priority. Note: fcfs gives no latency guarantee to short sequences; priority scheduling requires clients to set the field, so verify your client sends it.
Slide to see how increasing concurrent sequences shifts the throughput/latency tradeoff. Each stop reflects a real operating regime.
Three failure patterns, each with a distinct fingerprint:
"Running: X reqs, Waiting: Y reqs, GPU KV cache usage: 100.0%". vLLM preempts sequences via swap or recompute. Fix: lower --max-num-seqs or --max-model-len.vllm_scheduler_running_requests in is pegged at --max-num-seqs while vllm_num_waiting_requests grows monotonically.--max-num-seqs × avg_prompt_tokens fits inside the KV-cache page budget (compute: available VRAM after weights ÷ bytes-per-token-per-layer).--max-num-batched-tokens ≥ --max-num-seqs — if batched-tokens < seqs, the scheduler can never fill even one token per sequence per step.vllm_num_waiting_requests; it should not grow unboundedly.fcfs or priority) — a typo silently falls back to the default.With the scheduler tuned, each sequence in the batch generates tokens step by step. The engine does not wait for a sequence to finish before producing output. That per-step generation is exactly what makes token streaming possible.
Module 4 wires a Python client to the /v1/chat/completions endpoint with stream=True, consuming each decode step as a delta the moment the scheduler emits it. This turns the batch machinery you just tuned into a real-time token stream.
Enable `stream=True` on the `/v1/chat/completions` endpoint and consume the Server-Sent Events (SSE) delta stream in a Python client, handling the `[DONE]` sentinel and partial-chunk edge cases. You'll also see how streaming interacts with the batch scheduler — a streaming request holds a KV-cache slot for its full generation length, affecting the concurrency math from Module 3.
How to consume vLLM's SSE token stream correctly in Python, and why streaming requests hold KV-cache slots longer than non-streaming ones.
Why this matters: Streaming is the default UX for interactive LLM apps, but it silently degrades concurrency — understanding the slot-cost tradeoff lets you tune --max-num-seqs and client timeouts to avoid throughput collapse.
Decision this forces: Streaming vs. non-streaming — when does the latency benefit of streaming outweigh the concurrency cost of holding a KV slot?
Answer: a slot. Specifically, paged blocks for that sequence's key/value tensors. reduces fragmentation. But it cannot share a live sequence's blocks with another sequence. Streaming makes this constraint worse.
When you set stream=True, vLLM emits each token via . The sequence's KV-cache blocks stay allocated until the final token is sent. Not until the client acknowledges receipt.
A non-streaming request of the same length holds the slot for the same decode time. The slot releases the moment generation finishes. Network latency to the client doesn't matter. With streaming, slow clients stretch the hold time. Effective concurrency drops below what Module 3's --max-num-seqs math predicts.
Quantifying the cost: your scheduler allows N concurrent sequences. Each streaming response averages T seconds of wall-clock hold time. Your effective throughput ceiling is N/T requests per second. This matches the non-streaming formula. But T is now network-inflated.
--timeout-keep-alive and client-side read timeouts defensively.| Option | TTFT / perceived latency | Concurrency impact | Client complexity | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| stream=True (SSE) | First token visible immediately; perceived latency drops dramatically | Slot held for full wall-clock duration including network; reduces effective N | Must handle partial JSON chunks, [DONE] sentinel, and connection drops | Interactive UIs, chat interfaces, or any path where the user perceives first-token latency — streaming cuts perceived wait even when total generation time is identical. | Higher slot hold time per request; effective concurrency = N/T where T includes network RTT. | Higher — client must parse SSE deltas, handle [DONE], and tolerate partial chunks. |
| stream=False (batch) | Full response arrives at once; user waits for entire generation | Slot freed as soon as decode finishes; concurrency math holds cleanly | Standard JSON parse; no delta assembly or sentinel handling | Batch pipelines, offline processing, or server-to-server calls where the caller blocks on the full response and perceived latency is irrelevant. | Slot released immediately on generation end; network latency does not inflate hold time. | Lower — single JSON response, no streaming parser needed. |
frame from vLLM's /v1/chat/completions endpoint is a data: <JSON>\n\n line. The JSON carries choices[0].delta.content with the new token text, or an empty string on the first frame (role announcement). The stream ends with the literal frame data: [DONE]\n\n — this is not valid JSON and must be caught before parsing.
Pass stream_options={"include_usage": true} to receive a final data frame (before [DONE]) containing the usage object with prompt and completion token counts. Without this flag, streaming responses return no usage stats — a common oversight when you need to track token spend.
include_usage: true — appends a usage frame before [DONE]; the only way to get token counts in a stream.chunk_object: true — (vLLM-specific) forces each chunk to include the full object field; useful for strict OpenAI-compat validation.data: frame mid-JSON. Your reader must buffer incomplete lines and only parse on the double-newline boundary — never parse on every newline.import json, requests resp = requests.post( "http://localhost:8000/v1/chat/completions", json={"model": "meta-llama/Llama-3-8B-Instruct", "messages": [{"role": "user", "content": "Explain SSE."}], "stream": True}, stream=True, ) for line in resp.iter_lines(): data = json.loads(line.removeprefix("data: ")) # BUG print(data["choices"][0]["delta"].get("content", ""), end="")
stream=True (requests kwarg)resp.iter_lines()removeprefix("data: ")This loop looks reasonable but has two fatal flaws — predict both before reading on.
Bug 1: iter_lines() splits on every \n, so a partial chunk (frame split mid-JSON) reaches json.loads() as an incomplete string → JSONDecodeError. Bug 2: the [DONE] frame passes removeprefix and then json.loads('[DONE]') raises JSONDecodeError immediately — [DONE] is not JSON. Both bugs crash the loop before the full response is assembled.
import json, requests def stream_chat(prompt: str) -> str: resp = requests.post( "http://localhost:8000/v1/chat/completions", json={"model": "meta-llama/Llama-3-8B-Instruct", "messages": [{"role": "user", "content": prompt}], "stream": True, "stream_options": {"include_usage": True}}, stream=True, timeout=30, ) buf, done_seen, full_text = "", False, "" for raw in resp.iter_content(chunk_size=None): buf += raw.decode() while "\n\n" in buf: frame, buf = buf.split("\n\n", 1) payload = frame.removeprefix("data: ").strip() if payload == "[DONE]": done_seen = True; break # TODO: parse payload and accumulate delta content if not done_seen: raise RuntimeError("Stream ended without [DONE] — response truncated") return full_text
resp.iter_content(chunk_size=None)buf.split("\n\n", 1)stream_options: {"include_usage": True}timeout=30This version buffers on the double-newline boundary and guards against a missing [DONE] sentinel. The TODO is the crux of this module — fill it in before revealing the answer.
Replace the TODO with:
chunk = json.loads(payload)
delta = chunk["choices"][0]["delta"].get("content", "") if chunk.get("choices") else ""
full_text += delta
Changed lines vs Stage 1: (a) json.loads on payload (not raw line) — safe because [DONE] is already caught above; (b) chunk.get("choices") guard handles the usage-only frame (which has no 'choices' key) without crashing; (c) .get("content", "") handles the role-announcement frame where delta has no content.
Symptom: json.JSONDecodeError: Expecting value: line 1 column 1 — your loop called json.loads() on a line that was the tail of a split frame. Fix: buffer on \n\n, not \n.
Symptom: The assembled text is truncated with no error raised — the loop exits cleanly because the TCP connection closed. Guard with a done_seen flag; if the loop exits without it, treat the response as incomplete and retry or surface an error.
Symptom: GPU utilization drops while active sequence count stays at --max-num-seqs — the scheduler is blocked on slots held by clients that stopped reading. Observable via vLLM's Prometheus metric vllm:num_running_seqs staying pinned while vllm:gpu_cache_usage_perc is high. Fix: enforce a server-side --timeout-keep-alive and a client read timeout.
Assume --max-num-seqs = 64. Drag to see how network-inflated hold time T shrinks effective request throughput (64 / T req/s).
Wire vLLM's `/health` and `/metrics` (Prometheus) endpoints into a load balancer and alerting stack, then define the four signals that matter: TTFT, TPOT, queue depth, and GPU memory pressure. You'll configure readiness vs. liveness probes correctly — a common misconfiguration that causes premature traffic routing during model load.
Wire vLLM's health and metrics endpoints into Kubernetes probes and a Prometheus alerting stack, and learn which four signals distinguish queue pressure from GPU memory saturation.
Why this matters: Misconfigured probes are the most common cause of traffic routing to cold replicas; correct alerting on the four key signals is what separates a production deployment from a demo.
is wall-clock time from request receipt to first token output. is the per-token decode interval after that. Queue pressure inflates TTFT first — requests wait before prefill starts — while TPOT stays stable until GPU memory saturates.
This module wires both signals — plus queue depth and GPU memory pressure — into and a load balancer. It forces the probe-configuration decision that determines whether traffic reaches a cold replica.
/metrics endpoint exposes dozens of gauges, but four drive production alerting decisions.
Queue depth and GPU memory pressure are causal signals — they tell you why TTFT or TPOT is bad. Alert on latency for SLA breach; alert on the causal signals for capacity planning.
Click a failure scenario to see which signals spike. Points closer together share the same root cause. Labels are metric names; queries are failure events.
vLLM exposes two endpoints: /health (liveness — is the process alive?) and /health/ready (readiness — is the model fully loaded?). Conflating them is the most common probe misconfiguration in vLLM deployments.
The failure mode: if readiness probes point at /health instead of /health/ready, the load balancer routes traffic to replicas still loading weights. Every request during that window gets a 503 or hangs.
The inverse mistake is equally damaging: a liveness probe with too short an initialDelaySeconds kills a healthy replica mid-load. This triggers a restart loop that never lets the model finish loading.
/health with generous initialDelaySeconds (≥ model-load time + 30 s buffer); readiness → /health/ready with tighter period once warm. Never swap them.livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 300 # ≥ worst-case model load time
periodSeconds: 30
failureThreshold: 3 # tolerates transient GPU stalls
readinessProbe:
httpGet:
path: /health/ready
port: 8000
initialDelaySeconds: 60 # first check after partial load
periodSeconds: 10
failureThreshold: 6 # patient during slow weight shardinginitialDelaySeconds: 300failureThreshold: 3path: /health/readyperiodSeconds: 10This fragment shows the correct endpoint split and the delay/threshold values that prevent both premature routing and restart loops.
The liveness failureThreshold: 3 at 30 s period gives 90 s of tolerance — enough for a large prefill stall but not so long that a genuinely dead process lingers.
The liveness probe fires at t=180 s, while the model is still loading (load completes at ~240 s). Two consecutive misses (at t=180 and t=210) exhaust the failureThreshold, so Kubernetes sends SIGKILL at ~t=210 s. The pod enters CrashLoopBackOff. First signal: kubectl describe pod shows 'Liveness probe failed' events; the container log shows 'Loading model weights...' with no 'Model loaded' line. Fix: set initialDelaySeconds ≥ 270 (240 s + 30 s buffer) and raise failureThreshold to 3.
# prometheus.yml — scrape vLLM metrics scrape_configs: - job_name: vllm static_configs: - targets: ["vllm-host:8000"] metrics_path: /metrics scrape_interval: 15s # alert_rules.yml groups: - name: vllm_sla rules: - alert: HighTTFT expr: histogram_quantile(0.95, rate(vllm:e2e_request_latency_seconds_bucket[2m])) > 2 for: 3m - alert: KVCachePressure expr: vllm:gpu_cache_usage_perc > 0.90 for: 1m
histogram_quantile(0.95, rate(...))vllm:e2e_request_latency_seconds_bucketvllm:gpu_cache_usage_percfor: 1mThe scrape config targets vLLM's native /metrics endpoint; the alert rules cover the two most actionable signals — SLA-breach latency and KV-cache saturation.
Note the for: 3m on HighTTFT: a single p95 spike during a burst shouldn't page; sustained degradation should. KVCachePressure uses for: 1m because cache exhaustion escalates fast and the scheduler will start stalling within seconds of hitting the ceiling.
Your colleague is right that 5m is likely too long — a queue of 50 waiting requests will inflate TTFT within seconds, not minutes, so the alert fires well after the SLA is already breached. Shortening to 30s catches the problem faster but increases alert noise during legitimate short bursts (e.g., a traffic spike that drains in under a minute). The right value depends on your burst profile: if p99 burst duration is < 60s, use for: 1m; if bursts are rare and sustained, 2m is a reasonable middle ground. Changed lines: 'for: 1m' replaces 'for: 5m'.
Three failure patterns, each with a distinct observable symptom:
HTTP 503 Service Unavailable or requests hang for 30–120 s then time out. The replica's logs show Loading model weights... still in progress.Before trusting AI-generated Kubernetes probe or Prometheus config, check these points — LLMs frequently get them wrong:
/health and readiness uses /health/ready — not swapped.time vllm serve <model> on your hardware. Verify the delay exceeds it by ≥ 30 s.http://vllm-host:8000/metrics | grep vllm: and confirm exact histogram/gauge names match alert rules. vLLM renames metrics across minor versions.The next module — Production Hardening and Failure-Mode Mitigation — covers five failure modes (OOM from unbounded context, chat-template mismatch, allreduce/quantization fusion bugs) that your monitoring stack must catch before they cascade.
Work through the five most common production failure modes — OOM from unbounded context, chat-template mismatch, allreduce/quantization fusion bugs (patched in v0.25.1), cold-start traffic spikes, and misconfigured API keys — and apply the corresponding mitigations. You'll run a load test with `locust` or `wrk2` against your server from Modules 1–5, interpret the results, and audit an AI-generated `vllm serve` command for silent misconfigurations.
A systematic walkthrough of the five most common vLLM production failure modes — OOM, chat-template mismatch, quantization fusion bugs, cold-start spikes, and misconfigured API keys — with reproducers, load-test interpretation, and an AI-generated command audit.
Why this matters: Production deployments fail in predictable ways that only appear under real load; this module gives you the reproducer scripts, diagnostic signals, and hardening checklist to catch them before they hit users.
Your server from Modules 1–5 runs cleanly in isolation. Under concurrent long-context load, five failure classes emerge. Monitoring alone won't prevent them — you must reproduce and fix each one.
--max-model-len: each concurrent request claims up to max-model-len KV blocks. With enough concurrency, the allocator exhausts VRAM. The process crashes with torch.cuda.OutOfMemoryError or silent SIGKILL.--chat-template points to the wrong file. The model sees raw role strings instead of special tokens. Output becomes garbled and role-leaking. No error is raised.--api-key without enforcement on every route leaves the endpoint silently open. Rotating the key without restart leaves it silently broken.OOM under concurrent load is almost always a --max-model-len × --max-num-seqs product problem. Each sequence holds up to max-model-len KV blocks. Worst-case VRAM reservation is their product times per-token block size.
Cap both parameters: set --max-model-len to the 99th-percentile prompt+completion length from your load test. Don't use the model's architectural maximum. Then set --max-num-seqs so that max-model-len × max-num-seqs × bytes-per-block stays under your KV cache budget. The budget is reported at startup as # GPU blocks.
Chat-template mismatch is silent: the model generates tokens, but role boundaries collapse. Reproduce it by omitting --chat-template and sending a multi-turn request. Look for the literal string "<|user|>" or "<|assistant|>" appearing verbatim in the completion.
--chat-template $(python -c "from transformers import AutoTokenizer; t=AutoTokenizer.from_pretrained('meta-llama/Llama-3-8B-Instruct'); print(t.chat_template)" > /tmp/tmpl.jinja && echo /tmp/tmpl.jinja) or use --chat-template tokenizer (vLLM ≥ 0.4).import threading, requests, time URL = "http://localhost:8000/v1/chat/completions" HEADERS = {"Authorization": "Bearer my-secret-key"} LONG_PROMPT = "Summarize: " + ("word " * 6000) # ~6k tokens def fire(): requests.post(URL, headers=HEADERS, json={ "model": "meta-llama/Llama-3-8B-Instruct", "messages": [{"role": "user", "content": LONG_PROMPT}] }) threads = [threading.Thread(target=fire) for _ in range(20)] for t in threads: t.start() for t in threads: t.join()
"word " * 6000threading.Thread(target=fire)requests.post(..., json={...})This script fires 20 concurrent long-context requests — the minimal reproducer for KV cache exhaustion. The key insight is that vLLM reserves KV blocks eagerly at request admission, so 20 simultaneous 6k-token prompts can blow the budget before a single token is generated.
The KV allocator tries to reserve 20 × 8192 blocks concurrently. On a 40 GB A100 with a typical 30–35 GB KV budget after weights, this exhausts VRAM. You'll see torch.cuda.OutOfMemoryError in the server log (or a SIGKILL with no Python traceback if the OOM killer fires). The client gets a 500 or a connection reset — no 4xx, so your load balancer health check may not catch it. Fix: set --max-model-len 4096 and --max-num-seqs 32 to keep the worst-case reservation inside budget.
# Install: pip install locust # locustfile.py lives alongside this command locust \ --headless \ --users 50 \ --spawn-rate 5 \ --run-time 2m \ --host http://localhost:8000 \ --csv results # After run: awk -F',' 'NR>1{print $7, $8, $9}' results_stats.csv # columns: 50th, 95th, 99th latency (ms)
--users 50 --spawn-rate 5--csv resultsawk -F',' 'NR>1{print $7,$8,$9}'Run locust in headless mode against your vLLM server and read the CSV percentile columns directly. The p50/p95/p99 spread is the primary diagnostic: a wide spread signals queue starvation or prefill monopolization, not raw throughput limits.
A flat p50 with a blown-out p99 under high GPU utilization points to prefill queue saturation: long prompts are blocking short ones. The Module 3 knob is --max-num-batched-tokens. If it's set to the model's architectural max (e.g. 32768), a single long prefill monopolizes the batch slot. Try halving it (e.g. 16384) to let the scheduler interleave shorter prefills, which compresses p99 TTFT without hurting throughput significantly. Changed lines: --max-num-batched-tokens 16384.
An AI assistant produces the following command for your Llama-3-8B-Instruct deployment on a 2×A100 80 GB node:
vllm serve meta-llama/Llama-3-8B-Instruct --tensor-parallel-size 2 --gpu-memory-utilization 0.95 --max-model-len 131072 --quantization awq --dtype float16 --port 8000
--max-model-len 131072 with --quantization awq: AWQ reduces weight memory but not KV cache size. At 131k context × 2 GPUs × 0.95 utilization, the KV allocator will exhaust VRAM under even modest concurrency. The server starts fine; it crashes under load.--dtype auto (which resolves to bfloat16 on Ampere). Forcing float16 can trigger the pre-v0.25.1 fusion assertion or produce subtly wrong logits — no exception, just degraded output quality.dtype=....<|user|><|assistant|>) to catch template mismatch.Drag to see how --max-num-batched-tokens shifts the prefill-vs-latency tradeoff. The sweet spot depends on your actual prompt-length distribution from the load test.
Cold-start TTFT spikes are a readiness-probe problem, not a vLLM bug. The Module 5 /health endpoint returns 200 as soon as the HTTP server binds. Weights are not fully loaded yet. KV cache is not allocated. Use /health/ready or poll /v1/models as the readiness gate. The load balancer withholds traffic until the first request can be served.
For traffic spikes on a warm server, pre-warm the KV cache. Send a synthetic batch of representative prompts immediately after readiness. This fills the paged blocks and avoids the first-batch prefill surge.
API key rotation requires a server restart. vLLM reads --api-key once at startup. There is no hot-reload path. The safe pattern: bring up a new instance with the new key. Shift traffic via the load balancer. Drain and stop the old instance. A zero-downtime rotation without this two-instance dance leaves a window where the old or new key is silently rejected.
Before reviewing the build order, reconstruct from memory: what are the five flags you'd set on vllm serve for a production launch, which two metrics signal a scheduler bottleneck vs. a GPU memory bottleneck, and what is the first thing to check when a chat-tuned model returns garbled output?
Apply what you learned to vLLM Deployment.
You are choosing how to deploy vLLM on a Kubernetes cluster where the ops team enforces immutable, versioned artifacts and needs to roll back instantly on failure. Which installation path is the better fit and why?
A. pip install — faster iteration and no image-build step.
B. Docker image — reproducible, versioned artifact that supports instant rollback via image tag.
C. pip install — avoids Docker layer caching overhead at runtime.
D. Docker image — Docker images always use less VRAM than pip-installed packages.
Docker images are the right call here because they produce an immutable, tagged artifact that Kubernetes can roll back to by simply pointing to the previous image tag — that is exactly what the ops team needs. pip install is better for rapid local experimentation where you want to iterate quickly without a build step, but it does not give you a versioned artifact. Option C is a fabricated benefit — Docker layer caching has no effect on runtime VRAM. Option D is false; the installation path has no bearing on VRAM consumption.
A Llama-3 70B model in full BF16 precision has 70 billion parameters. Each parameter occupies 2 bytes in BF16. You are running on a single A100-80GB GPU. Without any KV cache or activation overhead, can the weights alone fit? Show your reasoning in one or two sentences, then state what flag you would add to vllm serve to make the weights fit on two A100s instead.
70 billion parameters at 2 bytes each equals 140 GB — nearly double the 80 GB available on one A100-80GB, so a single-GPU deployment is impossible at full precision. --tensor-parallel-size 2 shards the weight matrices across two GPUs, halving the per-GPU weight footprint to roughly 70 GB, leaving headroom for KV cache and activations. A common mistake is forgetting to multiply parameter count by bytes-per-parameter, or assuming BF16 is 4 bytes (that is FP32).
Consider this Python SSE consumer snippet:
for line in response.iter_lines():
if line == 'data: [DONE]': break
What silent bug does this code contain?
The critical missing piece is connection-drop handling. If the server closes the TCP connection before sending [DONE] — due to a timeout, OOM, or restart — iter_lines() raises a requests.exceptions.ChunkedEncodingError (or similar) that this code never catches, causing an unhandled crash. Option A is false — iter_lines() works correctly for SSE. Option B is false — requests does NOT strip the 'data: ' prefix; the consumer must do that itself, but the comparison shown would actually work for the sentinel line as sent by vLLM. Option C describes a real concern for a production consumer but is not a bug in the break logic shown — the snippet is only illustrating the sentinel check, not claiming to be a complete parser.
Your vLLM server's Prometheus metrics show: gpu_cache_usage_perc is at 35% and the scheduler's waiting queue depth is climbing steadily past 200 requests. Which conclusion and action are correct?
Low gpu_cache_usage_perc (35%) means the GPU has plenty of KV-cache headroom — it is not memory-pressured. A rising queue depth with idle GPU capacity is the textbook signal that the scheduler is artificially throttling how many sequences it admits per step. The fix is to raise --max-num-seqs or --max-num-batched-tokens so the scheduler feeds the GPU more work. Option A is wrong because tensor parallelism adds compute but does not fix a scheduler admission limit. Option C misreads the metric — low cache usage means the opposite of thrashing. Option D would make the problem worse by admitting even fewer sequences.
A colleague shares this vllm serve command for a production chat service and asks you to audit it:
vllm serve meta-llama/Llama-3-70b --port 8000 --tensor-parallel-size 4
Which set of silent misconfigurations is present?
Three real silent misconfigurations are present: (1) omitting --host means vLLM binds to 0.0.0.0 and the API is reachable from any network interface — a security exposure in production; (2) omitting --api-key means any caller can hit the endpoint without authentication; (3) omitting --max-model-len means vLLM will accept requests up to the model's full trained context length, and concurrent long-context requests can exhaust VRAM and trigger OOM. Option B is false — tensor-parallel-size 4 is perfectly valid and does not need to match parameter count. Option C is false — vLLM accepts Hugging Face repo IDs directly. Option D is false — full precision is sometimes intentional and fits on multi-GPU setups; it is a tradeoff, not always an error.