Shrink latency and cost without losing the behavior your product needs.
Optimization is only useful if you know what you are optimizing and what quality must be preserved.
Measurement identifies whether memory, compute, or routing is the problem.
Why this matters: It prevents making a cheaper system that silently gets worse.
The first step is a baseline: task quality, p50/p95 latency, tokens per second, memory use, cost per request, and failure slices. Without that, a cheaper model can quietly damage the product.
Problem anchor: a support summarizer is accurate with a 13B model but too slow for live chat. The team must reduce latency without losing escalation details.
Quantization can lower VRAM and bandwidth by representing weights or activations with fewer bits, but quality and speed depend on method and hardware.
Quantization stores model values with fewer bits.
Why this matters: It is the first lever when the same model fits poorly or costs too much.
Quantization maps high-precision values into lower-bit formats such as 8-bit or 4-bit representations. The common LLM goal is to fit a model into less memory or increase batch capacity while preserving enough quality.
The KB verdict is practical: use quantization when a model already works but needs lower memory, lower bandwidth, or cheaper inference.
params_billions = 7 for bits in [16, 8, 4]: gb = params_billions * 1_000_000_000 * bits / 8 / 1_000_000_000 print(bits, "bit ->", round(gb, 1), "GB")
input datadecision ruleprint(...)Run this as a tiny model of the mechanism. The point is to predict behavior before trusting the output.
It prints about 14.0 GB, 7.0 GB, and 3.5 GB. Real systems add overhead, activations, and KV cache, but bit width directly changes weight storage.
Distillation uses a teacher model or workflow to train a smaller student for a known task distribution.
Distillation transfers behavior into a different model.
Why this matters: It is useful when repeated internal tasks do not need the largest model.
Distillation trains a student model to imitate a stronger teacher on a known workload. The student may be smaller, faster, or easier to deploy, but it must be evaluated against both teacher and baseline.
Use distillation when you repeatedly run a capability such as routing, summarization, extraction, or grading and can collect realistic teacher outputs.
The teacher produces high-quality summaries but costs too much for every chat. The team samples real chats, filters sensitive data, generates teacher summaries, trains a smaller student, then compares escalation-detail recall on held-out chats.
Batching, KV-cache management, request routing, token caps, and caching can improve cost/latency without retraining.
Serving optimization changes how requests flow through the system.
Why this matters: It can reduce latency without altering model weights.
If quality is good and the model fits, serving tactics may be enough: continuous batching, prompt or semantic caching, shorter max output, routing simple requests to smaller models, or separating online and background work.
KV cache matters because decode stores keys and values for active sequences. Long contexts and many concurrent users can make memory scheduling the bottleneck.
When an AI proposes inference optimization, inspect the bottleneck and the guardrails.
The winning optimization is the one that meets quality tolerance at lower cost or latency, with provenance and rollback.
The final metric combines quality, latency, and cost.
Why this matters: Advanced systems need evidence that savings did not break the product.
Decision this forces: Choose quantization, distillation, serving optimization, or no ship.
| Option | Best when | Quality risk | Cost/complexity | When to choose | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|---|
| Quantization | Same model works but memory or bandwidth is too high. | Math, code, rare tokens, or long context may regress. | Medium; method and kernels matter. | Use after full-precision quality is acceptable. | Choose for VRAM or bandwidth pressure with acceptable quality loss. | Medium | Medium |
| Distillation | Known repeated workload can be imitated by a smaller model. | Student misses rare cases or inherits teacher errors. | High; needs data generation, training, evals. | Use for high-volume narrow tasks with teacher labels. | Choose when a smaller specialized model can preserve needed behavior. | High | High |
| Serving tactics | Runtime scheduling, caching, or routing is the bottleneck. | Routing can send hard cases to weak paths. | Medium; infra complexity rises. | Use before changing weights if orchestration can solve it. | Choose when better request handling meets the target. | Medium | Medium |
The 13B summarizer is accurate but p95 latency is too high and memory limits batch size. Start with quantized loading and serving changes. Try distillation only if the workload is stable and repeated enough to justify training.
From memory, classify each optimization: quantization changes numeric representation, distillation changes the model, serving tactics change request execution. Then name the eval that prevents a cheaper-but-worse launch.
Given a support summarizer that is accurate but too expensive, design a three-stage optimization experiment: quantized load, distilled student, and serving/router changes. Define the quality tolerance and rollback trigger. Next rung: serving LLMs with vLLM.
Before you look anything up: name the four dimensions you should measure as a baseline BEFORE applying any inference optimization. Write them from memory.
Optimizing without a baseline across all four dimensions means you cannot tell whether a change actually helped or accidentally hurt quality.
You quantize a 7B model from FP16 to INT4 and notice it scores well on your benchmark but occasionally produces confident, factually wrong answers it never produced before. Which quantization failure mode best describes this?
Large outlier activations are disproportionately damaged by aggressive low-bit quantization, which can silently corrupt specific layers and produce confident but wrong outputs even when aggregate benchmark scores look acceptable.
A team wants a model that is 4× smaller AND retains nuanced reasoning ability on a complex legal QA task. They have 10 000 labeled examples and two weeks of GPU time. Which approach is most likely to outperform INT4 quantization of the large model?
Distillation transfers the teacher's probability distribution (soft labels) to the student, which is especially valuable for nuanced tasks where hard labels alone lose reasoning signal — something quantization cannot recover.
Your inference costs spike because each request re-encodes the same 2 000-token system prompt from scratch. Which serving tactic directly eliminates this redundant computation?
KV cache prefix caching saves the computed key-value pairs for a repeated prompt prefix so subsequent requests skip re-encoding it, directly cutting latency and compute for shared-context workloads.
You have a quality tolerance of ≥ 85 % on your eval suite. INT4 quantization scores 83 %, serving optimizations alone score 86 % (at higher cost), and distillation would take 6 weeks. Your launch is in 3 weeks. What is the correct ship/no-ship decision and primary rationale?
Quality tolerance is a hard constraint, not a suggestion — INT4 falls below it, distillation misses the deadline, so serving optimization is the only option that meets both the quality bar and the ship date.