Serving a large language model is dominated by inference cost, and three techniques cut it. Quantization stores weights in lower precision (16-bit to…
See why the cost battle at scale is about serving, not training.
A model is trained once but serves requests millions of times, so at scale inference — not training — is where the money and latency go. Two things bind it: memory (the weights and the growing attention cache must fit in GPU memory) and compute per token.
Optimization attacks both. Some techniques shrink the model (quantization, distillation); others improve how it is executed (batching, caching, speculative decoding). The goal is more tokens per second per dollar, at acceptable quality.
Lower the precision of weights to save memory and speed compute.
Models are typically trained in 16-bit precision. Quantization stores the weights in fewer bits — 8-bit or even 4-bit — so the model takes a fraction of the memory and often runs faster, since memory bandwidth is the bottleneck. A model that needed a large GPU may then fit on a smaller or single one.
The cost is a small, usually acceptable accuracy drop. Methods like GPTQ and AWQ quantize carefully to minimize it, and 4-bit is common in practice. Push precision too low and quality degrades, so there is a precision-versus-quality knob to tune.
Quantization also enables cheap fine-tuning: QLoRA keeps a 4-bit quantized base frozen while training small adapters, letting large models be adapted on a single GPU.
Compress capability by training a small student from a big teacher.
Distillation trains a smaller student model to imitate a larger teacher, learning from the teacher's outputs (and often its probability distributions) rather than from labels alone. The student ends up far cheaper to run while retaining much of the teacher's behavior on the target task.
It works best when the student is specialized to a task distribution the teacher handles well — cloning a strong model's behavior into a compact one you can serve cheaply. The trade-off is that a small student rarely matches the teacher's full breadth, so it shines on focused tasks, not everything.
Raise throughput without changing the model, and weigh trade-offs.
Several wins come from execution, not the model. Batching serves many requests together to use the GPU fully. The KV cache stores attention keys and values from earlier tokens so each new token reuses them instead of recomputing the whole sequence — essential for fast generation. Continuous batching and paged attention (as in vLLM) manage that cache efficiently across many concurrent requests.
Speculative decoding uses a small draft model to propose several tokens that the large model verifies in one pass, accepting the correct ones — often a large speedup with identical output, since the big model still validates every token.
These stack but interact. Aggressive quantization can hurt quality — measure accuracy after, not just speed. Distillation costs an upfront training effort and narrows scope. Batching lowers cost per request but can raise individual latency. Choose based on whether you optimize for cost, latency, or fitting a memory budget, and always benchmark quality alongside speed.
At scale, inference dominates LLM cost, bound by memory and compute per token. Quantization stores weights in 8- or 4-bit to shrink memory and speed compute with minor accuracy loss (and enables QLoRA). Distillation trains a small student to mimic a big teacher, cheap on focused tasks. System-level wins — batching, KV caching, paged attention, and speculative decoding — raise throughput without changing the model. They stack, but always benchmark quality alongside speed.
You must serve a capable model on a single mid-range GPU under tight latency. Decide which of quantization, distillation, and serving optimizations you would apply and in what order, and describe the one measurement you would take after each to ensure quality hasn't dropped too far.
Why is inference optimization so important for LLMs?
Repeated serving makes per-token cost and memory the main levers, which quantization, distillation, and system tricks target.
What does quantization do?
Fewer bits per weight shrink memory and ease the bandwidth bottleneck; methods like GPTQ and AWQ limit the accuracy hit.
What is knowledge distillation?
Distillation transfers capability into a compact student, best on a focused task distribution, though it rarely matches the teacher's full breadth.
What does speculative decoding do?
Because the large model still verifies every token, speculative decoding accelerates generation without changing the output.