vLLM is an open-source inference server that makes self-hosted large language models fast and cost-efficient at scale. Its two key techniques are…
See the inefficiencies that plain LLM serving leaves on the table.
Serving an LLM naively wastes the expensive GPU two ways. First, batching: if you wait to gather a fixed batch of requests and run them together, short requests finish early and their slots sit idle until the longest one is done, so the GPU is underused. Second, the attention key-value (KV) cache: each request needs memory for the tokens it has generated, and reserving a big fixed block per request leaves much of it unused — memory that could have served more requests.
Because LLM inference is memory-bound and GPUs are costly, these inefficiencies directly raise cost and cut how many users you can serve. vLLM exists to remove them.
Understand how vLLM manages the KV cache without waste.
PagedAttention is vLLM's headline idea. Instead of reserving one large contiguous block of KV-cache memory per request, it splits the cache into small fixed-size blocks and allocates them on demand, like an operating system's virtual memory with pages. A request uses only the blocks it actually needs.
This nearly eliminates the wasted, reserved-but-unused memory, so far more requests fit in the same GPU memory at once. It also lets requests share cache blocks (for example, a shared prompt prefix), saving even more.
See how vLLM keeps the GPU busy by never waiting.
Continuous (or in-flight) batching solves the idle-slot problem. Instead of running a fixed batch to completion, vLLM works at the token level: at each generation step it can add newly arrived requests to the batch and drop finished ones, so a slot freed by a short request is immediately reused by a waiting one.
The GPU stays saturated with useful work, which is what raises throughput several-fold. Combined with PagedAttention's memory efficiency, vLLM serves many more concurrent requests per GPU than naive serving.
Run vLLM as a server and avoid the common mistakes.
In practice vLLM runs as a server hosting an open-weight model and exposes an OpenAI-compatible API, so client code written for a hosted provider often works by pointing at your vLLM endpoint. You give it the model and GPU settings, and it handles batching and cache management for you.
It supports quantized models to fit larger ones, and multi-GPU serving for models too big for one card. For high-volume self-hosting, this combination of throughput and a familiar API is why vLLM is a common default.
# start an OpenAI-compatible server vllm serve meta-llama/Llama-3.1-8B-Instruct # clients call it like a hosted API, base_url=http://localhost:8000/v1 POST /v1/chat/completions { "model": "...", "messages": [...] }
One command serves the model with continuous batching and PagedAttention on by default; apps call the standard chat endpoint, so existing OpenAI-style clients work by changing the base URL.
vLLM makes self-hosted LLM serving fast and cost-efficient by fixing two wastes of naive serving. PagedAttention manages the attention KV cache in small on-demand blocks like virtual memory, eliminating reserved-but-unused memory and enabling sharing. Continuous batching adds and drops requests from a running batch each step so the GPU stays saturated. Together they multiply throughput per GPU. vLLM runs as a server with an OpenAI-compatible API, supports quantized and multi-GPU models, and is a common default for high-volume self-hosting.
You must self-host an 8B model for a high-traffic app on limited GPUs. Explain how PagedAttention and continuous batching let you serve more users per GPU, and why using vLLM's OpenAI-compatible API simplifies switching from a hosted provider.
What problem does vLLM solve?
vLLM targets inference efficiency — idle batch slots and wasted KV-cache memory — to raise throughput and cut serving cost.
What is PagedAttention?
PagedAttention allocates KV-cache memory in pages as needed and can share blocks, fitting far more concurrent requests in GPU memory.
What is continuous batching?
Continuous batching reuses slots the moment a request finishes, avoiding the idle time of fixed batches and boosting throughput.
How do applications typically call a vLLM server?
vLLM exposes a standard chat-completions API, making it a near drop-in replacement for a hosted provider in self-hosting setups.