Infrastructure · Guide
AI Infrastructure: Serving, Scaling, and Cost
Run models in production — vLLM, Ollama, gateways, quantization, GPUs, cost and latency.
Running models in production means serving them, scaling them, and controlling cost and latency. This guide covers deploying open models with vLLM, running local models with Ollama, model gateways like LiteLLM, quantization and inference optimization, GPUs and TPUs, embedding pipelines at scale, and monitoring for model drift.
Generate your own lesson →What you'll learn
- Deployment Patterns for AI Apps
- Embedding Pipelines at Scale
- GPUs, TPUs, and AI Hardware
- Model Gateways with LiteLLM
- Model Registries and Versioning
- Monitoring Models: Drift and Quality
- Optimizing Cost and Latency in LLM Apps
- Running Hugging Face Transformers in Production
- Running Local Models with Ollama
- Serving LLMs at Scale with vLLM
Lessons in this guide (10)
Deployment Patterns for AI Apps
Deploying an AI app means putting a model-backed service into production reliably and affordably. The core patterns: separate the model-serving layer from your application logic, keep the app stateless so it scales horizontally, handle slow
Embedding Pipelines at Scale
An embedding pipeline turns a large, changing corpus into vectors in a search index, and at scale it becomes an engineering problem, not a one-off script. The core concerns are batching embeddings for throughput, updating incrementally as s
GPUs, TPUs, and AI Hardware
AI runs on specialized hardware because training and inference are mostly massive amounts of matrix multiplication that can be done in parallel. GPUs have thousands of cores built for exactly this, which is why they dominate AI; TPUs are Go
Model Gateways with LiteLLM
A model gateway is a single service your apps call instead of talking to LLM providers directly. It gives you one unified API across many providers, plus central control: routing, automatic fallbacks, rate limiting, cost tracking, caching,
Model Registries and Versioning
A model registry is a central catalog that tracks trained models, their versions, metadata, and lifecycle stage (like staging and production). Versioning treats each model as a numbered, immutable artifact with a record of how it was made —
Monitoring Models: Drift and Quality
A model that performed well at launch can silently degrade because the world changes — a problem called drift. Monitoring watches production inputs and outputs to catch this early: data drift (the inputs shift), concept drift (the input-to-
Optimizing Cost and Latency in LLM Apps
Cost and latency in an LLM app come mostly from tokens and model choice, so the biggest wins are picking the smallest model that does the job, cutting unnecessary tokens, and reusing work with caching. Prompt caching reuses a repeated prefi
Running Hugging Face Transformers in Production
The Hugging Face Transformers library is the standard way to load and run open models in Python, pairing a tokenizer with a model behind a simple API. It is excellent for prototyping and for non-LLM models, but for high-throughput LLM servi
Running Local Models with Ollama
Ollama is a tool that lets you download and run open-weight large language models on your own computer with a single command. Running locally keeps data private, removes per-token API costs, and works offline, at the price of your own hardw
Serving LLMs at Scale with vLLM
vLLM is an open-source inference server that makes self-hosted large language models fast and cost-efficient at scale. Its two key techniques are PagedAttention, which manages the attention key-value cache like virtual memory to eliminate w
Frequently asked questions
How do you serve an LLM in production?
Use an inference server like vLLM (for open models on your own GPUs) or a hosted API. It batches requests, manages the KV cache, and streams tokens — behind a gateway that handles routing, retries, and rate limits.
What is a model gateway?
A model gateway (e.g. LiteLLM) is a single API in front of many providers/models. It handles routing, failover, cost tracking, caching, and rate limiting so your app code doesn't change when you swap models.
How do you cut LLM cost and latency?
Stream responses for perceived speed, cache and reuse prompts, route easy calls to smaller models, quantize open models, and trim context to what's needed. Most latency is output-token generation, so shorter outputs help most.