Cost and latency in an LLM app come mostly from tokens and model choice, so the biggest wins are picking the smallest model that does the job, cutting…
Locate where the money and the milliseconds actually go.
Two levers dominate. Cost is billed per input and output token, so long prompts, big retrieved contexts, and verbose answers all add up. Latency is driven by the model's size (bigger is slower per token) and the number of output tokens (each is generated sequentially), plus any network and tool time.
The first move is always to measure: log tokens and latency per request and find the hot spots. Usually a small number of prompts — a bloated system prompt, an oversized retrieved context — account for most of the cost, so you fix those rather than micro-optimizing everything.
Cut cost at the source: smaller models and fewer tokens.
The single biggest lever is model selection. Model prices span a large range, and the biggest model is rarely needed for every task. Use the smallest, cheapest model that passes your quality bar for each job — classification and extraction often work fine on small models, reserving large ones for genuinely hard reasoning.
Then trim tokens: tighten verbose system prompts, retrieve only the passages you need instead of stuffing the context, and cap output length. Fewer tokens means both lower cost and lower latency, since each output token is generated in sequence.
Reuse work instead of paying for it twice.
Caching avoids redoing work. Prompt caching reuses the model's processing of a repeated prefix — a long system prompt or few-shot examples sent on every call — so that shared portion isn't reprocessed each time, cutting cost and time-to-first-token for repeated prefixes. Many providers support it, sometimes automatically.
Response caching goes further: if the same request comes in again, return the stored answer without calling the model at all. Exact-match caching is simple; semantic caching returns a cached answer for a sufficiently similar question. Both are powerful when queries repeat, which they often do.
Hide latency, route smartly, and avoid the common traps.
Streaming sends tokens to the user as they're generated, so the response starts appearing almost immediately. It doesn't reduce total time, but it transforms perceived latency, which is what users feel. Use it for any interactive output.
Model routing sends each request to the cheapest capable model and escalates only hard cases to a larger one — a cascade that can slash cost while keeping quality where it matters. Pair with batching for high-volume backends to use the hardware fully.
Watch for: reaching for the biggest model by default; optimizing before measuring, so you tune the wrong thing; caching without invalidation, serving stale answers when data changes; and cutting output so aggressively that quality drops. Optimize the measured hot spots, and always check that a cheaper setup still meets your quality bar with an eval.
LLM cost and latency come mostly from tokens and model size. Measure first — a few prompts usually dominate. Then pull the big levers: pick the smallest model that meets the bar, trim prompt and output tokens, and cache (prompt caching reuses a repeated prefix; response caching skips identical or similar calls). Stream to cut perceived latency, route easy queries to cheap models and escalate hard ones, and batch high-volume backends. Always confirm a cheaper setup still passes an eval.
Your LLM app's bill is climbing. Describe how you would find the costliest requests, two changes you would try first (model choice and caching), and the check you would run to be sure the cheaper version still meets your quality bar.
What are the two biggest drivers of LLM cost and latency?
Cost is per-token and latency scales with model size and output length, so tokens and model choice are the main levers.
What is the single biggest lever for reducing cost?
Right-sizing the model per task captures the largest savings; many tasks don't need the biggest, most expensive model.
What is the difference between prompt caching and response caching?
Prompt caching saves reprocessing a shared prefix each call; response caching skips the model entirely for repeated requests.
What does streaming improve?
Streaming shows tokens as they generate, making interactive apps feel fast even though the full generation takes the same time.