Control token, retrieval, tool, and serving costs without gutting quality.
A support workflow can only be optimized after the team names the unit of value, the user-visible latency target, and the quality floor.
A support workflow can only be optimized after the team names the unit of value, the user-visible latency target, and the quality floor.
Why this matters: Define cost per resolved task instead of cost per token alone.
Problem anchor: a support assistant handles billing questions, password resets, and refund edge cases. Finance wants a 35 percent reduction in monthly model spend, but product has promised that 95 percent of answered tickets show the first useful token within two seconds and keep the same resolution quality.
The tempting shortcut is to swap every request to a cheaper model. The disciplined approach is to optimize cost per resolved ticket at the latency target. That metric includes the first attempt, retries, fallback calls, retrieval, tool calls, and any human escalation caused by a weak answer.
For the running case, write the release budget as a small contract: billing and refund tickets may use the strong model; password resets and status lookups should use a fast small model; repeated policy explanations may use a cache only when tenant, prompt version, policy version, and locale match.
Latency optimization starts by decomposing the serving path into route selection, queueing, prefill/cache behavior, decode, tools, and fallback.
Latency optimization starts by decomposing the serving path into route selection, queueing, prefill/cache behavior, decode, tools, and fallback.
Why this matters: Name the moving parts that produce p95 latency.
A single chat turn can cross a gateway, a retrieval service, a model server, a tool, and a fallback provider. The user sees one delay, but the trace should show queue time, route choice, cache lookup, prefill, decode, tool time, and retry time as separate spans.
Two framings help. The product framing asks, "Where did the user wait?" The systems framing asks, "Which resource was saturated or repeated?" If queue age is high, adding a cheaper model will not help. If decode is slow, shorter answers or smaller models may help. If prefill dominates, prompt caching or context trimming may help.
A refund question misses semantic cache, waits 620 ms in the gateway queue, spends 900 ms in retrieval and reranking, then streams from the strong model after 1.7 seconds. The answer is good, but the first useful token arrives late. The fix is not "use a cheaper model"; first split refund traffic from batch jobs, cap retrieval top-k, and test whether stable policy text can be prefix cached.
This repeats the earlier budget idea: optimize the path that violates the product contract, not the component with the most visible bill.
| Option | Lever | What it improves | Failure to test | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Route tiering | Route easy requests to cheaper or faster models | Cost and median latency | Misrouted hard tickets or weaker policy handling | Use after labeling request types with a golden set. | Medium | Medium |
| Prefix or prompt cache | Reuse stable prompt prefixes or exact responses | Prefill time and repeated-call cost | Stale, cross-tenant, or wrong-policy answers | Use when prompts share stable system and document prefixes. | Medium | Medium |
| Retry budget | Limit automatic retries and fallback chains | Tail latency and runaway spend | Provider blips turn into user-visible failures too quickly | Use whenever agent loops or tools can trigger repeated calls. | Medium | Medium |
The best route policy is the simplest one that saves money without hiding quality, privacy, or incident-debugging risk.
The best route policy is the simplest one that saves money without hiding quality, privacy, or incident-debugging risk.
Why this matters: Compare direct calls, gateways, caches, and self-hosted serving.
Decision this forces: Choose the right level of rigor for Cost and Latency Optimization: baseline, production design, or advanced optimization.
A gateway policy, a semantic cache, and a self-hosted vLLM endpoint can all reduce cost, but they move risk to different places. The gateway centralizes model choice and budgets. The cache adds correctness and privacy questions. Self-hosting trades provider margin for GPU operations and utilization risk.
The useful question is not "Which is cheapest?" It is "Which lever saves money on this workload while preserving evidence?" For support, a cache may work for stable policy explanations; route tiering may work for password resets; strong-model fallback may remain necessary for refunds and legal language.
Password reset: small fast model, max 250 output tokens, no fallback unless the tool call fails. Billing status: small model plus account lookup, fallback to strong model only when the tool response is ambiguous. Refund exception: strong model first, retrieval over current policy, and no semantic cache because the answer depends on customer state.
The same policy records route_version=2026-06-support-v3, selected provider, cache hit, retry count, and final model. During review, finance can see savings and support leads can inspect quality regressions by ticket class.
A cost optimization is not real until a test can show cheaper, fast-enough, still-correct behavior on representative traffic.
A cost optimization is not real until a test can show cheaper, fast-enough, still-correct behavior on representative traffic.
Why this matters: Write a minimal routing simulation with trace fields.
requests = [
{"kind": "reset", "tokens": 500, "needs_strong": False},
{"kind": "refund", "tokens": 1600, "needs_strong": True},
{"kind": "policy", "tokens": 900, "needs_strong": False},
]
price = {"small": 0.15, "strong": 2.50} # dollars per 1M tokens
retry_rate = {"small": 0.18, "strong": 0.03}
def route(req):
if req["needs_strong"]:
return "strong"
return "small"
total = 0
for req in requests:
model = route(req)
first_try = req["tokens"] * price[model] / 1_000_000
expected_retry = first_try * retry_rate[model]
total += first_try + expected_retry
print(req["kind"], {"model": model, "expected_cost": round(first_try + expected_retry, 6)})
print({"expected_batch_cost": round(total, 6)})Route policy changes first. Trace fields should already expose the retry problem; token price alone cannot fix a path that often fails and repeats work.
When an AI assistant proposes a cost plan, review it like a release artifact. It should identify the user-visible latency budget, the traffic classes, the quality metric, the expected savings, and the measurements that prove retries did not erase the savings.
The synthesis is a release playbook: budget, decompose, choose levers, instrument, compare, and roll back if quality or latency moves.
The synthesis is a release playbook: budget, decompose, choose levers, instrument, compare, and roll back if quality or latency moves.
Why this matters: Recall the optimization loop without notes.
Decision this forces: Choose the right level of rigor for Cost and Latency Optimization: baseline, production design, or advanced optimization.
Close the lesson by answering this from memory: "For a support assistant with a fixed p95 target, how do I reduce cost without hiding worse quality?" A strong answer should mention unit-of-value budgeting, latency decomposition, route tiers, cache boundaries, retry budgets, traces, and rollback.
Spaced recall: connect this back to module 1. The budget was not dollars alone; it was cost per resolved ticket at an agreed latency and quality floor. Every later lever exists to protect that first sentence.
Pick one workflow with at least 30 representative examples. Label each request class, propose a small/strong/cache route, define the rollback switch, and run old policy versus new policy on the same examples. Report cost per resolved task, p95 time to first token, failure rate, and the three worst regressions.
Next rung: connect this release habit to monitoring model drift and quality, where the question becomes whether the optimized policy stays good after traffic changes.
From memory, reconstruct the cost-and-latency release loop: define cost per resolved task, split latency into traceable stages, choose routes and caches by risk, instrument retries and fallbacks, then ship only when quality and p95 latency stay inside budget.
Create a one-page release-ready plan for cost-latency budget with routing, caching, and fallback rules. Include: the user problem, realistic input, mechanism, design choice, runnable or reviewable check, metric (cost per resolved ticket at target p95 latency), failure case (a cheaper model saves money but triggers retries that cost more overall), owner, and the next rung after this lesson.
Your team cuts token usage by 40% but customer complaints about wrong answers double. According to the budgeting-first principle, what went wrong?
The quality floor is a hard constraint — cost savings are only valid if answer quality stays above the pre-defined minimum.
A trace shows your workflow's p95 latency is 4 s. Decode time is 0.4 s. Where should you look first to find the biggest savings?
Decode is only 10% of total latency here; queue time and cache misses are the dominant contributors and should be traced and addressed first.
Give ONE concrete example of how a poorly designed cache key can violate tenant or policy boundaries — and what the consequence would be.
Cache keys must encode every dimension that affects correctness and access rights; omitting tenant or policy context turns a performance win into a security or compliance failure.
You add an automatic fallback from a fast, cheap model to a slower, expensive model on any error. After a week, your cost savings have nearly vanished. What is the most likely cause?
Retries and fallbacks can silently consume the entire cost budget if their frequency isn't instrumented and capped — a key insight from the instrumentation module.
When rolling out a new route-cost policy, why is a controlled release (e.g., a canary or shadow deployment) preferred over a full cutover?
A controlled release surfaces unexpected interactions between the new policy and real traffic patterns without exposing all users to potential regressions.