Detect quality shifts from model, prompt, retrieval, and data changes.
A model can be available and fast while its decisions become wrong, biased, stale, or expensive.
A model can be available and fast while its decisions become wrong, biased, stale, or expensive.
Why this matters: Distinguish system health from model quality.
Problem anchor: a loan pre-screen model returns responses in 80 ms with no server errors, yet approval quality deteriorates after a new marketing campaign changes applicant mix. Uptime dashboards are green while the model is wrong for a growing slice of traffic.
Monitoring model quality means watching more than infrastructure. Data drift asks whether inputs changed. Prediction drift asks whether outputs changed. Concept drift asks whether the relationship between inputs and outcomes changed. Performance decay asks whether users or business outcomes got worse, often after labels arrive days or weeks later.
The release card says model v42 is live. The monitoring target records application channel, region, employment type, score band, decision, latency, model version, and later repayment or manual-review outcome. Reference window: last stable 14 days after v42 rollout. Alert slices: new applicants, self-employed, region, campaign source, and protected-class proxies only where policy allows.
The output of monitoring is not "drift detected." It is a decision: investigate data pipeline, adjust thresholds, retrain, roll back, or add human review for a slice.
Drift checks are useful only when the reference window, slice definitions, and sample sizes match the decision.
Drift checks are useful only when the reference window, slice definitions, and sample sizes match the decision.
Why this matters: Select a reference window for drift monitoring.
A reference window can be training data, a recent stable production period, or an approved benchmark. Training data may be too old; yesterday may include an incident; a benchmark may miss seasonal traffic. Choose deliberately and record why.
Slices matter because aggregate stability can hide harm. A recommender may look fine overall while new users fail. A fraud model may improve global precision while missing a new merchant type. A chatbot may keep average satisfaction while one language regresses.
Overall approval precision moves from .81 to .80, below the alert threshold. In one region, precision drops from .79 to .62 after a new data vendor starts filling missing employment length with zeros. A slice alert catches what the global metric hides.
This recalls module 1: monitoring protects decisions. If the decision affects a cohort, the monitoring view must be able to see that cohort within privacy and policy boundaries.
| Option | Signal | What it catches | What it cannot prove alone | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Data drift | Input distribution changed | new traffic mix, pipeline shifts, missing values | that business performance is worse | Use as early warning. | Medium | Medium |
| Prediction drift | Output distribution changed | threshold shifts, route changes, model behavior changes | which input caused the change | Use with data and version slices. | Medium | Medium |
| Delayed labels | Outcomes arrive after decisions | true performance decay | real-time cause without supporting traces | Use for retraining and rollback evidence. | Medium | Medium |
LLM apps need classic service and drift signals plus traces for prompts, retrieval, model routes, guardrails, and evaluations.
LLM apps need classic service and drift signals plus traces for prompts, retrieval, model routes, guardrails, and evaluations.
Why this matters: Name LLM-specific monitoring dimensions.
Decision this forces: Choose the right level of rigor for Monitoring Model Drift and Quality: baseline, production design, or advanced optimization.
For an LLM support assistant, quality can change because the prompt changed, retrieval index refreshed, model route switched, semantic cache reused stale content, a tool schema broke, or policy filters became stricter. Traditional accuracy labels may arrive slowly or not at all, so traces and eval samples become core monitoring evidence.
Useful dimensions include model and response model, prompt version, retrieval index version, top-k citations, tool outcome, guardrail action, refusal rate, token usage, cost, latency, route, cache hit, fallback chain, and user feedback. OpenTelemetry-style GenAI attributes give teams a shared vocabulary for many of these fields.
Dashboard rows group by prompt version, model route, retrieval index, and lesson category. Signals: validation pass rate, citation coverage, editor rejection rate, average tokens, p95 render latency, safety intervention rate, and top regression examples. A spike in cost with stable pass rate may be acceptable; a stable cost with lower citation coverage is not.
The alert message should name the owner and next action: "RAG citation coverage for Safety lessons dropped below 92 percent after index kb_v12; run retrieval smoke set and hold enrichment releases."
Monitoring becomes operational when a check produces a severity, owner, evidence bundle, and next action.
Monitoring becomes operational when a check produces a severity, owner, evidence bundle, and next action.
Why this matters: Write a minimal monitoring check.
def alert_if_needed(live, reference): report = { "missing_rate_delta": live.missing_rate - reference.missing_rate, "approval_rate_delta": live.approval_rate - reference.approval_rate, "p95_latency_ms": live.p95_latency_ms, "model_version": live.model_version, "slice": live.slice_name, } high = abs(report["approval_rate_delta"]) > 0.08 and live.sample_size >= 500 if high: return {"severity": "high", "owner": "risk-model-oncall", "report": report} return {"severity": "watch", "report": report} # In production, join delayed labels and link the alert to a playbook.
Without a sample-size guard, tiny windows can create noisy alerts that look severe but are not reliable enough for action.
Review AI-generated monitoring advice for missing operational pieces. It should name reference windows, slices, sample-size rules, model and prompt versions, delayed-label joins, privacy limits, thresholds, owners, playbooks, and rollback or retraining criteria.
The final skill is deciding what an alert means and which controlled action should follow.
The final skill is deciding what an alert means and which controlled action should follow.
Why this matters: Recall the monitoring loop from memory.
Decision this forces: Choose the right level of rigor for Monitoring Model Drift and Quality: baseline, production design, or advanced optimization.
Answer this from memory: "How do I monitor whether a deployed model is still useful?" Include service health, data drift, prediction drift, concept drift, delayed labels, slices, reference windows, LLM traces, privacy controls, owners, and playbooks.
Spaced recall: module 1 warned that healthy software can serve bad predictions. The final monitoring loop exists to catch that gap and choose a response with evidence.
Choose a model or LLM feature. Define one reference window, three slices, two drift signals, one delayed-label metric, version fields, privacy limits, alert thresholds, owner, and first-response playbook. Then write the exact alert message a human should receive.
Next rung: connect this to model registries and versioning, where release identity lets monitoring point to the artifact or policy that changed.
From memory, reconstruct the monitoring loop: log versioned predictions, choose reference windows and slices, watch data and prediction drift, join delayed labels, add LLM traces, and route alerts to playbooks.
Create a one-page release-ready plan for drift and quality monitor with alerts and review queues. Include: the user problem, realistic input, mechanism, design choice, runnable or reviewable check, metric (time from quality drop to triage), failure case (aggregate metrics stay flat while a high-value segment fails), owner, and the next rung after this lesson.
Your API's p99 latency is normal and error rates are flat, but a downstream team reports that your model's recommendations have felt 'off' for two weeks. What does this scenario best illustrate?
System health metrics (latency, error rate) measure infrastructure, not whether the model's outputs are still meaningful — model quality requires its own monitoring layer.
You are setting up drift monitoring for a credit-scoring model. Your aggregate PSI score looks stable, but complaints are spiking from one demographic cohort. What monitoring design mistake does this reveal?
Aggregate drift scores average across all users, hiding cohort-level failures; sliced monitoring is required to surface problems that affect subgroups disproportionately.
When monitoring an LLM-powered feature, you notice that 40 % of requests are being served from a cache. Why does cache status matter for quality monitoring?
Cache hits return pre-stored outputs, so including them in quality metrics conflates old model behavior with current model behavior and inflates or deflates your quality signal.
You are writing a monitoring alert for a fraud-detection model. Describe ONE specific privacy risk introduced by logging raw request payloads, and name ONE design choice that mitigates it.
Raw payload logging captures whatever the caller sends — often PII — which creates compliance exposure; structured feature logging or redaction keeps monitoring useful without storing sensitive data.
After an alert fires indicating prediction drift, your team investigates, retrains the model, and deploys a fix. What is the MOST important step to close the monitoring loop and prevent the same drift from going undetected next time?
Closing the loop means feeding the investigation's findings back into the monitoring system — refreshing the reference baseline and capturing the root cause — so each incident makes future monitoring smarter.