A model that performed well at launch can silently degrade because the world changes — a problem called drift. Monitoring watches production inputs and…
Understand why a good model gets worse over time.
A model learns patterns from data at a moment in time and then stays fixed, but the world it operates in keeps changing — new user behavior, new products, new fraud tactics, shifting language. As reality diverges from the training data, the model's predictions get quietly worse. This decay is called drift.
The danger is that it is silent: the model keeps returning confident outputs, so nothing obviously breaks. Accuracy erodes gradually, and without monitoring you find out from complaints or lost revenue rather than from a signal you could have caught.
Distinguish the two kinds of drift and why the difference matters.
Data drift is when the inputs change distribution: your traffic now looks different from the training data — new demographics, new phrasing, new value ranges. The relationship may still hold, but the model sees inputs it wasn't trained for.
Concept drift is deeper: the relationship between inputs and the correct output changes. What counted as fraud last year, or spam, or a good recommendation, has shifted, so even familiar inputs now map to different correct answers. Data drift often signals a coming accuracy drop; concept drift means the model's learned logic is now partly wrong. Both usually call for retraining on fresh data, but concept drift is the more urgent.
Track the right signals and alert before users notice.
Monitor at three levels. Inputs: track the distribution of incoming features and flag when it drifts from training. Outputs: watch prediction distributions and confidence for sudden shifts. Outcomes: where you can get ground truth (a click, a repayment, a correction), measure real accuracy over time — this is the truest signal but often arrives with a delay.
Set alerts on meaningful changes, not noise, so a genuine drift or accuracy drop pages someone. Also monitor the operational basics — latency, error rate, and cost — since those fail loudly and matter to users.
True accuracy needs the real outcome, which can lag by days or weeks. So drift detection on inputs and outputs acts as an early warning while you wait for ground truth to confirm a real quality change.
Adapt monitoring to LLM apps and avoid the common errors.
LLM apps need their own monitoring because there's rarely a single correct label. Track response quality with sampled evaluations (including LLM-as-a-judge), watch for hallucination and policy violations via output guardrails, and collect user feedback signals like thumbs-up or edits. Watch token cost and latency too, since prompt or traffic changes can spike them.
Close the loop: feed failures and low-rated responses back into your evaluation set and prompt or model iteration. Monitoring is only useful if it drives a fix — a retrain, a prompt change, or a rollback.
Watch for: monitoring only uptime and latency while quality silently drifts; alerting on so much noise that real signals are ignored; having no feedback loop, so you detect problems but never fix them; and assuming a model is 'done' at deployment. Plan for retraining and re-evaluation as a routine, not an emergency.
Models degrade after deployment because they're fixed while the world changes — drift. Data drift is a shifted input distribution; concept drift is a changed input-to-output relationship, and the more urgent. Monitor inputs, outputs, and (delayed) outcomes, alerting on meaningful change, plus latency, errors, and cost. LLM apps also need sampled quality evals, hallucination and policy guardrails, and user feedback. Crucially, close the loop: monitoring must drive a retrain, prompt change, or rollback.
A fraud model quietly loses accuracy over months. Describe what you would monitor to catch it early given that true fraud labels arrive weeks later, how you'd tell data drift from concept drift, and how monitoring would trigger a retrain.
Why do models degrade after deployment?
Drift erodes accuracy quietly because the model keeps producing confident outputs even as its training data becomes outdated.
What is the difference between data drift and concept drift?
Data drift means new-looking inputs; concept drift means the right answer for given inputs has changed, making the model's logic partly wrong.
Why is drift detection on inputs and outputs useful even before you have ground truth?
Real accuracy needs delayed outcomes, so monitoring distributions gives early signals while you wait to confirm a true change.
What is important when monitoring an LLM app specifically?
LLMs lack a single correct label, so monitoring samples quality and safety and feeds failures back into evaluation and iteration.