Track models, prompts, retrieval indexes, and eval sets as release artifacts.
Versioning is useful only when it covers every artifact needed to explain and reproduce a prediction.
Versioning is useful only when it covers every artifact needed to explain and reproduce a prediction.
Why this matters: List the artifacts in a complete model release bundle.
Problem anchor: a churn model starts over-discounting enterprise customers after a Friday deployment. The registry says "churn-model v42," but the team cannot identify the training data snapshot, feature code, prompt template for explanations, evaluation slice report, Docker image, or the alias switch that sent traffic to it.
A real release bundle names the model artifact, data snapshot, code commit, feature or preprocessing contract, prompt or retrieval index when relevant, eval set, metrics, package image, owner, approval, deployment alias, and rollback target. Without that, debugging becomes archaeology.
name=churn-risk, version=42, alias=canary, artifact=s3://models/churn/42, dataset=customers@2026-06-18, features=features/churn.py@abc123, image=registry/churn:42, eval_set=retention_gold_v7, metrics=auc .87, enterprise_recall .78, latency_p95=82ms, approver=ml-platform, rollback=version 41.
For an LLM feature, add prompt version, model provider or self-hosted endpoint, retrieval index version, tool schema version, and safety eval report. The principle is identical: prediction behavior comes from a bundle, not one weight file.
Training can create new artifacts freely; production should consume named versions and aliases with history.
Training can create new artifacts freely; production should consume named versions and aliases with history.
Why this matters: Explain immutable data and model references.
Experiment runs are allowed to be messy. Production references should not be. A release should point to immutable data snapshots, code commits, package tags, and registered model versions. Human-friendly aliases such as champion, staging, canary, and rollback are useful only when moving them leaves history.
DVC and lake-style tools can identify data versions; MLflow-style registries identify model versions and status. Deployment systems then resolve an alias to a version. The dangerous anti-pattern is a mutable path like models/current.pkl because it answers today and erases yesterday.
An incident team moves the model alias from v42 back to v41, but errors continue. The registry rollback worked; the hidden problem was that the feature service still served the new transformation code and the explanation prompt still used v42 assumptions. A model rollback without feature, prompt, and package rollback is only partial.
This is the spaced recall from module 1: the release was the whole bundle. Rollback must restore the bundle or explicitly prove which components are safe to keep.
| Option | Boundary | Immutable evidence | Failure if missing | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Data | Training or eval dataset | snapshot ID, table version, or DVC/lake commit | cannot explain why a retrain changed behavior | Before registering a candidate. | Medium | Medium |
| Model | Artifact and metrics | registry version, run ID, parameters, metrics | cannot compare or roll back reliably | At candidate creation. | Medium | Medium |
| Deployment | Serving alias and package | image tag, alias history, prediction logs | cannot tell which version answered users | At promotion and rollout. | Medium | Medium |
A registry records status; release gates decide whether the status is deserved.
A registry records status; release gates decide whether the status is deserved.
Why this matters: Distinguish registry status from quality control.
Decision this forces: Choose the right level of rigor for Model Registries and Versioning: baseline, production design, or advanced optimization.
A registry can say staging, production, champion, or archived. Those labels are dangerous if they are only labels. The evidence behind promotion should include aggregate metric change, slice behavior, calibration or ranking quality, latency and throughput, input/output schema compatibility, package tests, and an owner approval.
For LLM applications, the release bundle adds prompt version, retrieval index, model/provider route, tool schema, safety checks, and eval transcripts. A registry does not have to store all content directly, but it must link to immutable evidence.
Gate: overall AUC must not drop by more than .01; enterprise recall must improve or stay within .02; p95 latency under 100 ms; input schema unchanged or migration approved; package contract tests pass on 100 golden examples; prediction logs include version 42; rollback to v41 tested in staging.
The approver can now say what was accepted. If enterprise recall worsens next week, the team knows whether the risk was visible at release or emerged after traffic shifted.
A versioned release pays off when production logs can reconstruct which bundle answered each request.
A versioned release pays off when production logs can reconstruct which bundle answered each request.
Why this matters: Write a small prediction-log contract.
required = {
"request_id", "model_name", "model_version", "registry_alias",
"package_image", "input_schema", "prediction", "latency_ms"
}
def validate_prediction_log(row):
missing = required - set(row)
assert not missing, f"missing release identity: {sorted(missing)}"
assert row["registry_alias"] in {"canary", "champion", "rollback"}
return True
validate_prediction_log({
"request_id": "req_123",
"model_name": "churn-risk",
"model_version": "42",
"registry_alias": "canary",
"package_image": "registry/churn:42",
"input_schema": "churn_input_v5",
"prediction": 0.73,
"latency_ms": 81,
})You cannot reliably know which release answered a user request, so rollback verification and root-cause analysis become guesswork.
AI-generated MLOps plans often mention MLflow or a registry but skip lifecycle edges. Check whether the plan records data snapshots, code commit, package image, input/output schema, eval report, approver, alias history, prediction-log identity, and rollback procedure.
The learner should finish able to build a release card that supports promotion, incident review, and rollback.
The learner should finish able to build a release card that supports promotion, incident review, and rollback.
Why this matters: Recall the release bundle from memory.
Decision this forces: Choose the right level of rigor for Model Registries and Versioning: baseline, production design, or advanced optimization.
Answer this from memory: "What must a registry-backed model release record so I can debug and roll it back?" Include data snapshot, code commit, model artifact, package, schema, eval set, metrics, slice checks, latency, owner, alias history, prediction logs, and rollback target.
Spaced recall: a registry is not quality control. It is the place where quality evidence, artifact identity, and deployment status meet.
Choose a model or LLM feature. Fill in artifact, data, code, prompt/retrieval/tool versions if relevant, package image, eval report, slice metrics, latency, owner, alias, rollout plan, prediction-log fields, and rollback command. Then ask a teammate to answer: "Which version answered request X, and how do we roll it back?"
Next rung: connect this to monitoring drift and quality, where release identity becomes the join key for live performance changes.
From memory, reconstruct the registry-backed release: bundle data, code, model, package, prompt or retrieval versions, evals, approval, alias history, prediction identity, and rollback target.
Create a one-page release-ready plan for model registry card with version, lineage, metrics, and rollback data. Include: the user problem, realistic input, mechanism, design choice, runnable or reviewable check, metric (ability to reproduce a production answer), failure case (a rollback changes weights but leaves an incompatible prompt in place), owner, and the next rung after this lesson.
A team registers a new model version but only saves the model weights and a version number. During an incident, they cannot reproduce the exact predictions from last Tuesday. What is the most likely root cause?
A model-only registration omits the preprocessing code, data reference, and environment spec that together determine predictions — without them, exact reproduction is impossible.
Your team uses a mutable path (e.g., models/champion/latest) as the production pointer. An incident occurs and you need to know exactly which model version was serving traffic 48 hours ago. Why does the mutable path make this hard?
When a path is overwritten rather than versioned, there is no audit trail of what it pointed to at a given time — aliases over immutable version IDs preserve that history.
Which of the following best describes a promotion gate for moving a model from 'staging' to 'production' status in a registry?
Promotion gates require evidence — slice metrics, contract tests, and model-type-specific checks — not just overall accuracy or human intuition, before the alias is moved.
Name TWO fields that a prediction log entry should capture in order to tie a live prediction back to a specific registered model version during an incident review.
Logging the model version ID and the active alias (or equivalent registry pointer) lets investigators replay or audit exactly which model produced a given prediction.
You are reviewing an AI-generated registry plan for a new LLM release. The plan includes model weights, a version number, overall benchmark scores, and a 'production' alias update step. What critical gap should you flag before approving it?
A rollback-ready LLM release plan must include slice metrics, contract tests, and an explicit rollback reference — overall benchmarks alone are not sufficient promotion evidence.