Diagnose when a model memorizes, underfits, or generalizes.
You trace the gap between training performance and real-world performance, and establish train/validation/test splits as the measurement framework everything else depends on.
Establishes why training accuracy is not a reliable signal and introduces the train/validation/test split as the measurement framework for generalization.
Why this matters: Every model-quality decision you make — tuning, comparing, shipping — depends on measuring the right thing on the right data; this module gives you that foundation.
Your model hits 99% accuracy on training data. Should you ship it? Not until you know whether it learned a pattern or just memorized the examples.
Training metrics can't tell the difference. The goal of machine learning is : performing well on unseen data. The — the gap between training and held-out performance — is the single number that matters.
A large gap means : the model training examples and fails on new data. High training loss with a small gap means : the model missed the pattern entirely.
Bias, variance, regularization, and early stopping are all tools for controlling that gap. First, you must measure it.
You measure the generalization gap by holding data back from training and evaluating on it separately. The standard framework uses three non-overlapping splits, each answering a different question.
A team trains a spam classifier on 10,000 emails. Training accuracy: 98.5%. They ship it. Users report obvious spam getting through.
Post-mortem: the training set came from one company's inbox over six months. The model learned that "invoice" emails are safe — because that company's invoices were legitimate. On the broader internet, "invoice" appears constantly in phishing emails. The model memorized a company-specific quirk, not a general signal.
A test set from a different time window and senders would have caught this. It wasn't used before launch. The 98.5% figure came from training data, making it misleading: technically correct, practically useless.
The fix: always evaluate on a test set that reflects the distribution the model will face — different time, different source, or adversarial examples.
import numpy as np # All 1000 examples used for both training AND evaluation X, y = load_spam_dataset() # 1000 emails model = DecisionTreeClassifier(max_depth=None) model.fit(X, y) acc = model.score(X, y) # scored on the SAME data print(f"Accuracy: {acc:.2%}") # → Accuracy: 100.00%
max_depth=Nonemodel.score(X, y)This is the mistake that ships bad models: training and evaluating on the same data. Before reading the reveal, predict what the output will be and why it tells you nothing useful.
Output: Accuracy: 100.00%
It proves nothing. An unlimited-depth decision tree memorizes every training example perfectly — it has essentially stored a lookup table. Score on the training set measures memorization, not generalization. You'd see the same 100% even if the model fails on every new email it has never seen.
from sklearn.model_selection import train_test_split X_train, X_temp, y_train, y_temp = train_test_split( X, y, test_size=0.20, random_state=42 ) X_val, X_test, y_val, y_test = train_test_split( X_temp, y_temp, test_size=0.50, random_state=42 ) model.fit(X_train, y_train) print(f"Train : {model.score(X_train, y_train):.2%}") # → 100.00% print(f"Val : {model.score(X_val, y_val ):.2%}") # → 71.40%
train_test_split(..., test_size=0.20)train_test_split(X_temp, ..., test_size=0.50)random_state=42Now the same unlimited-depth tree is scored on data it never trained on. The gap between 100% and 71% is the made visible — and it tells you the model has badly overfit.
Much lower — around 71% here. The tree memorized the 800 training emails, including noise and company-specific quirks. On the 100 validation emails it has never seen, those memorized patterns don't transfer. The 29-point gap is the generalization gap in action.
# Same 80/10/10 split from Stage 2 — reuse X_train, X_val, y_train, y_val # TODO: create a DecisionTreeClassifier with max_depth=5 # (this is the key change — why does limiting depth reduce the gap?) model_limited = ??? model_limited.fit(X_train, y_train) print(f"Train : {model_limited.score(X_train, y_train):.2%}") print(f"Val : {model_limited.score(X_val, y_val ):.2%}") # Hint 1: the only change from Stage 2 is the max_depth argument. # Hint 2: a shallower tree can't memorize every example — what tradeoff does that create?
max_depth=5model_limited.score(X_val, y_val)Stop — attempt the TODO before revealing. The gap you saw in Stage 2 was caused by unlimited depth; this stage asks you to constrain it and predict what happens to both scores.
Answer: DecisionTreeClassifier(max_depth=5)
Changed line: max_depth=None → max_depth=5
Why it matters: depth=5 limits the tree to 32 leaf nodes, preventing it from memorizing individual examples.
Expected output (approximate):
Train : 89.50% ← dropped from 100% — the model no longer memorizes
Val : 84.20% ← rose from 71% — the gap shrank from 29 pts to ~5 pts
The tradeoff: training accuracy fell, but validation accuracy rose — a smaller generalization gap means better real-world performance. This is the core tension the rest of the lesson is about.
Drag to see how the training fraction changes what each split can tell you. There is no universally correct split — it depends on dataset size and model complexity.
Three failure modes produce misleading accuracy numbers. Each looks fine until it doesn't.
When you see an accuracy number, ask: which split? Was preprocessing fitted before or after the split? Is the data time-ordered? If any answer is unclear, the number is suspect.
fit() is called only on training data, the test set is scored exactly once at the end, and time-series data is split by index — not shuffled. These three checks catch most leakage bugs.With a reliable measurement framework in place, the next question is: what causes the generalization gap? The next module breaks the gap into two competing forces — and — and shows how they pull in opposite directions as model complexity changes.
You decompose total prediction error into bias, variance, and irreducible noise, and see how they pull in opposite directions as model complexity changes.
Decomposes prediction error into bias², variance, and irreducible noise, and shows how they shift as model complexity changes.
Why this matters: Understanding these three error sources tells you exactly why your model fails and which lever — simplify or complexify — will actually help.
The three splits are the (what the model learns from), the (what you tune decisions against), and the (the held-out final verdict).
A gap between training and validation error signals a problem. This module asks: what causes that gap, and can we decompose it into named parts?
Every prediction error on unseen data comes from three sources: , , and .
Bias is systematic error from a model too simple to capture the true pattern. It errs in the same direction every time. Variance is sensitivity error from a model too complex. It chases training data's random fluctuations and lands differently on each new dataset. Irreducible noise is randomness baked into the data itself. No model can remove it.
The formal decomposition is: Expected Test Error = Bias² + Variance + Noise. Bias and variance are the two levers you control.
As you increase — more parameters, higher-degree polynomials, deeper trees — bias falls and variance rises. This is the : you cannot reduce both at once with complexity alone.
On a test-error-vs-complexity plot, this produces a U-shape. Error is high on the left (high bias). It dips to a minimum at the sweet spot. Then it climbs on the right (high variance).
Imagine you are predicting house prices. You train three models on the same dataset and evaluate each on a held-out validation set.
Notice that noise is constant across all three — the same dataset, the same measurement error in the prices. Only bias and variance shift as you change complexity.
import numpy as np def bias_variance_estimate(model_class, X_train, y_train, X_test, y_test, n_boots=50): predictions = [] for _ in range(n_boots): idx = np.random.choice(len(X_train), len(X_train), replace=True) m = model_class().fit(X_train[idx], y_train[idx]) predictions.append(m.predict(X_test)) preds = np.array(predictions) # shape: (n_boots, n_test) mean_pred = preds.mean(axis=0) bias_sq = np.mean((mean_pred - y_test) ** 2) variance = np.mean(preds.var(axis=0)) return bias_sq, variance
np.random.choice(..., replace=True)preds.mean(axis=0)(mean_pred - y_test) ** 2preds.var(axis=0)This function estimates bias² and variance via bootstrap resampling — the standard empirical approach when you can't average over infinite datasets.
Each bootstrap iteration trains on a resampled version of the training set and records predictions on the fixed test set. After 50 iterations, the mean prediction approximates the model's expected output, and the spread around that mean is the variance.
bias_sq will be HIGH — the mean prediction is far from most true labels. variance will be near ZERO — the model always outputs the same number regardless of which bootstrap sample it sees. This matches Model A in the functional example.
# Continuing from Stage 1 — bias_sq and variance are already computed. # y_test_noisy has measurement noise added; y_test_true is the clean signal. def total_expected_error(bias_sq, variance, y_test_true, y_test_noisy): noise = np.mean((y_test_noisy - y_test_true) ** 2) # TODO: return the three-term decomposition as a dict # keys: 'bias_sq', 'variance', 'noise', 'total' # Hint: total = bias_sq + variance + noise pass
(y_test_noisy - y_test_true) ** 2passStop — attempt the TODO before revealing. The function already computes noise; your job is to assemble the final decomposition and return it.
The crux is recognizing that total expected error is exactly the sum of the three terms — not an approximation, but the identity from the decomposition formula.
return {'bias_sq': bias_sq, 'variance': variance, 'noise': noise, 'total': bias_sq + variance + noise}
Changed lines: only the return statement is new — Stage 1 computed bias_sq and variance; this stage adds noise (already computed above the TODO) and assembles all three into the decomposition identity. The key insight: total is not just train error or val error — it is the sum of all three named sources.
Drag the slider to see how bias and variance shift as model complexity increases. The U-shape emerges from their sum.
The next module zooms into the right side of the U-curve. That is where high variance produces the train-vs-validation gap that exposes overfitting.
You examine how an overly complex model memorizes training examples, see the train-vs-validation gap that exposes it, and work through a completion example spotting overfitting in a real loss curve.
This module shows how an overly complex model memorizes its training data instead of learning the underlying pattern, and how to detect it from a diverging train/validation loss curve.
Why this matters: Overfitting is the most common reason a model that looks great in training fails in production — spotting it early saves wasted compute and bad deployments.
High means predictions shift dramatically when training data changes. The model fits noise, not the true signal.
That instability is exactly what looks like: a model that aces training data but stumbles on new data.
This module zooms in on the mechanism — how a model memorizes rather than learns — and exposes the diagnostic signal.
Your model hit 99% training accuracy, but real users see wrong predictions. What went wrong?
When a model has too much relative to training set size, it stores examples instead of learning patterns. This is : fitting quirks, noise, and sampling accidents.
The result is : near-zero training loss, poor to new data. The model learned a lookup table, not a function.
Fix it by reducing capacity, adding data, or applying . But first, detect it reliably.
The clearest diagnostic is the : training loss and loss plotted against training epochs.
In a healthy run both curves fall together and level off close to each other. Overfitting shows a specific shape: training loss keeps falling while validation loss flattens and then rises — the two curves diverge.
That gap is the made visible. The epoch where validation loss starts climbing is the natural stopping point for .
You train a deep neural network to predict house sale prices. After 60 epochs you check the loss history and see this pattern:
The model has memorized the 800 training houses — their quirks, data-entry errors, and local anomalies — rather than learning the price-driving features.
The opens at epoch 20. Saving the model at epoch 20 (via ) would have preserved the best performance.
history = {"train_loss": [], "val_loss": []}
for epoch in range(1, num_epochs + 1):
train_loss = run_epoch(model, train_loader, optimizer)
val_loss = evaluate(model, val_loader)
history["train_loss"].append(train_loss)
history["val_loss"].append(val_loss)
print(history["train_loss"][-1], history["val_loss"][-1])run_epoch(model, train_loader, optimizer)evaluate(model, val_loader)history["val_loss"].append(val_loss)This loop records both losses every epoch so you can compare them later. The final print shows the last-epoch gap — if val_loss >> train_loss, overfitting has occurred.
Train loss (0.01) is near zero while val loss (0.78) is 78× higher — a massive generalization gap. The model has memorized the training set and will perform poorly on new houses. You should roll back to the checkpoint where val loss was lowest (around epoch 20 in the scenario above).
best_val_loss = float("inf") best_weights = None patience, strikes = 5, 0 for epoch in range(1, num_epochs + 1): train_loss = run_epoch(model, train_loader, optimizer) val_loss = evaluate(model, val_loader) if val_loss < best_val_loss: best_val_loss = val_loss best_weights = copy_weights(model) # save best checkpoint strikes = 0 else: strikes += 1 if strikes >= patience: break # TODO: restore best_weights here model = restore_weights(model, best_weights)
float("inf")patience, strikes = 5, 0copy_weights(model)breakThis extends Stage 1 with a patience counter that halts training when val loss stops improving. The TODO is the crux: what should happen the moment patience runs out, before the loop exits?
patience epochs of overfitting. Hint 2: you already have the right variable stored above the loop.Replace the TODO with: model = restore_weights(model, best_weights)
Changed lines vs Stage 1: added the patience/strikes counter and this restore call inside the if strikes >= patience branch.
Why inside the branch: you want to restore the best checkpoint the instant patience expires, then break — if you only restore after the loop, a KeyboardInterrupt or a loop that exhausts num_epochs without triggering patience would skip the restore entirely. Restoring inside the branch is the safe, deliberate act.
Drag to a training epoch and read what the train and validation losses are doing. Notice when the gap opens.
When reviewing AI-generated training loops, check: does it log val loss separately, save the best checkpoint (not the last), and guard against val-set leakage during hyperparameter search?
Next: the opposite failure — a model too simple to fit training data. Module 4 tackles underfitting.
You examine how a model with insufficient capacity fails to capture the true relationship, producing high error on both training and validation sets, and complete a diagnosis exercise distinguishing it from overfitting.
Examines underfitting — when a model is too simple to capture the true relationship — and teaches you to diagnose it from train and validation error.
Why this matters: Knowing the underfitting signature lets you distinguish the right fix (more model capacity) from the wrong one (more data or regularization), saving wasted experiment cycles.
Answer: training error is low and validation error is high — the is wide. That gap is the fingerprint of a model that memorized instead of learned.
This module flips the picture: what happens when both numbers stay high? That's the failure mode we haven't diagnosed yet.
occurs when a model lacks capacity to capture the true data relationship. It makes systematic errors on training and new data alike.
The root cause is high : the model's assumptions are too rigid. It consistently misses the signal regardless of which examples it sees. Adding training data won't help — the model's form is the problem.
The diagnostic signature is simple: training error and validation error are both high, and they sit close together. The model fails equally on seen and unseen data.
The fix is to increase : add features, use a more expressive architecture, or reduce constraints.
From Module 2: total error = bias² + variance + irreducible noise. Underfitting sits at the high- end of the . The model is so constrained that predictions are systematically wrong. stays low.
Low variance means the model is consistently wrong in the same direction. A precise miss is still a miss.
You're training a model to predict housing prices from square footage alone. After 50 epochs, you read off these numbers:
The gap between train and val is tiny (~0.02), so this is not overfitting. Both errors are high and plateau together — the model has hit its capacity ceiling. A linear model predicting price from one feature can't capture the non-linear relationship in the data.
The fix is not more data or stronger — it's more expressive features (e.g., add neighbourhood, age, floor area²) or a more flexible model class.
def diagnose(train_err, val_err, threshold=0.20): gap = val_err - train_err both_high = train_err > threshold and val_err > threshold if both_high and gap < 0.05: return "Underfitting — high bias; increase model complexity" elif gap >= 0.10: return "Overfitting — high variance; regularize or get more data" else: # TODO: return the label for a well-fitted model pass print(diagnose(0.42, 0.44)) # → Underfitting print(diagnose(0.05, 0.31)) # → Overfitting
gap = val_err - train_errboth_high = train_err > threshold and val_err > thresholdgap < 0.05gap >= 0.10passThis function encodes the two-number diagnostic: check whether both errors are high (underfitting) or whether the gap is large (overfitting), and infer the remaining case.
The TODO is the crux — filling it in forces you to articulate what the absence of both failure modes actually means.
return "Good fit — low bias, low variance"
# Changed line: line 9 — the else branch now returns a positive label
# instead of pass. This is the crux: you must recognise that small gap +
# both errors low is the only remaining case after ruling out the two
# failure modes. The threshold and gap checks above do all the heavy
# lifting; your job is to name what's left.
Drag to see how training and validation error move together as model complexity increases from too-simple (underfit) to too-complex (overfit).
Quick check: if doubling training data leaves both errors unchanged, the bottleneck is model capacity, not data quantity.
You can now name both failure modes from a pair of numbers. In practice you rarely get a single snapshot — you get a curve evolving over epochs or dataset sizes.
Module 5 shows how to plot and read to locate the failure mode precisely. It covers for reliable error estimates when a single train/val split is too noisy.
You plot and interpret learning curves to locate the failure mode, then use k-fold cross-validation to get a reliable estimate of generalization error — revisiting the train/validation split logic from Module 1 under real data constraints.
Decision this forces: Single train/val split vs. k-fold cross-validation: which gives a reliable enough estimate for your data size?
Both errors high → (high ). Training low, validation high → (high ). Module 4 labeled these patterns. This module teaches you to measure them.
A plots training error and validation error against the number of training examples seen. As you add data, the curves converge — or fail to — in ways that reveal the failure mode.
The shape of the gap — not just its size — tells you which lever to pull next.
You're training a house-price regression model and plot learning curves after each data increment. At 200 examples, training RMSE is 18k and validation RMSE is 52k — a 34k gap. At 800 examples, training RMSE is 22k and validation RMSE is 44k — the gap shrank to 22k but validation is still high.
The gap is narrowing with data, which is the high-variance fingerprint: the model is memorizing, and more data is helping — but slowly. If you instead saw both errors stuck at 45k regardless of data size, that's the high-bias fingerprint: adding data won't fix a model that's too simple.
Drag to see how the train/validation gap evolves as training set size grows. A persistent wide gap signals high variance; a gap that closes but stays high signals high bias.
A single gives one error estimate whose value depends heavily on which examples landed in which set. With small datasets, that variance in the estimate itself can mislead you.
K-fold partitions the data into k equal folds, trains k times (each time holding out one fold as validation), and averages the k error scores. The mean gives a lower-variance estimate of error; the standard deviation across folds tells you how stable that estimate is.
The cost is k times the compute. For large datasets a single split is usually fine; for small datasets (< ~5k examples) k-fold is worth it.
import numpy as np # Stage 1: manual k-fold split (k=5) def kfold_indices(n, k=5, seed=42): rng = np.random.default_rng(seed) idx = rng.permutation(n) return [idx[i * n // k : (i + 1) * n // k] for i in range(k)] # Stage 2: evaluate model across folds def cross_val_score(model, X, y, k=5): folds = kfold_indices(len(X), k) scores = [] for i, val_idx in enumerate(folds): train_idx = np.concatenate([folds[j] for j in range(k) if j != i]) model.fit(X[train_idx], y[train_idx]) scores.append(model.score(X[val_idx], y[val_idx])) return np.mean(scores), np.std(scores)
rng.permutation(n)np.concatenate([folds[j] for j in range(k) if j != i])model.fit / model.scorenp.mean(scores), np.std(scores)Stage 1 builds the fold index lists; Stage 2 loops over them, trains on k-1 folds, scores on the held-out fold, and returns the mean ± std. The std is as important as the mean — a high std means your estimate is unstable, often because the dataset is small or the split is unlucky.
0.09 is the standard deviation of scores across the 5 folds. A std of 0.09 on a mean of 0.81 is large (~11% relative) — the estimate is noisy. You should not fully trust 0.81 as a stable generalization score; try more folds or collect more data before making model decisions.
import numpy as np def learning_curve(model, X, y, train_sizes, k=5): train_errors, val_errors = [], [] for size in train_sizes: X_sub, y_sub = X[:size], y[:size] folds = kfold_indices(len(X_sub), k) fold_train, fold_val = [], [] for i, val_idx in enumerate(folds): train_idx = np.concatenate( [folds[j] for j in range(k) if j != i] ) model.fit(X_sub[train_idx], y_sub[train_idx]) fold_train.append(model.score(X_sub[train_idx], y_sub[train_idx])) # TODO: append the validation score for this fold # Hint 1: use model.score on the held-out fold indices # Hint 2: mirror the fold_train.append line above train_errors.append(np.mean(fold_train)) val_errors.append(np.mean(fold_val)) return train_errors, val_errors
train_sizesX[:size], y[:size]fold_train / fold_valStop — attempt the TODO before revealing. This function wraps k-fold CV inside a loop over training sizes to produce a full learning curve. The missing line is the crux: without it, val_errors stays empty and the curve is blind.
fold_val.append(model.score(X_sub[val_idx], y_sub[val_idx]))
--- Changed line vs. the worked example ---
Only this one line is new. It evaluates the model on the held-out fold (val_idx), not the training fold — that's the whole point of cross-validation: the score must come from data the model never trained on in this iteration.
You map each diagnosed failure mode to its targeted fix — regularization and pruning for overfitting, capacity increases for underfitting — and solo-build a diagnostic-and-fix decision for a given model's learning curve.
Maps each diagnosed failure mode — overfitting or underfitting — to its targeted fix, and builds the judgment to avoid applying the wrong lever.
Why this matters: Diagnosis without a fix is useless; this module closes the loop so you can act on a learning curve, not just read it.
Decision this forces: Given a diagnosed failure mode, which lever — regularization strength, model capacity, or data volume — is the right first move?
. Module 5 gave you the diagnosis; this module gives you the fix.
The question this module forces: once you know which failure mode you have, which lever do you pull first — regularization strength, model capacity, or data volume?
Each failure mode has a distinct cause. So each has a distinct class of fixes.
. Regularization trades some variance for higher bias. Capacity increases do the opposite.
The mismatch trap is applying an overfitting fix to an underfitting model. Or vice versa. The result is worse performance, while you assume the fix just needs more tuning.
You trained a neural network to predict customer churn. After 50 epochs the learning curve shows: training loss = 0.12, validation loss = 0.41, and the gap has been widening since epoch 20.
. The model has memorized training examples rather than learning the underlying pattern.
to the dense layers.
— exactly the trade you want when variance is the dominant error source.
# Running example: churn prediction neural network # Diagnosis: overfitting (train=0.12, val=0.41, gap widens at epoch 20) best_val_loss = float('inf') patience_counter = 0 PATIENCE = 5 for epoch in range(100): train_loss = train_one_epoch(model, train_loader, optimizer) val_loss = evaluate(model, val_loader) # TODO: implement early stopping here. # Hint 1: compare val_loss to best_val_loss. # Hint 2: if no improvement for PATIENCE epochs, break. ???
float('inf')patience_countersave_checkpoint(model)breakThis is the churn model's training loop with early stopping stubbed out — your job is to fill in the TODO.
Stop — attempt the TODO before revealing. The missing block is the crux of early stopping: tracking improvement and deciding when to halt.
# CHANGED LINES — this is the early-stopping block:
if val_loss < best_val_loss:
best_val_loss = val_loss # update the best seen so far
patience_counter = 0 # reset: we're still improving
save_checkpoint(model) # keep the best weights
else:
patience_counter += 1 # no improvement this epoch
if patience_counter >= PATIENCE:
print(f'Early stop at epoch {epoch}')
break
# Why: the counter accumulates consecutive non-improving epochs.
# After PATIENCE=5 such epochs, training halts and the best checkpoint is used.
| Option | Bias effect (does it reduce bias?) | Variance effect (does it reduce variance?) | Data requirement | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| L1 / L2 Regularization | Increases bias slightly | Reduces variance directly | Works on small datasets | Validation loss >> training loss; model is shallow or linear; you want interpretability (L1) or smooth shrinkage (L2). | Negligible compute overhead | Low — one hyperparameter (λ) to tune |
| Dropout | Slight bias increase | Strong variance reduction | Helps most with moderate data | Neural network is overfitting; validation gap appears after several epochs; model has many parameters relative to data. | Slight training slowdown; zero inference cost if disabled at test time | Low — one rate hyperparameter per layer |
| Early Stopping | Minimal effect on bias | Prevents variance from growing | Requires a held-out validation set | Validation loss starts rising while training loss keeps falling; you want the simplest possible intervention with no extra hyperparameters. | Saves compute by stopping early | Very low — just a patience counter |
| Increase Model Capacity | Directly reduces bias | Increases variance risk | More capacity needs more data to avoid overfitting | Both training and validation loss are high and plateau together; model cannot fit even the training data well. | Higher training and inference cost | Medium — architecture search or feature engineering |
| Collect More Data | Little effect on bias | Reduces variance reliably | Is the data — no constraint here | Overfitting is confirmed and regularization alone isn't closing the gap; or capacity was increased and variance spiked. | Can be expensive or slow | High — labeling, collection, cleaning |
Drag to see how increasing λ shifts the bias-variance balance. The sweet spot is where validation error is lowest.
The most common mistake is applying the wrong fix confidently. Here are three patterns to watch for.
after applying a fix. If the failure mode signature changed, your fix worked, even partially. If it didn't change, you likely mismatched fix to failure.
You now have the full diagnostic-and-fix loop. The capstone challenge asks you to run it solo on an unseen learning curve. Diagnose the failure mode. Select the fix. Predict the bias-variance shift. Justify why the alternative fix would be a mismatch.
Before reading the summary: from memory, sketch the U-shaped error curve — label where bias dominates, where variance dominates, and the sweet spot. Then name the one learning-curve signature that tells you which regime you're in, and the first fix you'd reach for in each case.
Apply what you learned to Overfitting, Underfitting, and the Bias-Variance Tradeoff.
You train a model and get 99% accuracy on the training set. Which conclusion is safest before you see validation or test results?
The correct answer is that training accuracy only measures fit to seen data, not performance on new cases. The first option assumes seen-data performance transfers automatically, which is the generalization mistake; the third overstates the evidence because high training accuracy alone does not prove overfitting without validation behavior; the fourth removes the split meant to measure final unseen-data performance.
As model complexity increases from very simple to very flexible, what usually happens to bias² and variance?
The correct answer is that more flexible models can capture more structure, lowering bias², but they also react more to the particular training sample, raising variance. The second option reverses the usual tradeoff; the third ignores why test error often rises again at high complexity; the fourth is wrong because irreducible noise is part of the data-generating process, not something model complexity can eliminate.
A training run shows this pattern after many epochs: training loss keeps decreasing toward zero, while validation loss decreases at first and then starts increasing. What diagnosis best fits?
The correct answer is overfitting: the widening train-validation gap shows the model is specializing to the training data instead of generalizing. The underfitting option conflicts with the near-zero training loss; the well-generalized option ignores the validation loss reversal; the high-bias option focuses only on validation being higher, but high bias would usually keep both training and validation errors high.
You have a small dataset, and a single train/validation split gives a validation accuracy that changes a lot when you change the random seed. What should you use if your goal is a more reliable performance estimate?
The correct answer is k-fold cross-validation because it reduces estimate variance by evaluating across several train/validation partitions. Reporting the best seed is biased cherry-picking; repeatedly using the test set leaks final-evaluation information into model selection; stronger regularization may be a fix for diagnosed overfitting, but split instability alone is about estimate reliability, especially with small data.
A model has high training error and high validation error, and adding more training examples barely changes either curve. What failure mode is most likely, and what is a good first fix?
The right diagnosis is underfitting/high bias because the model performs poorly even on the training data, so it is too constrained to capture the pattern. More of the same data is not the best first move when the learning curve has plateaued with both errors high; overfitting-focused fixes like stronger regularization or dropout would usually make the high-bias problem worse.