Learn why clean evaluation starts before model training.
A split creates examples the model cannot learn from directly.
Learners understand splits as evaluation design, not bookkeeping.
Why this matters: This prevents inflated metrics from accidental exposure.
Problem anchor: a team builds a ticket-priority classifier. If the same customer issue, duplicate ticket, or future pattern appears in both training and evaluation, the score can look excellent while the deployed model fails. Splitting is how we create an honest exam.
Train data fits model parameters. Validation data helps choose features, thresholds, and hyperparameters. Test data estimates final performance after choices are made.
Start with 1000 labeled tickets. Before fitting text vectorizers or models, assign 700 to train, 150 to validation, and 150 to test. If multiple tickets come from the same customer incident, keep the incident in one split so the model cannot memorize the duplicate wording.
Training learns weights; validation guides model selection.
Learners separate model fitting from model selection.
Why this matters: This prevents using the final exam as a tuning guide.
Training data is used for fitting parameters: weights, tree splits, centroids, or learned preprocessing. Validation data is used to choose decisions: model family, regularization strength, threshold, features, or prompt/reranker setting. The test set waits until those choices are locked.
The ticket classifier outputs risk scores. On validation, threshold 0.70 catches most urgent tickets but creates many false alarms; threshold 0.85 misses too many urgent cases. The team chooses 0.76 from validation because it matches support capacity. They do not keep adjusting after looking at test.
A clean test set gives a less biased final performance estimate.
Learners protect the final estimate.
Why this matters: This avoids optimistic results caused by repeated test peeking.
A test set is useful because the model-selection process has not adapted to it. If you test, tweak, test, tweak, the test set becomes part of the design loop. The reported score becomes too optimistic.
| Option | Split | Allowed uses | Forbidden uses | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Train | Train | fit model and train-only preprocessing | final performance claim | Always used for parameter fitting. | Low | Medium |
| Validation | Validation | choose hyperparameters, thresholds, features | fit final parameters directly | During model selection. | Low | Medium |
| Test | Test | one final locked-model estimate | repeated tuning or feature shopping | After choices are frozen. | Low | Medium |
Leakage makes evaluation look better than deployment reality.
Learners see split hygiene as an active check.
Why this matters: This catches the most common source of inflated beginner metrics.
train = [
{"ticket_id": 1, "customer_id": "A", "text": "login fails"},
{"ticket_id": 2, "customer_id": "B", "text": "billing issue"},
]
validation = [
{"ticket_id": 3, "customer_id": "C", "text": "upgrade question"},
]
test = [
{"ticket_id": 4, "customer_id": "A", "text": "login still fails"},
]
def ids(rows):
return {row["customer_id"] for row in rows}
overlap = {
"train_test": ids(train) & ids(test),
"train_validation": ids(train) & ids(validation),
"validation_test": ids(validation) & ids(test),
}
print({name: sorted(values) for name, values in overlap.items() if values})It prints train_test with customer A. The model could learn customer-specific patterns from train and look artificially good on test.
Checklist: split before fitting vectorizers/scalers/encoders; fit preprocessing on train only; transform validation and test with train-fitted preprocessing; group duplicates together; use time-based split for future prediction; and never tune on test.
Split strategy should reflect what deployment will ask the model to do.
Learners choose split strategies deliberately.
Why this matters: This prevents random splits from hiding dependence or future leakage.
Random splits can work when rows are independent and label balance is healthy. Stratified splits preserve rare label proportions. Grouped splits keep related records, such as one customer or patient, together. Time-based splits evaluate whether the model trained on the past works on the future.
Retrieval prompt: reconstruct clean evaluation by naming train, validation, test, model selection, leakage, stratified/group/time splits, preprocessing fit rules, and the one-time final test estimate.
Design splits for a ticket-priority classifier. Decide whether to split randomly, by customer, or by time; write which preprocessing fits only on train; define validation decisions; and reserve test for one final estimate. Next rung: add cross-validation or grouped CV.
You build a ticket-priority classifier and want an honest estimate of how it will perform on brand-new support tickets your team has never seen. Which split gives you that honest estimate?
The test set is kept untouched until the very end, so its accuracy reflects true generalization — not memorization of data the model or developer has already seen.
During development you try five different decision thresholds for flagging a ticket as 'urgent' and pick the one with the best F1 score. Which dataset should you use for this comparison?
Threshold selection is a hyperparameter decision; using the validation set for it keeps the test set clean and unbiased.
A teammate fits a TF-IDF vectorizer on the entire dataset (train + validation + test combined) before splitting. What problem does this cause, and what should they do instead?
Fitting any preprocessor on data that includes the test set leaks information into the model pipeline, making test performance look better than it really is.
Your ticket dataset has 90 % 'low-priority' tickets and only 10 % 'urgent' tickets. You do a plain random 80/20 split. What is the main risk?
With rare labels, random splits can produce folds where the minority class is severely under-represented; stratified splitting preserves the class ratio in every fold.
After weeks of tuning you check the test set, see a lower score than expected, and adjust one more hyperparameter to improve it. Why is this a problem?
Every time you make a decision based on test-set feedback, you overfit to that set; the reported score then flatters the model and cannot be trusted as a real-world estimate.