Learn why clean evaluation starts before model training.
You discover why a model trained and tested on the same data always looks better than it really is, using a student-exam analogy that runs through the whole lesson. You leave with a clear mental picture of the evaluation problem splitting is designed to solve.
This module explains why training and testing a model on the same data always produces a falsely high score — and why splitting data into separate groups is the fix.
Why this matters: Every model you build needs an honest performance score; without splitting, you can't tell if your model has actually learned anything useful.
You train a model and check performance on the same data it learned from. The score looks great. But on new data, it fails. Why?
Models find patterns in examples. Testing on those same examples lets it replay memorized answers — no real understanding needed.
This is the core problem: training-data scores are always too optimistic. They measure memory, not ability.
A model that has memorized its training data gives perfect answers on those examples but stumbles on anything new.
A model that has genuinely learned picks up the underlying pattern — and applies it correctly to examples it has never seen before.
The technical name for memorizing instead of learning is . The goal is — performing well on data the model has never seen.
Meet Alex, a student preparing for a history exam. Alex's teacher gives out a practice set of 100 questions to study from.
On exam day, the teacher — by mistake — uses those exact same 100 questions as the real test. Alex scores 98%. The teacher is thrilled.
The next week, a different teacher gives Alex a fresh history test with new questions on the same topics. Alex scores 54%.
The 98% measured memorization. The 54% measured actual knowledge. A model trained and tested on the same data is Alex taking the exam early — the score is real, but it means nothing.
The naive approach: collect data, train on all of it, measure accuracy on that same dataset.
The model learned quirks of those specific 1,000 emails — not the pattern separating spam from real mail. The 97% score was false.
Even when you know the rule, the same-data trap shows up in disguised forms. Here are the three most common ones.
You train on the full dataset and run a quick accuracy check on it. The number looks fine, so you move on. You never see the real performance — because you never tested on unseen data.
You adjust settings (called ) repeatedly until the score improves. But if you're measuring on training data each time, you're just chasing memorization, not real improvement.
A spam filter trained on last year's emails memorizes last year's spam tricks. New spam arrives with different wording. The model never learned the pattern — it learned the examples. Accuracy collapses.
Slide to see how a model's behaviour shifts as it memorizes more of its training data instead of learning the underlying pattern.
Keep some data hidden from the model during training. Use that hidden data to measure real performance.
A fair exam uses questions never seen before. Practice and real exam stay separate. Data splitting does exactly that.
Divide your data into separate groups. The model trains on one, gets evaluated on another it never touched.
This score reflects : how well the model performs on new data in the real world.
You know why splitting is necessary. One hidden set isn't enough — you need three separate groups, each with a distinct job.
The fits the model. The guides tuning. The gives the final, trustworthy score.
You map out exactly what each of the three sets does: the training set fits the model, the validation set guides tuning decisions, and the test set gives the final unbiased verdict. Using the student-exam scenario, you see how mixing any two roles corrupts the result.
Maps the three data splits — training, validation, and test — to their distinct jobs and shows why mixing them corrupts results.
Why this matters: Getting the splits right is the foundation of every trustworthy model evaluation; mistakes here make every accuracy number you report meaningless.
Decision this forces: When does a dataset need all three splits versus just two?
Module 1 showed that a model trained and tested on the same data looks better than it really is. The model memorises the answers instead of learning the pattern — that's . The fix is to keep separate data for each job.
This module maps out exactly what those jobs are and who does them. The driving question: if you have three separate sets, what is each one actually for?
Every split has exactly one job, and mixing jobs corrupts the result. The is the only data the model learns from — it adjusts its internal numbers here.
The is used to make tuning decisions — things like how many layers to use or how fast to learn. These tuning knobs are called (settings you choose before training starts).
The is locked away until the very end. It gives the final, unbiased — a verdict the model has never influenced.
Follow one student — and one model — through the three-set process to see each job in action.
The student reads chapters 1–7 of the textbook (70 % of all questions). The model sees 70 % of the labelled data and adjusts its parameters to fit those examples. At this point the model only knows what it has studied.
The student takes a practice exam (15 % of questions, never seen before). They score 60 % and decide to spend more time on chapter 5. The model checks its error on the validation set and the builder adjusts hyperparameters — maybe a lower learning rate. This loop repeats until the validation score stops improving.
On exam day the student opens a paper they have never seen (the final 15 %). The model is evaluated on the held-out test set — once, with no changes allowed after. This score measures real : how well the model performs on data it has never touched.
Imagine the student peeks at the sealed exam after the practice paper. They study the exact questions they saw, sit the exam, and score 95 %. That score no longer measures what they know — it measures how well they memorised the exam.
The same thing happens in ML. Every time you look at the test score and adjust your model, the test set leaks information into your decisions. After enough peeks, the test set has become a second — and you have no clean, unbiased verdict left.
The rule is simple: touch the test set exactly once, after all tuning is finished.
Stop — attempt this before revealing the answer. Read the scenario below and decide which split each action belongs to.
You now have the mental map: three sets, three jobs, no sharing. The next module asks a harder question: what if information from the test set sneaks into training before you even split the data? That invisible contamination is called , and it is the topic of the next module.
Drag to see how changing the training share reshapes the validation and test portions. A 70/15/15 split is a common starting point.
Three failure patterns show up most often — here is what each one looks like.
You trace exactly how data leakage happens — through preprocessing done before splitting, through target encoding, and through time-ordered data shuffled incorrectly — and why it produces a model that looks accurate but fails in the real world. This module revisits the 'honest evaluation' idea from Module 1 to show what breaks when it's violated.
This module shows exactly how data leakage enters a machine learning pipeline — through preprocessing done too early, through features that encode the answer, and through time-ordered data shuffled incorrectly.
Why this matters: Leakage is the most common reason a model looks great in development but fails in the real world — spotting it protects every evaluation you run.
Each set has one job: the fits the model. The guides tuning. The gives the final unbiased verdict — used once, at the very end.
That separation only works if the sets are truly independent. This module shows how that independence breaks through — and why the damage is invisible until too late.
The driving question: how does information from the future (or held-out data) sneak into training without you noticing?
happens when information that would not be available at prediction time gets used during training. The model learns a shortcut that only exists in your dataset, not in the real world.
The cruel part: leakage makes your model look better on paper. Validation scores are suspiciously high, and you feel confident — right up until the model meets real data and collapses.
Leakage breaks — the core idea from Module 1. When the test conditions are contaminated, the score no longer measures how well the model ; it measures how well it cheated.
Leakage enters through three main doors, each easy to miss.
Each route produces the same symptom: flattering validation scores and disappointing real-world scores.
Imagine you're building a model to predict house prices. Your dataset has 10,000 houses with features like square footage, neighbourhood, and age.
You run a scaler (a tool that rescales numbers to a common range) on all 10,000 rows to normalise square footage. Then you split into train/validation/test. Problem: the scaler used the minimum and maximum price from every row, including the 2,000 rows you set aside as validation and test. Those rows secretly shaped the scaler — they're no longer unseen.
You add a feature called "price_per_sqft" computed from the sale price. But sale price is the target — you're handing the model the answer disguised as a feature. Accuracy soars in training; in production, that feature doesn't exist yet.
Your data spans 2018–2023. You shuffle all rows randomly, so some 2023 sales land in the training set and some 2018 sales land in the test set. The model learns from future market conditions to predict past ones — a trick that's impossible in real deployment.
# WRONG: scaler sees all rows before the split scaler = StandardScaler() X_scaled = scaler.fit_transform(X_all) # ← leakage happens here X_train, X_test, y_train, y_test = train_test_split( X_scaled, y_all, test_size=0.2 ) model.fit(X_train, y_train) print(model.score(X_test, y_test)) # inflated — test rows shaped the scaler
scaler.fit_transform(X_all)train_test_split(...)model.score(X_test, y_test)This is the most common leakage mistake. Fitting the scaler on the full dataset before splitting lets held-out rows shape the transformation — contaminating the test set before the model even trains.
It exposes the validation and test rows. The scaler computes its mean and standard deviation from ALL rows — including the held-out ones. Those rows secretly influence the scaling applied to the training data, so the test set is no longer truly unseen. The score on line 9 is inflated as a result.
# CORRECT: split first, then fit the scaler on train only X_train, X_test, y_train, y_test = train_test_split( X_all, y_all, test_size=0.2 ) scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) # fit on train only X_test_scaled = scaler.transform(X_test) # apply — do NOT re-fit model.fit(X_train_scaled, y_train) print(model.score(X_test_scaled, y_test)) # honest score
scaler.fit_transform(X_train)scaler.transform(X_test)Split first, then fit the scaler on training rows only. The test set is transformed using the training statistics — it never influences what the scaler learns.
transform() applies the statistics the scaler already learned from the training set — it does NOT recompute them. If you called fit_transform(X_test), the scaler would learn new statistics from the test set, which is leakage. The rule: fit() touches training data only; transform() is applied everywhere using those same training-derived statistics. Lines 7–8 are the changed lines from Stage 1 — the split moved before the fit, and transform() replaced fit_transform() on the test set.
Drag to see how contaminating more of your held-out data inflates the apparent validation accuracy — and widens the gap with real-world performance.
Three failure patterns to watch for, each with a concrete signal:
The next module — "How to Partition Data Correctly" — walks you through a fully guided split: choosing ratios, shuffling safely, applying to preserve class balance, and handling time-ordered data so none of these leakage routes can open.
You work through a fully guided example of splitting a dataset: shuffling, choosing ratios, applying stratification to preserve class balance, and handling time-series data where shuffling is forbidden. You complete a partial split setup to check your own understanding before the solo capstone.
A step-by-step guide to cutting a dataset into train, validation, and test sets — covering shuffle order, ratio choice, stratification for imbalanced classes, and chronological splitting for time-series data.
Why this matters: Getting the split right is the foundation of every honest model evaluation; a bad split makes a broken model look good and a good model look bad.
Decision this forces: Should this dataset be split randomly or chronologically, and does it need stratification?
Module 3 named three leakage paths: done before splitting (e.g. scaling on all rows), (a feature that contains the answer), and (future data shuffled into training).
This module covers the mechanics that prevent all three. It shows how to cut data into the right pieces, in the right order, with the right balance.
The driving question: should your dataset be split randomly or in time order, and does it need stratification?
A split has three decisions: shuffle order, ratio, and .
Without stratification on imbalanced data, your test set might have almost no fraud rows. Accuracy looks great while the model misses real fraud cases.
Imagine you have 10 000 bank transactions: 9 500 are legitimate and 500 are fraud — a 95 / 5 split.
Randomise the row order. Fraud cases were logged in bursts, so without shuffling they'd cluster at the end of the file and land mostly in the test set.
10 000 rows is a medium dataset, so 80 / 10 / 10 is appropriate: 8 000 train, 1 000 val, 1 000 test.
A plain random cut might put only 30 fraud rows in the test set — too few to measure anything reliably. Stratification forces each set to keep the 5 % fraud rate: 400 fraud rows in train, 50 in val, 50 in test.
With 50 fraud rows in the test set you can actually measure recall (how many frauds the model catches). Without stratification, a model that predicts 'legitimate' for every row would score 97 % accuracy — and you'd never know it misses all fraud.
Slide to your dataset size and see which ratio is appropriate. Larger datasets can afford to give more rows to validation and test without starving training.
When rows are ordered in time (stock prices, sensor readings, daily sales), shuffling causes . The model trains on future data and 'predicts' the past, which is trivially easy.
Instead, use a : keep rows in chronological order and cut at fixed time boundaries. For example: earliest 80% of days → train, next 10% → val, final 10% → test.
import random def split_dataset(rows, label_col, val_ratio=0.10, test_ratio=0.10, seed=42): random.seed(seed) # Group rows by class label groups = {} for row in rows: key = row[label_col] groups.setdefault(key, []).append(row) train, val, test = [], [], [] for label, items in groups.items(): random.shuffle(items) n = len(items) n_test = int(n * test_ratio) n_val = int(n * val_ratio) # TODO: slice items into test, val, and train # Hint 1: test gets the LAST n_test rows (items[-n_test:]) # Hint 2: val gets the slice just before test # Hint 3: train gets everything that remains return train, val, test
groups.setdefault(key, [])int(n * test_ratio)items[-n_test:]train += items[...]random.seed(seed)This function performs a stratified split entirely in plain Python — no libraries needed. It groups rows by class label first, shuffles within each group, then slices each group proportionally into test, val, and train.
# Changed lines (the three slices — everything else stays the same):
test += items[-n_test:]
val += items[-(n_test + n_val):-n_test]
train += items[:-(n_test + n_val)]
# Why each line changed from the TODO:
#
#
# - train takes the front slice; nothing overlaps.
Three mistakes account for most bad splits — each one makes your model look better than it really is.
You learn how k-fold cross-validation rotates the validation window to get a more reliable performance estimate on small datasets, and you audit a checklist of the most common splitting mistakes — including leaking the test set, forgetting to stratify, and preprocessing before splitting. This module brings back the core ideas from Modules 1–3 as a final self-check.
This module teaches k-fold cross-validation and how to audit a data pipeline for the four most common splitting mistakes.
Why this matters: Knowing when to use cross-validation and how to spot leakage errors lets you build models whose reported accuracy you can actually trust.
Decision this forces: Does this dataset and task call for a simple train/val/test split or k-fold cross-validation?
Answer: when you split randomly, a rare class (say, 5% of your data) might land almost entirely in one set by chance. fixes this by keeping the class proportions the same across every split — so your always looks like the real world.
You've now handled shuffling, ratios, and stratification correctly. But what if your dataset is too small to give up 20% as a validation set? That's the problem this module solves.
(specifically ) solves the small-dataset problem by reusing your data cleverly. You divide the data into k equal chunks, called folds.
In each round, one fold becomes the and the remaining k−1 folds are used for training. You repeat this k times, rotating which fold is held out. At the end, you average the k validation scores to get one reliable estimate of .
The key payoff: every example is used for validation exactly once, so no data is wasted. A single split can get unlucky (your one validation fold might be easy or hard by chance). K-fold averages that luck away.
Imagine you're building a model to predict whether a patient has diabetes, and you only have 400 labeled records. A single 80/20 split gives you just 80 rows for validation — too few to trust one score.
You first lock away 40 rows (10%) as your untouched . Then you run 5-fold cross-validation on the remaining 360 rows.
Notice: the test set is never touched during cross-validation. Peeking at it even once to 'check' would turn it into a second validation set and destroy its value as an unbiased final verdict.
These four mistakes are the most common reasons a model looks great in development but fails in the real world. Each one corrupts your evaluation in a different way.
You peek at the to tune a threshold, pick a feature, or check a score — even once. The test set is now a second validation set. Your final number is optimistic and you have no honest estimate left. Symptom: the model scores well in evaluation but underperforms when deployed.
You fit a scaler or imputer on the whole dataset, then split. The scaler has already 'seen' the validation and test rows, so their statistics leak into training. This is a form of . Fix: split first, then fit only on the training fold.
On an imbalanced dataset (e.g. 95% healthy, 5% diabetic), a random split can put nearly all the rare class in one set. Your model trains without seeing enough positive examples, or validates on a skewed sample. Always use when exists.
If your data has a time order (e.g. patient visits over months), shuffling before splitting lets future data train the model. This is . The model appears to predict the future but is actually memorizing it. Fix: always split time-series data chronologically — earlier rows train, later rows validate.
# BROKEN pipeline — spot the mistakes before reading on from sklearn.preprocessing import StandardScaler from sklearn.model_selection import train_test_split, cross_val_score scaler = StandardScaler() X_scaled = scaler.fit_transform(X) # BUG 1: scales before splitting X_train, X_test, y_train, y_test = train_test_split( X_scaled, y, test_size=0.2 # BUG 2: no stratify on imbalanced y ) scores = cross_val_score(model, X_train, y_train, cv=5) print(scores.mean()) # looks fine — but it's already tainted
StandardScaler()fit_transform(X)train_test_split(..., stratify=y)cross_val_score(model, X_train, y_train, cv=5)This pipeline looks clean but contains two classic leakage traps. Spotting them before training is the whole skill — the model will never warn you.
BUG 1 (line 5): scaler.fit_transform(X) sees the entire dataset — including the future test rows — before the split. Their mean and variance leak into the scaler, so the test set is no longer unseen. Fix: split first, then fit the scaler only on X_train. BUG 2 (line 8): train_test_split has no stratify=y argument. On an imbalanced label column, the rare class may be under-represented in one set by chance. Fix: add stratify=y.
# FIXED pipeline — one line is missing; supply it before revealing from sklearn.preprocessing import StandardScaler from sklearn.model_selection import train_test_split, cross_val_score X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, stratify=y, random_state=42 ) scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) # fit on train only X_test_scaled = ___________________________ # TODO: your line here scores = cross_val_score(model, X_train_scaled, y_train, cv=5) print("CV mean accuracy:", scores.mean())
scaler.transform(X_test)random_state=42scores.mean()This is a small variation of the broken pipeline — the split and stratify are now correct, and only one line is missing. The gap is the crux: knowing when to call transform vs fit_transform is the exact skill that prevents preprocessing leakage.
X_test_scaled = scaler.transform(X_test) # CHANGED LINE
Why: scaler.fit_transform(X_train) already learned the mean and std from training data. Calling scaler.transform(X_test) applies those same statistics to the test set — no new fitting, no leakage. Using fit_transform on X_test would be the bug from the previous block all over again.
Before looking anything up: in your own words, name the three splits and the one job each is allowed to do. Then recall two ways data leakage can enter a pipeline and the one rule that prevents both. Finally, state when you would choose k-fold cross-validation over a simple three-way split.
Apply what you learned to Train, Validation, and Test Splits.
You train a spam classifier and report 99% accuracy. A colleague points out you measured accuracy on the same emails the model trained on. Why is that score misleading?
When a model is evaluated on its own training data it can score high simply by memorizing examples — this is overfitting. The score tells you nothing about unseen data. Accuracy vs. precision is a separate concern unrelated to the split problem. Training-set size does not fix the fundamental issue of evaluating on seen data. And a model that memorized imperfectly could score below 100%, so a 99% score does not rule out memorization.
A team is building a model and uses validation-set results to pick the best hyperparameters. They then keep peeking at the test set after each tuning round to make sure they are on the right track. What goes wrong?
Every time you use test-set feedback to guide a decision, you are effectively tuning to that set, which destroys its ability to give an unbiased final estimate. The test set must stay locked until all tuning is finished. The other options misidentify the mechanism: the validation set is not contaminated by looking at the test set, and peeking at the test set does not cause underfitting.
Consider this preprocessing step:
scaler.fit(full_dataset)
X_train, X_test = split(full_dataset)
What is wrong with this code?
Fitting the scaler on the full dataset means it has seen the test examples' values when computing mean and variance. That is a form of data leakage — the test set is no longer truly held out. The correct order is: split first, then fit the scaler only on the training portion and apply (transform) it to the test portion. Split ratio and model order are irrelevant to this specific bug. Fitting the scaler on the test set would be even worse.
You have a dataset of daily stock prices from 2015 to 2024 and want to predict tomorrow's price. A teammate suggests a random 70/15/15 train/validation/test split. When should you override that suggestion, and what should you use instead?
Time-series data has temporal dependencies: future values can influence past predictions if data is shuffled randomly, which is a form of leakage. A chronological split ensures the model is always trained on the past and evaluated on the future, mimicking real deployment. Dataset size does not change this requirement. Stratification by year still shuffles within years and does not prevent leakage. Stock prices are not independent day to day — they carry autocorrelation and trends.
You have a medical imaging dataset with only 200 labeled scans. You want to tune a model's hyperparameters and get a reliable estimate of its performance. Should you use a simple train/validation/test split or k-fold cross-validation for the tuning phase? Explain your reasoning in 2–3 sentences.
Small datasets are the primary use case for k-fold cross-validation. A single validation split on 200 samples could easily be unrepresentative, leading to noisy hyperparameter choices. K-fold rotates which samples are held out, producing a more stable average score. The test set should still remain locked and separate from the cross-validation loop to preserve an unbiased final estimate.