Turn raw data into reliable model inputs without leaking validation data.
Feature engineering defines what the model can see.
Learners see features as product choices, not automatic columns.
Why this matters: This prevents models from seeing data that will not exist at inference.
Problem anchor: predict whether a support ticket is urgent. Raw data includes ticket text, customer plan, account age, previous ticket count, agent notes, and final resolution time. Resolution time is tempting because it predicts urgency, but it is known only after the ticket is handled. It is leakage, not a usable feature.
Good feature engineering states source, timestamp, transformation, allowed use, and owner. The model can only learn from the view you give it.
Feature: recent_ticket_count_30d. Source: ticket table. Time rule: count tickets opened before current ticket timestamp. Transform: log1p count. Owner: support analytics. Leakage check: excludes current ticket resolution and future tickets.
Learned preprocessing must be fit on training data and applied to validation/test.
Learners understand the core anti-leakage rule.
Why this matters: This catches inflated metrics from preprocessing on all rows.
A scaler learns means and variances. An imputer learns replacement values. A category encoder learns category mapping. If these are fit on all rows before splitting, validation and test information has influenced training. A pipeline helps by fitting preprocessing inside cross-validation or train-only fit calls.
The team standardizes account_age using the mean from all tickets, including validation. The validation distribution has a launch-week surge of new accounts. The train features now contain a hint of validation distribution. The effect may be small, but the discipline is simple: fit preprocessing on train only.
ColumnTransformer and Pipeline make preprocessing and model fitting reproducible.
Learners build the reliable shape used in real ML projects.
Why this matters: This prevents notebook-only transformations from drifting in production.
A support-ticket model may scale numeric features, one-hot encode product category, and pass text-derived counts through unchanged. A pipeline ties those transformations to the model so training, validation, and inference use the same steps.
| Option | Component | Learns during fit | Production risk | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Imputer | Imputer | replacement values from train | validation/test distribution leak if fit globally | Missing numeric or categorical values. | Low | Medium |
| Scaler | Scaler | mean and variance from train | distance or coefficient distortion if inconsistent | Linear, SVM, kNN, neural models. | Low | Medium |
| One-hot encoder | One-hot encoder | category mapping from train | unknown categories at inference | Categorical columns. | Low | Medium |
| Model | Model | parameters from transformed train rows | cannot reproduce predictions if preprocessing is separate | Always tied to preprocessing. | Medium | Medium |
Leakage checks should be coded and reviewed like tests.
Learners implement a leakage-safe pipeline and checklist.
Why this matters: This makes feature safety concrete for build work.
from sklearn.compose import ColumnTransformer from sklearn.pipeline import Pipeline from sklearn.preprocessing import OneHotEncoder, StandardScaler from sklearn.impute import SimpleImputer from sklearn.linear_model import LogisticRegression numeric = ["account_age_days", "recent_ticket_count_30d", "text_length"] categorical = ["plan_tier", "product_area"] numeric_pipe = Pipeline([ ("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler()), ]) category_pipe = Pipeline([ ("impute", SimpleImputer(strategy="most_frequent")), ("onehot", OneHotEncoder(handle_unknown="ignore")), ]) preprocess = ColumnTransformer([ ("num", numeric_pipe, numeric), ("cat", category_pipe, categorical), ]) model = Pipeline([ ("preprocess", preprocess), ("clf", LogisticRegression(max_iter=1000)), ]) # Correct pattern after splitting: # model.fit(X_train, y_train) # validation_score = model.score(X_val, y_val)
Validation statistics influence the training representation. Metrics can look better than deployment reality because preprocessing saw information from the evaluation rows.
Checklist: split before fit; learned transforms are inside Pipeline or ColumnTransformer; target-derived and future-derived features are excluded; aggregations use only past data; unknown categories are handled; and feature definitions include source and timestamp rules.
Production feature quality needs data checks, drift alerts, and owner review.
Learners extend feature engineering beyond training notebooks.
Why this matters: This prevents silent production breakage from data changes.
After deployment, product_area may get new categories, text_length may shift after a UI change, or recent_ticket_count_30d may break when an event pipeline changes. Monitor missingness, value ranges, category rates, freshness, and prediction slices. A feature owner should know when a contract changes.
Retrieval prompt: reconstruct feature pipelines by naming raw columns, transformations, train-only fit, ColumnTransformer, leakage checks, validation slices, and monitoring for feature drift.
Build a leakage-safe pipeline for support-ticket priority. Include text length, product category, account age, and recent ticket count; fit preprocessing on train only; validate with held-out data; add leakage tests and feature monitoring notes. Next rung: add feature store lineage.
A support ticket dataset has a field called resolution_time_hours — the number of hours it took to close the ticket. You want to predict whether a ticket will be escalated. Why should this field be excluded from your feature set?
Fields that are only populated after the outcome occurs are 'future fields' — using them leaks information from the future into training and makes the model useless in production.
You fit a StandardScaler on your entire dataset (train + test combined), then split into train and test sets. What problem does this cause?
Fitting on the full dataset lets test-set distribution influence the scaler's parameters — a classic preprocessing leakage path that inflates evaluation metrics.
In a scikit-learn Pipeline with a ColumnTransformer, what is the main production benefit of keeping the preprocessor and the model bundled together as a single pipeline object?
A bundled pipeline serialises the fitted transformers alongside the model, so scoring new data always goes through the identical preprocessing steps — making it a reliable production artifact.
You add a feature called pct_tickets_escalated_by_agent — the percentage of that agent's tickets that were escalated — computed over the full training dataset. Your cross-validation AUC jumps from 0.72 to 0.91. Give ONE reason why this metric jump is likely misleading.
Aggregates derived from the target variable across the full dataset are a form of target leakage — the feature already 'knows' the outcome, so the model appears far more accurate than it will ever be on real unseen data.
Three months after deployment, your model's precision drops noticeably. Your monitoring system flags that the ticket_category column now contains a new value, 'billing_v2', that never appeared in training. What type of drift is this, and which monitoring check would have caught it earliest?
A new, unseen category value is a schema drift event (the contract for allowed values has changed); a category-set monitor that compares live values against the training vocabulary would flag it immediately.