Compare linear models, trees, support-vector machines, neighbors, and clustering.
Linear and logistic models are strong first baselines for many tabular tasks.
Learners get a concrete starting point for model choice.
Why this matters: This prevents skipping simple models that are cheap, stable, and interpretable.
Problem anchor: a support team wants to predict customer churn from days since last login, ticket count, plan tier, and billing failures. A linear or logistic model treats each feature as weighted evidence. More billing failures may push churn risk up; more recent logins may push it down.
This family is a good baseline because it is fast, explainable, and hard to overfit compared with very flexible models. It struggles when the decision boundary bends sharply or features interact in ways a weighted sum cannot express.
Score = 0.8*billing_failures + 0.4*ticket_count - 0.6*recent_login + 0.3*enterprise_plan. A customer with two billing failures, three tickets, a recent login, and no enterprise plan gets 0.8*2 + 0.4*3 - 0.6*1 + 0 = 2.2. The exact numbers are learned, but the structure is inspectable.
Tree models capture nonlinear rules but can overfit without constraints.
Learners see a second bias: branching rules instead of weighted sums.
Why this matters: This helps choose trees when interactions matter and auditability is needed.
A decision tree might first ask whether billing_failures > 0, then whether days_since_login > 14, then whether support_tickets > 2. That makes interactions easy to express: a stale login may matter more when billing also failed.
Trees are readable, handle mixed feature types, and capture nonlinear splits. They can also memorize quirks if grown too deep, so validation depth and pruning matter.
The churn tree reveals a useful business rule: billing failure plus no login for 30 days is high risk. But an unconstrained tree also creates a leaf for a rare legacy plan with two training examples. Validation error rises, warning that the rare leaf is memorization.
Geometry-based models care about scaling, distance, and support points.
Learners compare geometric model families to lines and trees.
Why this matters: This prevents using distance methods on unscaled or irrelevant features.
A support-vector machine tries to find a boundary with a wide margin between classes. k-nearest neighbors keeps the training examples and predicts by nearby examples. Both depend heavily on feature scale and distance meaning. If account_value ranges to millions while ticket_count ranges to ten, distance becomes distorted unless features are scaled.
| Option | Family | Best fit | Common failure | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Linear/logistic | Linear/logistic | strong baseline, sparse or tabular data, interpretability | misses curved interactions | First supervised baseline. | Low | Medium |
| Decision tree | Decision tree | nonlinear splits and readable rules | overfits deep rare leaves | Need interactions and inspectable paths. | Low | Medium |
| SVM | SVM | clear margin in scaled feature space | sensitive to scaling and kernel choice | Medium-sized data with useful geometric boundary. | Medium | Medium |
| k-nearest neighbors | k-nearest neighbors | local similarity with meaningful distances | slow prediction and noisy dimensions | Small data where neighbors are meaningful. | Medium | Medium |
A runnable scikit-learn comparison shows baseline, tree, and neighbor tradeoffs.
Learners see the zoo as an experiment table.
Why this matters: This keeps model selection evidence-based.
from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression from sklearn.tree import DecisionTreeClassifier from sklearn.neighbors import KNeighborsClassifier X, y = load_iris(return_X_y=True) X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=7, stratify=y) models = { "logistic": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)), "tree_depth_3": DecisionTreeClassifier(max_depth=3, random_state=7), "knn_5": make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5)), } for name, model in models.items(): model.fit(X_train, y_train) print(name, round(model.score(X_val, y_val), 3))
Logistic regression and kNN are sensitive to feature scale. Trees split by thresholds and usually do not need feature scaling in the same way.
Checklist: every model uses the same split; preprocessing is inside a pipeline; distance-based models are scaled; random seeds are fixed for examples; validation, not training score, chooses the candidate; and the report mentions interpretability and prediction cost.
Model choice balances validation performance, interpretability, data size, latency, and maintenance.
Learners learn when to stop or upgrade.
Why this matters: This prevents needless complexity and mismatched algorithms.
Linear models are inspectable baselines. Trees express rules but can overfit. SVMs and neighbors need meaningful geometry. Ensembles often improve accuracy but reduce simplicity. Clustering belongs in the zoo too: it groups unlabeled data for exploration, anomaly review, and candidate segments, but it does not replace supervised labels for a churn target.
A beginner decision rule: if you have labels and a target, start supervised. If you do not have labels and need to understand structure, use clustering as hypothesis generation. If you need both, cluster first to explore, then label or evaluate before training a supervised model.
| Option | Family | Best fit | Common failure | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| k-means clustering | k-means clustering | unlabeled compact groups around centroids | forces every point into a cluster even when groups are weak | Explore unlabeled data or candidate segments. | Low | Medium |
| DBSCAN-style clustering | DBSCAN-style clustering | dense groups plus noise/outliers | sensitive to distance scale and density settings | Find repeated issue pockets with outliers. | Medium | Medium |
| Supervised classifier | Supervised classifier | known target labels such as churn/no churn | learns shortcuts if labels or features leak | Predict a defined outcome. | Medium | Medium |
Retrieval prompt: reconstruct the classic ML zoo by naming linear models, trees, SVMs, nearest neighbors, ensembles, clustering, their assumptions, failure modes, and the validation evidence that chooses among them.
For a customer-churn table, compare logistic regression, decision tree, k-nearest neighbors, and random forest. Write what each model assumes, run a small validation comparison, and choose the simplest model that meets the metric and interpretability need. Next rung: add calibration and feature leakage checks.
A linear model predicts house prices by multiplying each feature (size, age, rooms) by a weight and summing them up. What does a large positive weight on 'size' tell you?
In a linear model, a large positive weight means that feature contributes strongly in the positive direction — more size, higher predicted price.
You train a decision tree on a small dataset and it gets 100% accuracy on training data but only 58% on the test set. What is the most likely cause?
A perfect training score paired with a much lower test score is the classic signature of overfitting — the tree grew deep enough to memorize noise rather than learn general rules.
You want to classify emails as spam or not. Your features include word counts on very different scales (e.g., 'links': 0–3 vs. 'characters': 0–5000). Which model family is MOST sensitive to this scaling difference?
KNN measures raw distance between points, so a feature with a huge numeric range will dominate the distance calculation and drown out smaller-scale features — scaling is essential.
You run a model zoo comparison and see: Linear model — train 82%, test 81%; Decision Tree — train 99%, test 74%; SVM — train 85%, test 84%. Which model would you recommend moving forward with, and why? (Write 1–2 sentences.)
A small train-test gap indicates good generalization; the decision tree's large gap is a red flag for overfitting, making the SVM or linear model the safer pick.
A colleague suggests using k-means clustering instead of logistic regression to predict whether a customer will churn (yes/no). What is the key problem with this plan?
Clustering is unsupervised — it finds natural groups without using any labels, so it cannot be a direct substitute for a supervised classifier that predicts a known outcome like churn.