Lesson 13

Model Selection

Compare multiple algorithms fairly using cross-validation, understand the no-free-lunch theorem, and choose the right model for linear, non-linear, and interpretable problems.

Intermediate25 min

Table mapping five algorithms to their best use cases

No algorithm is best on every problem — this is the no-free-lunch theorem. The practical approach is to cross-validate several candidates on your data and pick the one with the best balance of accuracy, speed, and interpretability.

Comparing five classifiers

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score
import numpy as np
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["senior"].values
models = [
    ("LogisticRegression", make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))),
    ("KNN(11)           ", make_pipeline(StandardScaler(), KNeighborsClassifier(11))),
    ("DecisionTree(4)   ", DecisionTreeClassifier(max_depth=4, random_state=0)),
    ("RandomForest      ", RandomForestClassifier(n_estimators=100, random_state=0)),
    ("GradientBoosting  ", GradientBoostingClassifier(n_estimators=100, random_state=0)),
]
for name, clf in models:
    sc = cross_val_score(clf, X, y, cv=5).mean()
    print(f"{name}: {sc:.3f}")

Output:

text

LogisticRegression: 0.940
KNN(11)           : 0.913
DecisionTree(4)   : 0.893
RandomForest      : 0.913
GradientBoosting  : 0.900

Choosing a model: practical guide

Situation	Start with
Linear boundary, few features	Logistic regression
Distance-based, small data	KNN
Need interpretability	Decision tree
General-purpose, tabular	Random forest
Best accuracy on tabular	Gradient boosting
Very large dataset	SGD-based models

Start simple. A logistic regression baseline often comes within a few points of a complex model, and its failure modes are easier to diagnose.

Checking with a held-out test set

Cross-validation scores are used to select a model, not to report final accuracy. Once you have chosen a model, train it on all training rows and score it on the test set exactly once.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score, train_test_split
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["senior"].values
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
gb = GradientBoostingClassifier(n_estimators=100, random_state=0)
cv_mean = cross_val_score(gb, X_train, y_train, cv=5).mean()
gb.fit(X_train, y_train)
print(f"cv mean: {cv_mean:.3f}")
print(f"test:    {gb.score(X_test, y_test):.3f}")

Output:

text

cv mean: 0.908
test:    0.883

Key takeaways

No single algorithm is best on all problems — this is the no-free-lunch theorem
Use cross-validation to compare candidates fairly on training data
Score the chosen model on the test set exactly once
Start simple: a logistic regression baseline reveals how hard the problem is
Gradient boosting often wins on tabular data, but is slower to train and tune

Loading playground…

Check your understanding

Model Selection Quiz

What does the no-free-lunch theorem say?

1 / 10

← Ensemble Methods

ML Workflow →