Lesson 15

Cross-Validation

Score a model on several different test folds with cross_val_score, KFold, and StratifiedKFold, and report a mean and spread instead of one lucky split.

Intermediate25 min

Cross-Validation hero Module 27 Lesson 15

One train and test split gives one number, and with 40 test rows that number depends a lot on which 40 were picked. Cross-validation splits the data into several folds, lets each fold be the test set once, and trains on the rest each time. You get several scores, and their average is far more trustworthy.

Five rounds of training and testing where the test fold moves along each round

One split can be lucky

Same model, same data, five different random_state values. The test accuracy swings by several points just from which rows were held back.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X = learners[["hours", "practice"]]
y = learners["passed"]
for seed in range(5):
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=seed)
    model = LogisticRegression().fit(X_train, y_train)
    print(seed, model.score(X_test, y_test))

Output:

text

cross_val_score

cross_val_score does the splitting, fitting, and scoring. cv=5 gives five folds and five scores. Report the mean together with the standard deviation, which shows how much the score moves.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

X = learners[["hours", "practice"]]
y = learners["passed"]
scores = cross_val_score(LogisticRegression(), X, y, cv=5)

print(scores)
print(f"mean {scores.mean():.3f}, spread {scores.std():.3f}")

Output:

text

[0.9   0.8   0.85  0.775 0.8  ]
mean 0.825, spread 0.045

For classifiers, cv=5 already uses stratified folds, so every fold keeps roughly the same pass rate.

KFold with shuffling

Folds are taken in row order unless you shuffle. If the file is sorted, say by date or score, unshuffled folds can be very unlike each other. Pass a KFold with shuffle=True as cv.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, cross_val_score

X = learners[["hours", "practice"]]
y = learners["score"]
folds = KFold(n_splits=5, shuffle=True, random_state=0)

for train_rows, test_rows in folds.split(X):
    print(len(train_rows), len(test_rows), test_rows[:4])

print(cross_val_score(LinearRegression(), X, y, cv=folds).round(3))

Output:

text

160 40 [ 4  5  7 12]
160 40 [ 8 14 19 24]
160 40 [ 0  2 10 13]
160 40 [ 1  3  6 11]
160 40 [ 9 21 25 29]
[0.666 0.715 0.834 0.709 0.81 ]

Choose the scoring

scoring= picks the metric. Error metrics are given negated, because scikit-learn always treats bigger as better; flip the sign to read them.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score

X = learners[["hours", "practice"]]
y = learners["score"]

r2 = cross_val_score(LinearRegression(), X, y, cv=5, scoring="r2")
mae = cross_val_score(LinearRegression(), X, y, cv=5, scoring="neg_mean_absolute_error")
print("R squared:", r2.mean().round(3))
print("MAE:", (-mae.mean()).round(2))

Output:

text

R squared: 0.762
MAE: 5.86

Compare models fairly

Cross-validate a whole pipeline so the scaler is refitted inside every fold. Comparing means across models is the honest way to choose between them.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X = learners[["hours", "practice", "video_minutes"]]
y = learners["passed"]
candidates = {
    "logistic": make_pipeline(StandardScaler(), LogisticRegression()),
    "knn": make_pipeline(StandardScaler(), KNeighborsClassifier()),
    "forest": RandomForestClassifier(random_state=0),
}
for name, model in candidates.items():
    scores = cross_val_score(model, X, y, cv=5)
    print(f"{name:>8}: {scores.mean():.3f} +/- {scores.std():.3f}")

Output:

text

logistic: 0.835 +/- 0.044
     knn: 0.840 +/- 0.020
  forest: 0.830 +/- 0.062

Key takeaways

A single split's score depends on which rows were held back
cross_val_score trains and tests once per fold and returns every score
Report the mean and the spread of the fold scores
Shuffle folds with KFold(shuffle=True) when rows are ordered
Error metrics are negated in scoring=; flip the sign to read them
Next: Hyperparameter Tuning
Previous: Pipelines

Loading playground…

Check your understanding

Cross-Validation Quiz

Why is one train and test split not enough?

1 / 10

← Pipelines

Hyperparameter Tuning →