Lesson 16

Hyperparameter Tuning

Tell learned parameters from settings you choose, search settings with GridSearchCV and cross-validation, and test the winning model exactly once.

Intermediate25 min

Hyperparameter Tuning hero Module 27 Lesson 16

A model learns its parameters, such as coefficients or tree splits, during fit. Hyperparameters are the settings you choose before fitting, such as max_depth or n_neighbors. Tuning means trying many settings and keeping the best, judged by cross-validation on the training rows so the test set stays untouched.

A grid of cross-validated accuracy for each combination of tree depth and leaf size

Settings versus learned values

get_params lists the settings, which exist before fitting. Learned values, ending in an underscore, only appear after fit.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(max_depth=3, random_state=0)
settings = tree.get_params()
print({name: settings[name] for name in ["max_depth", "min_samples_leaf", "random_state"]})
print(hasattr(tree, "tree_"))

tree.fit(learners[["hours", "practice"]], learners["passed"])
print(hasattr(tree, "tree_"), tree.get_depth())

Output:

text

{'max_depth': 3, 'min_samples_leaf': 1, 'random_state': 0}
False
True 3

Try values by hand

A loop with cross_val_score is the simplest search: one setting at a time, scored on the training rows only.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.tree import DecisionTreeClassifier

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
for depth in [1, 2, 3, 5, 8]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0)
    print(depth, cross_val_score(tree, X_train, y_train, cv=5).mean().round(3))

Output:

text

GridSearchCV

GridSearchCV tries every combination in param_grid, cross-validates each, then refits the best on all training rows. It is itself an estimator, so it has fit, predict, and score.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.tree import DecisionTreeClassifier

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
param_grid = {"max_depth": [1, 2, 3, 5, 8], "min_samples_leaf": [1, 5, 20]}
search = GridSearchCV(DecisionTreeClassifier(random_state=0), param_grid, cv=5)
search.fit(X_train, y_train)

print(search.best_params_)
print(round(search.best_score_, 3))

Output:

text

{'max_depth': 3, 'min_samples_leaf': 20}
0.825

Read every result

cv_results_ holds the score of every combination, and its params entry lists the settings in grid order. As a DataFrame it is easy to sort and see how close the runners-up were.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.tree import DecisionTreeClassifier

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
param_grid = {"max_depth": [1, 2, 3, 5, 8], "min_samples_leaf": [1, 5, 20]}
search = GridSearchCV(DecisionTreeClassifier(random_state=0), param_grid, cv=5).fit(X_train, y_train)

results = pd.DataFrame(search.cv_results_["params"])
results["mean_score"] = search.cv_results_["mean_test_score"].round(3)
results["rank"] = search.cv_results_["rank_test_score"]
print(results.sort_values("rank", kind="stable").head(5).to_string(index=False))

Output:

text

 max_depth  min_samples_leaf  mean_score  rank
         3                20       0.825     1
         5                20       0.825     1
         8                20       0.825     1
         3                 1       0.819     4
         3                 5       0.819     4

Several settings tie for first place. best_params_ is the first of them in the grid, so when scores tie it pays to prefer the simplest model yourself.

Tune a pipeline, then test once

To tune a step inside a pipeline, join the step name and the setting with two underscores. Score the winner on the test set exactly once, at the very end; if you keep tuning against the test set it stops being honest.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X = learners[["hours", "practice", "video_minutes"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
pipe = make_pipeline(StandardScaler(), KNeighborsClassifier())
search = GridSearchCV(pipe, {"kneighborsclassifier__n_neighbors": [3, 5, 11, 21, 41]}, cv=5)
search.fit(X_train, y_train)

print(search.best_params_)
print("cv:  ", round(search.best_score_, 3))
print("test:", search.score(X_test, y_test))

Output:

text

{'kneighborsclassifier__n_neighbors': 41}
cv:   0.844
test: 0.875

Key takeaways

Parameters are learned in fit; hyperparameters are settings you choose
get_params lists the settings of any estimator
GridSearchCV cross-validates every combination and refits the best
best_params_, best_score_, and cv_results_ report the search
Use step__setting to tune inside a pipeline; test the winner once
Next: K-Means Clustering
Previous: Cross-Validation

Loading playground…

Check your understanding

Hyperparameter Tuning Quiz

What is a hyperparameter?

1 / 10

← Cross-Validation

K-Means Clustering →