Lesson 11

Learning Curves

Plot training and validation scores against training set size to diagnose whether your model suffers from high bias, high variance, or neither, and decide whether more data will help.

Intermediate25 min

Training and validation curves converging as training size grows

A learning curve plots model performance against the number of training examples. The shape tells you whether you have a bias problem (both curves low), a variance problem (large gap), or a good fit (curves close and high).

Plotting the learning curve

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import learning_curve
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["senior"].values
sizes, train_scores, val_scores = learning_curve(
    DecisionTreeClassifier(max_depth=4, random_state=0),
    X, y, cv=5, train_sizes=[0.2, 0.4, 0.6, 0.8, 1.0], scoring="accuracy",
)
for sz, tr, vl in zip(sizes, train_scores.mean(1), val_scores.mean(1)):
    print(f"n={sz:3d}  train {tr:.3f}  val {vl:.3f}")

Output:

text

n= 48  train 0.996  val 0.917
n= 96  train 0.967  val 0.917
n=144  train 0.965  val 0.910
n=192  train 0.966  val 0.900
n=240  train 0.957  val 0.893

The validation score rises with more data, and the gap between train and val shrinks. This is the signature of high variance: more data helps.

Validation curve — varying a hyperparameter

validation_curve sweeps one hyperparameter instead of the data size. This shows the bias-variance tradeoff directly: low parameter values underfit, high values overfit.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import validation_curve
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["senior"].values
depths = [1, 2, 3, 4, 6, 8, None]
train_s, val_s = validation_curve(
    DecisionTreeClassifier(random_state=0), X, y,
    param_name="max_depth", param_range=depths, cv=5,
)
for depth, tr, vl in zip(depths, train_s.mean(1), val_s.mean(1)):
    print(f"depth {str(depth):4s}  train {tr:.3f}  val {vl:.3f}")

Output:

text

depth 1     train 0.918  val 0.910
depth 2     train 0.918  val 0.910
depth 3     train 0.948  val 0.910
depth 4     train 0.957  val 0.893
depth 6     train 0.983  val 0.873
depth 8     train 0.997  val 0.877
depth None  train 1.000  val 0.873

Deep trees or no limit: training score approaches 1.0 but validation drops. Shallow trees: both stay low. The best validation score shows the right depth.

Key takeaways

Learning curves plot performance vs training size to diagnose bias or variance
High bias: both train and val scores are low, even with more data
High variance: large train-val gap that shrinks as data grows
Validation curves sweep a hyperparameter to show the bias-variance tradeoff directly
learning_curve and validation_curve are in sklearn.model_selection

Loading playground…

Check your understanding

Learning Curves Quiz

What does a learning curve plot on its x-axis?

1 / 10

← Imbalanced Data

Ensemble Methods →