Lesson 02

Bias and Variance

Learn why models underfit or overfit, how model complexity controls the bias-variance tradeoff, and how to recognise each from training and test scores.

Intermediate25 min

Training score rises with complexity; test score peaks then drops

Every model makes errors. Those errors come from two sources: bias (the model is too simple to fit the data) and variance (the model is so flexible that it memorises noise). The goal is to find a complexity that minimises both.

High bias: underfitting

A model with high bias is too simple. It misses the real pattern, so it scores poorly on both training and test data. A depth-1 tree, for example, can only ask one yes or no question.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
X = employees[["experience", "certifications", "overtime", "education"]]
y = employees["senior"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
for depth in [1, 4, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
    print(f"depth {str(depth):4s}  train {tree.score(X_train, y_train):.3f}  test {tree.score(X_test, y_test):.3f}")

Output:

text

depth 1     train 0.912  test 0.933
depth 4     train 0.950  test 0.883
depth None  train 1.000  test 0.867

Depth 1 underfits: low scores everywhere. No limit overfits: train is perfect but test drops. Depth 4 is the sweet spot here.

High variance: overfitting

A model with high variance fits the training data too closely, including its noise. A good sign is a large gap between training score and test score. Polynomial regression makes the gap easy to see.

python

import numpy as np
from numpy.random import default_rng
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split

rng = default_rng(7)
x = rng.uniform(0, 10, 80)
y_true = 3 * x - 0.4 * x**2 + rng.normal(0, 4, 80)
X2 = x.reshape(-1, 1)
Xtr, Xte, ytr, yte = train_test_split(X2, y_true, test_size=0.25, random_state=7)
for deg in [1, 3, 10]:
    pipe = make_pipeline(PolynomialFeatures(deg), LinearRegression())
    pipe.fit(Xtr, ytr)
    print(f"deg {deg:2d}  train {pipe.score(Xtr, ytr):.3f}  test {pipe.score(Xte, yte):.3f}")

Output:

text

deg  1  train 0.466  test 0.393
deg  3  train 0.658  test 0.618
deg 10  train 0.691  test 0.577

Degree 10 trains at near-perfect but collapses on the test set. The gap is the variance tax.

Recognising each from the scores

Pattern	Training score	Test score
High bias (underfitting)	Low	Low
Good fit	Medium-high	Close to training
High variance (overfitting)	High	Much lower than training

The fix for high bias is a more complex model or more features. The fix for high variance is a simpler model, more data, or regularisation.

Key takeaways

Bias errors come from a model that is too simple
Variance errors come from a model that memorises training noise
A large train-test gap signals high variance
Similar low train and test scores signal high bias
Fix bias with more complexity; fix variance with simpler models or more data

Loading playground…

Check your understanding

Bias and Variance Quiz

What causes high bias?

1 / 10

← What Is Machine Learning

Gradient Descent →