Lesson 05

Measuring Regression Error

Measure how far regression predictions miss with residuals, mean absolute error, RMSE, and R squared, and compare the model against a simple baseline.

Intermediate25 min

Measuring Regression Error hero Module 27 Lesson 5

A model is only useful if you know how wrong it tends to be. Every regression metric starts from the residual: the actual value minus the prediction. The metrics then summarise those misses in different ways, and each answers a slightly different question about the model.

Points around a fitted line with vertical lines showing each residual

Residuals

Predict the test rows, subtract, and look at a few misses. Positive means the learner did better than predicted.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

X = learners[["hours", "practice"]]
y = learners["score"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = LinearRegression().fit(X_train, y_train)
predicted = model.predict(X_test)
residuals = y_test - predicted

print(y_test.head().tolist())
print(predicted[:5].round(1))
print(residuals.head().round(1).tolist())

Output:

text

[62.0, 47.0, 57.0, 54.0, 83.0]
[66.5 46.5 47.4 51.  68.5]
[-4.5, 0.5, 9.6, 3.0, 14.5]

Mean absolute error

MAE drops the signs and averages the misses. It is in the same units as the target, so it reads naturally: the typical prediction is this many points off.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split

X = learners[["hours", "practice"]]
y = learners["score"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
predicted = LinearRegression().fit(X_train, y_train).predict(X_test)

print(round(mean_absolute_error(y_test, predicted), 2))
print(round(np.abs(y_test - predicted).mean(), 2))

Output:

text

5.83
5.83

MSE and RMSE

Mean squared error squares each miss before averaging, so one large miss counts far more than several small ones. Its units are squared points, which is hard to read, so people take the square root to get RMSE.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

X = learners[["hours", "practice"]]
y = learners["score"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
predicted = LinearRegression().fit(X_train, y_train).predict(X_test)

mse = mean_squared_error(y_test, predicted)
print("MSE: ", round(mse, 2))
print("RMSE:", round(np.sqrt(mse), 2))

Output:

text

MSE:  52.5
RMSE: 7.25

RMSE is always at least as large as MAE. The bigger the gap between them, the more a few large misses dominate.

R squared

r2_score compares the model's squared error with the error of always predicting the mean. It is the same number model.score returns.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split

X = learners[["hours", "practice"]]
y = learners["score"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = LinearRegression().fit(X_train, y_train)

print(round(r2_score(y_test, model.predict(X_test)), 3))
print(round(model.score(X_test, y_test), 3))

Output:

text

0.706
0.706

Beat a baseline

A number means little alone. DummyRegressor always predicts the training average, which is the least any real model should beat.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split

X = learners[["hours", "practice"]]
y = learners["score"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
for model in [DummyRegressor(strategy="mean"), LinearRegression()]:
    predicted = model.fit(X_train, y_train).predict(X_test)
    print(f"{type(model).__name__}: MAE {mean_absolute_error(y_test, predicted):.2f}")

Output:

text

DummyRegressor: MAE 11.23
LinearRegression: MAE 5.83

Key takeaways

A residual is actual minus predicted
MAE is the average miss in the target's own units
MSE punishes large misses; RMSE brings it back to the target's units
R squared compares the model with always predicting the mean
Always compare against a DummyRegressor baseline
Next: Logistic Regression
Previous: Linear Regression

Loading playground…

Check your understanding

Measuring Regression Error Quiz

What is a residual?

1 / 10

← Linear Regression

Logistic Regression →