Lesson 07

Measuring Classification

Go beyond accuracy with the confusion matrix, precision, recall, and F1, and see why a model that never says yes can still score high accuracy.

Intermediate25 min

Measuring Classification hero Module 27 Lesson 7

Accuracy counts correct answers, but it hides which mistakes were made. Saying a struggling learner is fine is a different mistake from worrying about one who is doing well. The confusion matrix separates the four outcomes, and precision and recall turn them into two numbers that answer two different questions.

A two by two confusion matrix of actual against predicted pass results

When accuracy lies

Only a few learners score 75 or more. A dummy model that always answers no gets most rows right without finding a single top learner. Accuracy alone cannot tell you that.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score

top = (learners["score"] >= 75).astype(int)
print("share of top learners:", top.mean())

always_no = DummyClassifier(strategy="most_frequent").fit(learners[["hours"]], top)
predicted = always_no.predict(learners[["hours"]])
print("accuracy:", accuracy_score(top, predicted))
print("top learners found:", int(predicted.sum()))

Output:

text

share of top learners: 0.11
accuracy: 0.89
top learners found: 0

The confusion matrix

Rows are the actual classes and columns are the predictions, both in the order 0 then 1. ravel unpacks the four cells: true negatives, false positives, false negatives, and true positives.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
predicted = LogisticRegression().fit(X_train, y_train).predict(X_test)
matrix = confusion_matrix(y_test, predicted)
print(matrix)

tn, fp, fn, tp = matrix.ravel()
print("tn", tn, "fp", fp, "fn", fn, "tp", tp)

Output:

text

[[17  4]
 [ 3 16]]
tn 17 fp 4 fn 3 tp 16

Precision and recall

Precision asks: of the learners predicted to pass, how many did? Recall asks: of the learners who passed, how many did the model find? The manual formulas give the same numbers.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, precision_score, recall_score
from sklearn.model_selection import train_test_split

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
predicted = LogisticRegression().fit(X_train, y_train).predict(X_test)
tn, fp, fn, tp = confusion_matrix(y_test, predicted).ravel()

print("precision:", round(precision_score(y_test, predicted), 3), round(tp / (tp + fp), 3))
print("recall:   ", round(recall_score(y_test, predicted), 3), round(tp / (tp + fn), 3))

Output:

text

precision: 0.8 0.8
recall:    0.842 0.842

F1 combines the two

F1 is the harmonic mean of precision and recall. It is only high when both are high, so it suits classes of very different sizes. The dummy model from the first section scores zero.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
from sklearn.model_selection import train_test_split

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
predicted = LogisticRegression().fit(X_train, y_train).predict(X_test)
print("F1:", round(f1_score(y_test, predicted), 3))

top = (learners["score"] >= 75).astype(int)
print("always no, F1:", f1_score(top, np.zeros(len(top), dtype=int), zero_division=0))

Output:

text

F1: 0.821
always no, F1: 0.0

Everything at once

classification_report prints precision, recall, and F1 for every class, plus support: how many test rows belong to that class.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
predicted = LogisticRegression().fit(X_train, y_train).predict(X_test)
print(classification_report(y_test, predicted, target_names=["not passed", "passed"]))

Output:

text

              precision    recall  f1-score   support

  not passed       0.85      0.81      0.83        21
      passed       0.80      0.84      0.82        19

    accuracy                           0.82        40
   macro avg       0.82      0.83      0.82        40
weighted avg       0.83      0.82      0.83        40

Key takeaways

Accuracy hides which mistakes were made and misleads on unbalanced classes
The confusion matrix has actual classes as rows and predictions as columns
Precision: of the predicted yes, how many were right
Recall: of the real yes, how many were found
F1 is high only when precision and recall are both high
Next: K-Nearest Neighbours
Previous: Logistic Regression

Loading playground…

Check your understanding

Measuring Classification Quiz

Why can accuracy mislead?

1 / 10

← Logistic Regression

K-Nearest Neighbours →