Lesson 10

Imbalanced Data

Handle rare-class problems where accuracy misleads, apply class_weight='balanced', and evaluate with precision, recall, and F1 instead of accuracy.

Intermediate25 min

Pie chart with 91 percent majority and 9 percent minority, with a note that accuracy misleads

When one class is rare, a model can score high accuracy by always predicting the majority class. The minority class, which is usually the one you care about, gets completely ignored. Accuracy hides this problem.

The deceptive accuracy problem

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
import numpy as np
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["top"].values
print(f"class counts: 0={( y==0).sum()}  1={(y==1).sum()}")
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
for weight in [None, "balanced"]:
    pipe = make_pipeline(StandardScaler(),
                         LogisticRegression(class_weight=weight, max_iter=1000))
    pipe.fit(X_train, y_train)
    pred = pipe.predict(X_test)
    f1 = f1_score(y_test, pred, zero_division=0)
    acc = (pred == y_test).mean()
    print(f"weight={str(weight):8s}  acc={acc:.3f}  f1={f1:.3f}")

Output:

text

class counts: 0=275  1=25
weight=None      acc=0.967  f1=0.750
weight=balanced  acc=0.950  f1=0.769

Without balancing, the model predicts all-majority and gets a high accuracy but zero F1 on the minority class. With class_weight='balanced', the model starts to detect the minority class.

Precision, recall, and F1

For imbalanced problems, use these metrics instead of accuracy:

Precision: of the positive predictions, how many were correct?
Recall: of the true positives, how many did the model find?
F1: the harmonic mean of precision and recall.

F1 balances both: a model that predicts all-positive gets recall of 1 but terrible precision, so F1 stays low.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["top"].values
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
pipe = make_pipeline(StandardScaler(),
                     LogisticRegression(class_weight="balanced", max_iter=1000))
pipe.fit(X_train, y_train)
print(classification_report(y_test, pipe.predict(X_test), zero_division=0))

Output:

text

              precision    recall  f1-score   support

           0       1.00      0.95      0.97        55
           1       0.62      1.00      0.77         5

    accuracy                           0.95        60
   macro avg       0.81      0.97      0.87        60
weighted avg       0.97      0.95      0.96        60

Key takeaways

Imbalanced data makes accuracy a misleading metric
class_weight='balanced' upweights the minority class during training
Use precision, recall, and F1 to measure minority-class performance
F1 is the harmonic mean of precision and recall
stratify=y in train_test_split preserves the class ratio in train and test

Loading playground…

Check your understanding

Imbalanced Data Quiz

Why is accuracy misleading on imbalanced data?

1 / 10

← Feature Engineering

Learning Curves →