Lesson 12

Ensemble Methods

Combine multiple models with bagging, voting, and stacking to reduce variance and improve accuracy beyond any single estimator.

Intermediate25 min

Three models — tree, logistic regression, KNN — each predicting into a final vote

A single model can fail in ways that a diverse group would not. Ensemble methods combine predictions from multiple models. The diversity between models is key: if all models make the same errors, combining them does not help.

Bagging

BaggingClassifier trains many copies of the same model on different random subsets of the training data (with replacement). It reduces variance because errors on random subsets tend to cancel out.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["senior"].values
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
tree = DecisionTreeClassifier(max_depth=4, random_state=0)
bag = BaggingClassifier(DecisionTreeClassifier(max_depth=4),
                        n_estimators=50, random_state=0)
tree.fit(X_train, y_train)
bag.fit(X_train, y_train)
print(f"single tree: {tree.score(X_test, y_test):.3f}")
print(f"bagging:     {bag.score(X_test, y_test):.3f}")

Output:

text

single tree: 0.883
bagging:     0.917

Voting classifier

VotingClassifier with voting='soft' averages the predicted probabilities from each estimator. Mixing different algorithms — a tree, logistic regression, and KNN — is more effective than duplicating the same one.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.ensemble import VotingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["senior"].values
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
tree = DecisionTreeClassifier(max_depth=4, random_state=0)
lr = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
knn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=11))
vote = VotingClassifier([("tree", tree), ("lr", lr), ("knn", knn)], voting="soft")
for name, clf in [("single tree", tree), ("bagging (50)", None), ("voting", vote)]:
    if clf is None:
        from sklearn.ensemble import BaggingClassifier
        clf = BaggingClassifier(DecisionTreeClassifier(max_depth=4), n_estimators=50, random_state=0)
    clf.fit(X_train, y_train)
    print(f"{name:11s}: {clf.score(X_test, y_test):.3f}")

Output:

text

single tree: 0.883
bagging    : 0.917
voting     : 0.933

Key takeaways

Ensembles combine diverse models to reduce error
Bagging trains the same model on different random subsets of the data
Random forests are bagged decision trees with random feature selection
VotingClassifier averages predictions from different algorithm types
Diversity between models is essential — identical models gain nothing

Loading playground…

Check your understanding

Ensemble Methods Quiz

What makes an ensemble better than a single model?

1 / 10

← Learning Curves

Model Selection →