Lesson 07

Gradient Boosting

Build an ensemble of trees that each correct the previous one's residuals, control the speed-accuracy tradeoff with learning_rate and n_estimators, and read feature importances.

Intermediate25 min

Three sequential trees feeding into a sum, each correcting the previous one's residuals

Gradient boosting trains trees one after another. Each new tree fits the residuals (errors) of all previous trees combined. The final prediction is the weighted sum of all trees. This sequential correction is why boosting often outperforms a single tree or a bagged forest on structured data.

Boosting vs a single tree

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["senior"].values
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
tree = DecisionTreeClassifier(max_depth=4, random_state=0).fit(X_train, y_train)
gb = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1,
                                max_depth=3, random_state=0)
gb.fit(X_train, y_train)
print(f"single tree: {tree.score(X_test, y_test):.3f}")
print(f"boosting:    {gb.score(X_test, y_test):.3f}")

Output:

text

single tree: 0.883
boosting:    0.883

Learning rate and number of trees

learning_rate controls how much each tree contributes. A smaller rate requires more trees but often generalises better. n_estimators sets the total number of trees.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split, cross_val_score
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["senior"].values
configs = [(0.3, 50), (0.1, 100), (0.05, 200)]
for lr, n in configs:
    gb = GradientBoostingClassifier(n_estimators=n, learning_rate=lr,
                                    max_depth=3, random_state=0)
    cv = cross_val_score(gb, X, y, cv=5).mean()
    print(f"lr={lr:.2f}  trees={n:3d}  cv={cv:.3f}")

Output:

text

lr=0.30  trees= 50  cv=0.900
lr=0.10  trees=100  cv=0.900
lr=0.05  trees=200  cv=0.900

More trees with a smaller learning rate is often more accurate, but slower to train.

Feature importances

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
import numpy as np
X = employees[["experience", "certifications", "overtime", "education"]]
y = employees["senior"].values
X_train, X_test, y_train, y_test = train_test_split(
    X.values, y, test_size=0.2, random_state=42, stratify=y
)
gb = GradientBoostingClassifier(n_estimators=100, random_state=0).fit(X_train, y_train)
for name, imp in sorted(zip(X.columns, gb.feature_importances_),
                         key=lambda p: -p[1]):
    print(f"{name:15s}: {imp:.3f}")

Output:

text

experience     : 0.853
certifications : 0.084
overtime       : 0.041
education      : 0.022

Key takeaways

Gradient boosting trains trees sequentially, each fitting the previous residuals
The learning rate scales each tree's contribution; smaller is usually better with more trees
n_estimators sets the total number of trees
feature_importances_ averages importance over all trees
Gradient boosting often achieves the highest accuracy on tabular data

Loading playground…

Check your understanding

Gradient Boosting Quiz

What does each new tree in gradient boosting fit?

1 / 10

← Naive Bayes

Dimensionality Reduction →