Skip to main content
MahiDev — Learn. Build. Become AI-ready.

Lesson 10

Random Forests

Combine many decision trees with RandomForestClassifier, see how their votes beat a single tree, and tune n_estimators and max_depth for new data.

Intermediate25 min
Random Forests hero Module 27 Lesson 10Random Forests hero Module 27 Lesson 10

One deep tree memorises its training rows and makes quirky mistakes. A random forest trains many trees, each on a random resample of the rows and with a random choice of features at every split. Their quirks point in different directions, so when they vote, the mistakes tend to cancel out.

Seven small trees each voting pass or fail, combined into one majority answerSeven small trees each voting pass or fail, combined into one majority answer

Many trees, one answer

The training sets lesson showed an unlimited tree predicting scores: nearly perfect on its own rows and poor on new ones. Forests predict numbers too, with RandomForestRegressor, which averages the trees instead of counting votes. Same split, same features:

python
import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor

X = learners[["hours", "practice"]]
y = learners["score"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
tree = DecisionTreeRegressor(random_state=0).fit(X_train, y_train)
forest = RandomForestRegressor(n_estimators=100, random_state=0).fit(X_train, y_train)

print(f"tree:   train {tree.score(X_train, y_train):.3f}  test {tree.score(X_test, y_test):.3f}")
print(f"forest: train {forest.score(X_train, y_train):.3f}  test {forest.score(X_test, y_test):.3f}")

Output:

text
tree:   train 0.999  test 0.442
forest: train 0.957  test 0.758

The rest of this lesson goes back to classifying who passed.

Look inside the forest

estimators_ holds the fitted trees. For a borderline learner the individual trees disagree, and the forest averages their probabilities into one answer.

python
import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
forest = RandomForestClassifier(n_estimators=100, random_state=0).fit(X_train, y_train)
learner = X_test.iloc[[23]]

print(learner)
print(len(forest.estimators_))
print([int(tree.predict(learner.to_numpy())[0]) for tree in forest.estimators_[:10]])
print(forest.predict_proba(learner))

Output:

text
    hours  practice
73    7.7        22
100
[0, 1, 1, 0, 0, 1, 0, 1, 0, 0]
[[0.44 0.56]]

to_numpy() is there because the inner trees were fitted on plain arrays by the forest, so they do not know the column names.

How many trees

More trees make the vote steadier, but the gains flatten out quickly and every extra tree costs time, so a hundred or so is a common default. With only 40 test rows, one learner moves the score by 0.025, so small ups and downs between rows here are noise.

python
import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
for count in [1, 5, 25, 100, 300]:
    forest = RandomForestClassifier(n_estimators=count, random_state=0).fit(X_train, y_train)
    print(f"{count:>3} trees: test {forest.score(X_test, y_test):.3f}")

Output:

text
  1 trees: test 0.825
  5 trees: test 0.800
 25 trees: test 0.850
100 trees: test 0.825
300 trees: test 0.825

Limit each tree

The trees inside a forest accept the same settings as a single tree. Capping max_depth narrows the train and test gap.

python
import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
for depth in [2, 4, None]:
    forest = RandomForestClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
    print(f"depth {str(depth):>4}  train {forest.score(X_train, y_train):.3f}  test {forest.score(X_test, y_test):.3f}")

Output:

text
depth    2  train 0.869  test 0.825
depth    4  train 0.881  test 0.800
depth None  train 1.000  test 0.825

Importance across all trees

feature_importances_ averages over every tree, which makes it steadier than a single tree's. The unrelated video_minutes column still gets some credit, because deep trees split on noise too.

python
import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X = learners[["hours", "practice", "video_minutes"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
forest = RandomForestClassifier(random_state=0).fit(X_train, y_train)
for name, share in zip(X.columns, forest.feature_importances_):
    print(f"{name}: {share:.2f}")

Output:

text
hours: 0.54
practice: 0.23
video_minutes: 0.23

Key takeaways

  • A random forest trains many trees on random resamples and averages their votes

  • Forests usually generalise better than a single deep tree

  • estimators_ holds the individual fitted trees

  • n_estimators sets the number of trees; gains flatten after about a hundred

  • Tree settings such as max_depth apply to every tree in the forest

  • Next: Feature Scaling

  • Previous: Decision Trees

Loading playground…

Check your understanding

Random Forests Quiz

What is a random forest?

1 / 10

Previous

← Decision Trees

Next

Feature Scaling →