Lesson 09

Decision Trees

Build a DecisionTreeClassifier, read its questions with export_text, control memorising with max_depth, and see which features the tree relies on.

Intermediate25 min

A decision tree learns a flowchart of yes or no questions such as "hours <= 8.9?". Each answer leads to another question until a leaf gives the prediction. Trees are easy to read and need no scaling, but left unchecked they keep asking questions until they have memorised every training row.

A depth two decision tree splitting learners on hours and practice into pass and fail leaves

Questions in a tree

max_depth=2 allows two questions on any path. export_text prints the learned rules, with class: 1 meaning passed.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, export_text

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
tree = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X_train, y_train)
print(export_text(tree, feature_names=list(X.columns)))

Output:

text

|--- hours <= 8.95
|   |--- hours <= 4.45
|   |   |--- class: 0
|   |--- hours >  4.45
|   |   |--- class: 0
|--- hours >  8.95
|   |--- practice <= 6.00
|   |   |--- class: 1
|   |--- practice >  6.00
|   |   |--- class: 1

Both leaves on a side can give the same class. The second question still helped: it separated learners who are almost certain from those who are borderline, which shows up in predict_proba below.

Predict follows the branches

Each newcomer answers the questions and lands in a leaf. predict_proba is the share of training learners in that leaf who passed.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
tree = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X_train, y_train)
newcomers = pd.DataFrame({"hours": [3.0, 8.0, 11.0], "practice": [30, 10, 30]})

print(tree.predict(newcomers))
print(tree.predict_proba(newcomers).round(2))

Output:

text

[0 0 1]
[[0.96 0.04]
 [0.55 0.45]
 [0.   1.  ]]

Depth controls memorising

With no limit the tree grows until every leaf is pure, so training accuracy reaches 1.0. That perfect training score is memorisation: on the test rows the unlimited tree does no better than a depth of 3, which is far simpler and easier to trust.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
for depth in [1, 2, 3, 5, 8, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
    print(f"depth {str(depth):>4}  train {tree.score(X_train, y_train):.3f}  test {tree.score(X_test, y_test):.3f}")

Output:

text

depth    1  train 0.781  test 0.725
depth    2  train 0.781  test 0.725
depth    3  train 0.869  test 0.825
depth    5  train 0.919  test 0.775
depth    8  train 0.963  test 0.800
depth None  train 1.000  test 0.825

Which features matter

feature_importances_ shows how much each feature helped the splits. The values add up to 1.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
tree = DecisionTreeClassifier(max_depth=3, random_state=0).fit(X_train, y_train)
for name, share in zip(X.columns, tree.feature_importances_):
    print(f"{name}: {share:.2f}")

Output:

text

hours: 0.79
practice: 0.21

No scaling needed

A tree compares one feature with a threshold at a time, so large numbers do not drown out small ones. Given video_minutes, which has nothing to do with passing, this tree does not use it at all.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X = learners[["hours", "practice", "video_minutes"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
tree = DecisionTreeClassifier(max_depth=3, random_state=0).fit(X_train, y_train)
print("test accuracy:", tree.score(X_test, y_test))
for name, share in zip(X.columns, tree.feature_importances_):
    print(f"{name}: {share:.2f}")

Output:

text

test accuracy: 0.825
hours: 0.79
practice: 0.21
video_minutes: 0.00

Key takeaways

A decision tree is a flowchart of yes or no questions learned from data
export_text prints the rules the tree learned
Without max_depth a tree memorises the training rows
feature_importances_ shows which features drove the splits
Trees do not need feature scaling
Next: Random Forests
Previous: K-Nearest Neighbours

Loading playground…

Check your understanding

Decision Trees Quiz

What does a decision tree learn?

1 / 10

← K-Nearest Neighbours

Random Forests →