Lesson 08

K-Nearest Neighbours

Classify a learner by the votes of the most similar learners with KNeighborsClassifier, choose n_neighbors, and see why distance needs scale.

Intermediate25 min

K-Nearest Neighbours hero Module 27 Lesson 8

K-nearest neighbours is the simplest idea in this module. To classify a new learner, find the few training learners closest to them and let those neighbours vote. Fitting just stores the rows; all the work happens at prediction time, when distances are measured.

Pass and fail points with lines from a new point to its five nearest neighbours

Neighbours vote

KNeighborsClassifier uses 5 neighbours by default. The interface is the one you already know.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
model = KNeighborsClassifier().fit(X_train, y_train)
print(model.n_neighbors)
print("test accuracy:", model.score(X_test, y_test))

Output:

text

5
test accuracy: 0.875

Look at the neighbours

kneighbors returns the distances to the nearest training rows and their positions. Looking up their labels shows the vote behind a prediction.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
model = KNeighborsClassifier().fit(X_train, y_train)
newcomer = pd.DataFrame({"hours": [6.5], "practice": [18]})

distances, positions = model.kneighbors(newcomer)
print(distances.round(2))
print("votes:", y_train.iloc[positions[0]].tolist())
print("prediction:", model.predict(newcomer))

Output:

text

[[1.1  1.28 1.3  1.56 1.7 ]]
votes: [0, 0, 0, 0, 0]
prediction: [0]

Choosing n_neighbors

One neighbour copies the single closest learner, noise and all, so it is perfect on training rows and shakier on new ones. Very many neighbours blur everything towards the majority. Values in between usually work best.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

X = learners[["hours", "practice"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
for n in [1, 3, 5, 11, 25, 75]:
    model = KNeighborsClassifier(n_neighbors=n).fit(X_train, y_train)
    print(f"n={n:>2}  train {model.score(X_train, y_train):.3f}  test {model.score(X_test, y_test):.3f}")

Output:

text

n= 1  train 1.000  test 0.825
n= 3  train 0.850  test 0.875
n= 5  train 0.863  test 0.875
n=11  train 0.844  test 0.850
n=25  train 0.781  test 0.875
n=75  train 0.662  test 0.650

Distance needs comparable scales

Add video_minutes, which runs into the thousands. Distances are now almost entirely about video time, which has nothing to do with passing, and accuracy collapses. The scaling lesson fixes this.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

X = learners[["hours", "practice", "video_minutes"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
model = KNeighborsClassifier().fit(X_train, y_train)
print("with video_minutes:", model.score(X_test, y_test))
print(X_train.max().tolist())

Output:

text

with video_minutes: 0.575
[12.0, 40.0, 2979.0]

Neighbours for numbers too

KNeighborsRegressor averages the neighbours' values instead of voting, so it can predict a score.

python

import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.neighbors import KNeighborsRegressor

model = KNeighborsRegressor(n_neighbors=5).fit(learners[["hours", "practice"]], learners["score"])
newcomer = pd.DataFrame({"hours": [6.5], "practice": [18]})

_, positions = model.kneighbors(newcomer)
print(learners["score"].iloc[positions[0]].tolist())
print(model.predict(newcomer))

Output:

text

[42.0, 50.0, 36.0, 59.0, 53.0]
[48.]

Key takeaways

KNN predicts from the closest training rows
fit only stores the data; prediction measures distances
Small n_neighbors memorises noise; large n_neighbors blurs everything
Features on large scales dominate distance, so scale before KNN
KNeighborsRegressor averages the neighbours' values
Next: Decision Trees
Previous: Measuring Classification

Loading playground…

Check your understanding

K-Nearest Neighbours Quiz

How does KNN classify a new row?

1 / 10

← Measuring Classification

Decision Trees →