Lesson 06

Naive Bayes

Apply Bayes' theorem to classification, understand the naive independence assumption, and use GaussianNB for continuous features.

Intermediate25 min

Probability bars showing how each feature contributes to predicting senior

Naive Bayes uses Bayes' theorem to compute the probability of each class given the features, then picks the most probable class. It is called 'naive' because it assumes each feature is independent of the others — rarely true, but often good enough.

Bayes' theorem

[ P(class | features) \propto P(class) \times \prod_i P(feature_i | class) ]

The prior P(class) is the class frequency. Each P(feature|class) is estimated from the training data. For continuous features, GaussianNB models each one as a Gaussian distribution.

GaussianNB

GaussianNB fits a Gaussian (normal distribution) for each feature within each class. To predict, it multiplies the per-feature probabilities and picks the highest class.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import train_test_split, cross_val_score
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["senior"].values
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
nb = GaussianNB()
nb.fit(X_train, y_train)
print(f"test accuracy: {nb.score(X_test, y_test):.3f}")
cv = cross_val_score(nb, X, y, cv=5).round(3)
print(f"cv scores: {cv}")
print(f"mean: {cv.mean():.3f}")

Output:

text

test accuracy: 0.950
cv scores: [0.967 0.9   0.933 0.917 0.9  ]
mean: 0.923

Class means learned by GaussianNB

theta_ holds the feature means per class. Comparing them shows which features differ most between senior and non-senior employees.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.naive_bayes import GaussianNB
import numpy as np
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["senior"].values
nb = GaussianNB().fit(X, y)
names = ["experience", "certifications", "overtime", "education"]
print("              junior  senior")
for name, jr, sr in zip(names, nb.theta_[0], nb.theta_[1]):
    print(f"{name:15s}  {jr:5.1f}   {sr:5.1f}")

Output:

text

              junior  senior
experience         6.0    14.9
certifications     2.7     3.3
overtime          22.6    22.7
education         14.0    14.4

Senior employees have more experience and certifications on average. GaussianNB detected this automatically.

Key takeaways

Naive Bayes uses Bayes' theorem to estimate class probability from feature probabilities
The naive assumption: features are independent given the class
GaussianNB models each continuous feature as a Gaussian per class
Naive Bayes is fast, works with small data, and handles many features well
theta_ holds the learned per-class feature means

Loading playground…

Check your understanding

Naive Bayes Quiz

What does Bayes' theorem compute for a classifier?

1 / 10

← Support Vector Machines

Gradient Boosting →