Lesson 08

Dimensionality Reduction

Use PCA to summarise many correlated features into a smaller set of uncorrelated principal components, read the explained variance ratio, and combine PCA with a classifier.

Intermediate25 min

Four-dimensional data projected onto two principal components

When features are correlated, much of the information is redundant. Principal Component Analysis (PCA) finds new axes — principal components — that capture the most variance. You keep the first few components and discard the rest, reducing noise and memory without losing much signal.

Explained variance

explained_variance_ratio_ tells you what fraction of the total variance each component captures. Components are sorted from highest to lowest, so the first one always explains the most.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np
X = employees[["experience", "certifications", "overtime", "education"]].values
X_scaled = StandardScaler().fit_transform(X)
pca = PCA(n_components=4)
pca.fit(X_scaled)
ratio = pca.explained_variance_ratio_
for i, r in enumerate(ratio, 1):
    print(f"PC{i}: {r:.3f}  cumulative: {ratio[:i].sum():.3f}")

Output:

text

PC1: 0.274  cumulative: 0.274
PC2: 0.260  cumulative: 0.534
PC3: 0.242  cumulative: 0.776
PC4: 0.224  cumulative: 1.000

The first two components capture most of the variance. Using two instead of four features loses little information.

Reducing dimensions

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np
X = employees[["experience", "certifications", "overtime", "education"]].values
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X_scaled)
print(f"original shape: {X.shape}")
print(f"reduced shape:  {X_2d.shape}")
print(f"variance kept:  {pca.explained_variance_ratio_.sum():.3f}")

Output:

text

original shape: (300, 4)
reduced shape:  (300, 2)
variance kept:  0.534

PCA in a pipeline

Wrapping PCA in a pipeline prevents leakage: the scaler and PCA fit on training rows only, then transform test rows with the same learned parameters.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["senior"].values
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
full = make_pipeline(StandardScaler(), KNeighborsClassifier(11))
reduced = make_pipeline(StandardScaler(), PCA(n_components=2), KNeighborsClassifier(11))
full.fit(X_train, y_train)
reduced.fit(X_train, y_train)
print(f"all 4 features: {full.score(X_test, y_test):.3f}")
print(f"2 components:   {reduced.score(X_test, y_test):.3f}")

Output:

text

all 4 features: 0.867
2 components:   0.717

Key takeaways

PCA finds uncorrelated components that explain the most variance
explained_variance_ratio_ shows how much each component contributes
Always scale features before PCA
Use PCA inside a pipeline to prevent leakage
PCA is useful for visualisation, noise reduction, and speeding up slow models

Loading playground…

Check your understanding

Dimensionality Reduction Quiz

What is the goal of PCA?

1 / 10

← Gradient Boosting

Feature Engineering →