Lesson 05

Support Vector Machines

Understand how SVMs find the widest margin between classes, how the C parameter balances margin and misclassification, and how kernels extend SVMs to non-linear problems.

Intermediate25 min

Two clusters separated by a decision boundary with support vectors highlighted

A support vector machine draws a decision boundary that maximises the margin — the gap between the nearest examples of each class (the support vectors). A wider margin typically generalises better.

Linear SVM

SVC(kernel='linear') finds the widest straight line (or hyperplane in higher dimensions) that separates the classes. The C parameter controls the tradeoff: a small C allows some misclassifications in exchange for a wider margin.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
X = employees[["experience", "certifications"]].values
y = employees["senior"].values
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
for kernel in ["linear", "rbf"]:
    pipe = make_pipeline(StandardScaler(), SVC(kernel=kernel, C=1.0))
    pipe.fit(X_train, y_train)
    print(f"{kernel:6s}  test {pipe.score(X_test, y_test):.3f}")

Output:

text

linear  test 0.900
rbf     test 0.917

Kernel trick for non-linear data

When classes are not linearly separable, the kernel trick implicitly maps the data to a higher-dimensional space where a linear boundary works. The RBF (radial basis function) kernel is the most common choice and often handles complex boundaries well.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split, cross_val_score
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["senior"].values
for C in [0.1, 1.0, 10.0]:
    pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=C))
    cv = cross_val_score(pipe, X, y, cv=5).mean()
    print(f"C={C:4.1f}  cv accuracy {cv:.3f}")

Output:

text

C= 0.1  cv accuracy 0.923
C= 1.0  cv accuracy 0.913
C=10.0  cv accuracy 0.910

Feature scaling is essential for SVMs — different scales distort the margin.

Key takeaways

SVMs find the decision boundary that maximises the margin between classes
Support vectors are the training examples closest to the boundary
C controls the margin-misclassification tradeoff: smaller C → wider margin
Kernels (RBF, poly) extend SVMs to non-linear boundaries
Always scale features before SVC

Loading playground…

Check your understanding

Support Vector Machines Quiz

What does the margin in an SVM measure?

1 / 10

← Regularization

Naive Bayes →