Lesson 04
Regularization
Add L1 or L2 penalties to prevent overfitting, understand how Ridge keeps all features and Lasso zeros some out, and tune the regularization strength.
Intermediate25 min
Regularisation adds a penalty to the loss function for large coefficients. This discourages the model from memorising noise and improves generalisation. Ridge shrinks all coefficients; Lasso can zero some out entirely.
Ridge (L2 penalty)
Ridge adds the sum of squared coefficients to the loss:
[ \text{loss} = \text{MSE} + \alpha \sum w_i^2 ]
A larger alpha means stronger shrinkage. All features stay in the model.
import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
experience=experience, certifications=certifications,
overtime=overtime, education=education, salary=salary,
senior=senior, top=top,
))
from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
X = employees[["experience", "certifications", "overtime", "education"]]
y = employees["salary"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
for alpha in [0, 500, 2000, 10000]:
pipe = make_pipeline(StandardScaler(), Ridge(alpha=alpha))
pipe.fit(X_train, y_train)
print(f"alpha {alpha:5d} train {pipe.score(X_train, y_train):.3f} test {pipe.score(X_test, y_test):.3f}")Output:
alpha 0 train 0.944 test 0.931
alpha 500 train 0.520 test 0.490
alpha 2000 train 0.196 test 0.162
alpha 10000 train 0.045 test 0.010As alpha grows past the sweet spot, training and test scores both fall.
Lasso (L1 penalty)
Lasso adds the sum of absolute coefficients:
[ \text{loss} = \text{MSE} + \alpha \sum |w_i| ]
The geometry of the L1 penalty creates corners where some coefficients become exactly zero. Lasso therefore does automatic feature selection.
import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
experience=experience, certifications=certifications,
overtime=overtime, education=education, salary=salary,
senior=senior, top=top,
))
from sklearn.linear_model import Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
import numpy as np
X = employees[["experience", "certifications", "overtime", "education"]]
y = employees["salary"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
pipe = make_pipeline(StandardScaler(), Lasso(alpha=800, max_iter=5000))
pipe.fit(X_train, y_train)
coef = pipe.named_steps["lasso"].coef_
names = X.columns.tolist()
for name, c in zip(names, coef):
print(f"{name:15s}: {c:9.1f}")
print(f"test R2: {pipe.score(X_test, y_test):.3f}")Output:
experience : 16945.6
certifications : 1601.4
overtime : 280.2
education : 2208.8
test R2: 0.915Ridge vs Lasso comparison
Running both side by side makes the difference clear:
import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
experience=experience, certifications=certifications,
overtime=overtime, education=education, salary=salary,
senior=senior, top=top,
))
from sklearn.linear_model import Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
X = employees[["experience", "certifications", "overtime", "education"]]
y = employees["salary"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
for name, model in [("Ridge", Ridge(alpha=1000)), ("Lasso", Lasso(alpha=500, max_iter=5000))]:
pipe = make_pipeline(StandardScaler(), model)
pipe.fit(X_train, y_train)
print(f"{name} train {pipe.score(X_train, y_train):.3f} test {pipe.score(X_test, y_test):.3f}")Output:
Ridge train 0.336 test 0.304
Lasso train 0.942 test 0.923Key takeaways
- Regularisation adds a coefficient penalty to the loss to prevent overfitting
- Ridge (L2) shrinks all coefficients but keeps them non-zero
- Lasso (L1) can shrink some coefficients to exactly zero (feature selection)
- alpha controls regularisation strength; cross-validate to find the best value
- ElasticNet mixes L1 and L2 for a middle ground
Loading playground…
Check your understanding
Regularization Quiz
What does regularisation add to the loss function?
1 / 10