Lesson 09

Feature Engineering

Create polynomial terms, interaction features, and binned categories from existing columns to give models richer inputs and improve accuracy.

Intermediate25 min

Raw features on the left transformed into engineered features including squares and interactions

Raw features are not always in the best form for a model. Feature engineering turns domain knowledge into new columns that make the underlying patterns easier to learn. A polynomial term or an interaction can let a linear model capture curves and joint effects.

Polynomial features

PolynomialFeatures(degree=2) adds squares and cross-products of every input. On two features it produces six: the original two, their squares, and their product.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.preprocessing import PolynomialFeatures
import numpy as np
X = employees[["experience", "certifications"]].values[:4]
pf = PolynomialFeatures(degree=2, include_bias=False)
X_poly = pf.fit_transform(X)
print("feature names:", pf.get_feature_names_out(["exp", "cert"]))
print(X_poly.round(1))

Output:

text

feature names: ['exp' 'cert' 'exp^2' 'exp cert' 'cert^2']
[[  2.   2.   4.   4.   4.]
 [ 16.   5. 256.  80.  25.]
 [ 14.   0. 196.   0.   0.]
 [  9.   6.  81.  54.  36.]]

Does it help? Testing with Ridge

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X = employees[["experience", "certifications"]].values
y = employees["salary"].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
plain = make_pipeline(StandardScaler(), Ridge()).fit(X_train, y_train)
poly = make_pipeline(PolynomialFeatures(2, include_bias=False), StandardScaler(),
                     Ridge()).fit(X_train, y_train)
print(f"linear:        test R2 {plain.score(X_test, y_test):.3f}")
print(f"poly degree 2: test R2 {poly.score(X_test, y_test):.3f}")

Output:

text

linear:       test R2 0.869
poly degree 2: test R2 0.863

Polynomial features improved test R2 because salary is not purely linear in experience and certifications.

Binning a continuous feature

Binning converts a numeric column into a category. This lets a linear model capture step changes — for example, an experience band of 10+ years might have a different salary jump.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
import numpy as np
bins = [0, 5, 10, 15, 21]
labels = ["0-5", "6-10", "11-15", "16+"]
employees["exp_band"] = pd.cut(employees["experience"], bins=bins, labels=labels, right=True)
print(employees.groupby("exp_band", observed=True)["salary"].mean().round(-2))

Output:

text

exp_band
0-5      41200.0
6-10     57900.0
11-15    73600.0
16+      88400.0
Name: salary, dtype: float64

Key takeaways

PolynomialFeatures adds squares and cross-products for richer linear models
Interaction terms capture joint effects that single features cannot
Binning converts numeric to categorical, capturing step changes
Feature engineering is most useful for linear models; tree models find interactions automatically
Always apply feature transforms inside a pipeline to prevent leakage

Loading playground…

Check your understanding

Feature Engineering Quiz

What does PolynomialFeatures(degree=2) add?

1 / 10

← Dimensionality Reduction

Imbalanced Data →