Skip to main content
MahiDev — Learn. Build. Become AI-ready.

Lesson 14

Pipelines

Chain preprocessing and a model into one object with Pipeline, handle numeric and text columns together with ColumnTransformer, and predict raw rows.

Intermediate25 min
Pipelines hero Module 27 Lesson 14Pipelines hero Module 27 Lesson 14

Scaling, encoding, and filling gaps each need a fit on training rows and a transform on every other set. Doing that by hand is easy to get wrong. A pipeline bundles the steps and the model into one estimator: fit runs every step in order on the training rows, and predict pushes new raw rows through the same steps automatically.

A pipeline passing raw columns through a column transformer into a modelA pipeline passing raw columns through a column transformer into a model

Pipeline chains steps

A Pipeline is a list of named steps. Every step but the last must transform; the last is the model. The whole thing then behaves like one model.

python
import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X = learners[["hours", "practice", "video_minutes"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
pipe = Pipeline([("scale", StandardScaler()), ("knn", KNeighborsClassifier())])
pipe.fit(X_train, y_train)
print("test accuracy:", pipe.score(X_test, y_test))

Output:

text
test accuracy: 0.85

That is the same score the scaling lesson reached with four lines of manual fit and transform.

make_pipeline names steps for you

make_pipeline skips the names and uses each class name in lower case. named_steps lets you reach any step after fitting.

python
import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

pipe = make_pipeline(StandardScaler(), KNeighborsClassifier())
pipe.fit(learners[["hours", "practice", "video_minutes"]], learners["passed"])

print(list(pipe.named_steps))
print(pipe.named_steps["standardscaler"].mean_.round(1))

Output:

text
['standardscaler', 'kneighborsclassifier']
[   6.7   20.9 1542.9]

ColumnTransformer for mixed columns

Numbers and text need different treatment. ColumnTransformer sends each group of columns to its own transformer and glues the results back together side by side.

python
import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

prepare = ColumnTransformer([
    ("numbers", StandardScaler(), ["hours", "practice", "video_minutes"]),
    ("text", OneHotEncoder(handle_unknown="ignore"), ["track"]),
])
table = prepare.fit_transform(learners[["hours", "practice", "video_minutes", "track"]])

print(table.shape)
print(prepare.get_feature_names_out())
print(table[:2].round(2))

Output:

text
(200, 6)
['numbers__hours' 'numbers__practice' 'numbers__video_minutes'
 'text__track_Data' 'text__track_Python' 'text__track_Web']
[[ 0.61 -1.39  1.41  0.    0.    1.  ]
 [-0.67 -1.64 -1.03  1.    0.    0.  ]]

One object from raw rows to answers

Put the column transformer and a model in a pipeline, fit it on the raw training DataFrame, and predict straight from raw rows, text and all.

python
import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

X = learners[["hours", "practice", "video_minutes", "track"]]
y = learners["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
prepare = ColumnTransformer([
    ("numbers", StandardScaler(), ["hours", "practice", "video_minutes"]),
    ("text", OneHotEncoder(handle_unknown="ignore"), ["track"]),
])
pipe = make_pipeline(prepare, LogisticRegression())
pipe.fit(X_train, y_train)
print("test accuracy:", pipe.score(X_test, y_test))

newcomers = pd.DataFrame({
    "hours": [4.0, 9.5], "practice": [15, 30],
    "video_minutes": [600, 1200], "track": ["Web", "Data"],
})
print(pipe.predict(newcomers))

Output:

text
test accuracy: 0.875
[0 1]

Reach inside a fitted pipeline

Index a pipeline like a list, or use named_steps, to inspect any part. Here the model's coefficients line up with the transformer's output names.

python
import numpy as np
import pandas as pd

rng = np.random.default_rng(27)
learners = pd.DataFrame({
    "hours": rng.uniform(1, 12, 200).round(1),
    "practice": rng.integers(0, 41, 200),
    "video_minutes": rng.integers(0, 3001, 200),
    "track": rng.choice(["Python", "Data", "Web"], 200),
})
bonus = learners["track"].map({"Python": 0, "Data": 6, "Web": -5})
noise = rng.normal(0, 6, 200)
learners["score"] = (
    22 + learners["hours"] * 3.4 + learners["practice"] * 0.55 + bonus + noise
).clip(0, 100).round()
learners["passed"] = (learners["score"] >= 60).astype(int)

from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

prepare = ColumnTransformer([
    ("numbers", StandardScaler(), ["hours", "practice", "video_minutes"]),
    ("text", OneHotEncoder(handle_unknown="ignore"), ["track"]),
])
pipe = make_pipeline(prepare, LogisticRegression())
pipe.fit(learners[["hours", "practice", "video_minutes", "track"]], learners["passed"])

names = pipe[0].get_feature_names_out()
weights = pipe.named_steps["logisticregression"].coef_[0]
for name, weight in zip(names, weights):
    print(f"{name:>22}: {weight:+.2f}")

Output:

text
        numbers__hours: +2.77
     numbers__practice: +1.46
numbers__video_minutes: -0.23
      text__track_Data: +1.25
    text__track_Python: -0.12
       text__track_Web: -1.12

Key takeaways

  • A Pipeline chains transformers and a final model into one estimator

  • fit runs every step on training rows; predict reuses them on new rows

  • make_pipeline names steps after their classes in lower case

  • ColumnTransformer applies different steps to different columns

  • Pipelines prevent leaks and let you predict straight from raw rows

  • Next: Cross-Validation

  • Previous: Missing Values

Loading playground…

Check your understanding

Pipelines Quiz

What problem do pipelines solve?

1 / 10

Previous

← Missing Values

Next

Cross-Validation →