Skip to main content
MahiDev — Learn. Build. Become AI-ready.

Lesson 15

Model Monitoring

Detect data drift and concept drift after deployment, monitor prediction distributions over time, and know when to retrain.

Intermediate25 min
Prediction rate stable then dropping after a distribution shift event

Deploying a model is not the end of the ML workflow. Production data changes over time: the feature distributions shift (data drift) or the relationship between features and labels changes (concept drift). Either way, accuracy falls without retraining.

Simulating data drift

Data drift is a change in the input feature distribution. If a model was trained on experienced employees and then deployed against new hires, the feature means shift, and the model's predictions become unreliable.

python
import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["senior"].values
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
clf = GradientBoostingClassifier(n_estimators=100, random_state=0).fit(X_train, y_train)
# Simulate distribution shift: new hires, 5 years less experience on average
X_shifted = X_test.copy()
X_shifted[:, 0] = np.clip(X_shifted[:, 0] - 5, 1, 20)
print(f"original experience mean: {X_test[:, 0].mean():.1f}")
print(f"shifted  experience mean: {X_shifted[:, 0].mean():.1f}")
print(f"original % predicted senior: {(clf.predict(X_test) == 1).mean():.2f}")
print(f"shifted  % predicted senior: {(clf.predict(X_shifted) == 1).mean():.2f}")

Output:

text
original experience mean: 10.5
shifted  experience mean: 6.2
original % predicted senior: 0.48
shifted  % predicted senior: 0.23

The model predicts fewer seniors on the shifted data, even though no label changed. This is data drift in action.

Monitoring with population stability index

A simple drift detector compares the mean and standard deviation of each feature in training versus the latest production data. A large difference signals drift. Formal tests (Kolmogorov-Smirnov, PSI) make this quantitative.

python
import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
import numpy as np
from sklearn.model_selection import train_test_split
X = employees[["experience", "certifications", "overtime", "education"]].values
y = employees["senior"].values
X_train, X_test, _, _ = train_test_split(X, y, test_size=0.2, random_state=42)
# Simulate a shifted production window
X_shifted = X_test.copy()
X_shifted[:, 0] = np.clip(X_shifted[:, 0] - 5, 1, 20)
names = ["experience", "certifications", "overtime", "education"]
print(f"{'feature':15s}  train_mean  prod_mean  delta")
for i, name in enumerate(names):
    train_m = X_train[:, i].mean()
    prod_m = X_shifted[:, i].mean()
    print(f"{name:15s}  {train_m:9.2f}  {prod_m:9.2f}  {abs(train_m - prod_m):5.2f}")

Output:

text
feature          train_mean  prod_mean  delta
experience           10.30       6.70   3.60
certifications        2.98       3.15   0.17
overtime             22.40      23.60   1.20
education            14.22      14.10   0.12

When to retrain

A practical retraining trigger is a drop in a tracked metric (accuracy, F1, or RMSE on labelled production samples) beyond a threshold. Without labels, use the prediction distribution: if the fraction of positive predictions drifts significantly from the training baseline, trigger a review.

A simple strategy:

  1. Save the model's output distribution on a labelled holdout at deploy time.
  2. Weekly, score the holdout with the live model.
  3. If the score drops more than 2–3 percent, retrain on new data.

Key takeaways

  • Data drift: the input feature distribution changes after deployment
  • Concept drift: the relationship between features and labels changes
  • Monitor feature means and prediction distributions in production
  • A significant drop in a tracked metric is a retraining trigger
  • The ML workflow is a loop that continues after deployment

Loading playground…

Check your understanding

Model Monitoring Quiz

What is data drift?

1 / 10

Previous

← ML Workflow