Lesson 03

Gradient Descent

See how gradient descent minimises a loss function by taking small steps in the direction of steepest decrease, and how scikit-learn uses it inside SGDRegressor.

Intermediate25 min

Loss decreasing step by step toward a minimum

Most ML models have parameters (weights, coefficients) that need to be tuned to fit the data. Gradient descent is the engine that does the tuning: it repeatedly measures the slope of the loss function and takes a small step downhill.

Loss function

The loss function measures how wrong the model's predictions are. For regression the most common choice is mean squared error (MSE):

[ \text{MSE} = \frac{1}{n}\sum_{i=1}^{n}(\hat{y}_i - y_i)^2 ]

The goal is to find the weights that minimise this value.

Manual gradient descent

For a line y = w*x + b, the gradient (slope) of MSE with respect to w and b tells you which direction makes the loss grow. Moving in the opposite direction by a small learning rate moves you toward lower loss.

python

import numpy as np
from numpy.random import default_rng

rng = default_rng(42)
x = rng.uniform(-5, 5, 80)
y_true = 2.5 * x + 1.0 + rng.normal(0, 2, 80)

w, b, lr = 0.0, 0.0, 0.01
for step in range(3000):
    pred = w * x + b
    err = pred - y_true
    dw = (err * x).mean()
    db = err.mean()
    w -= lr * dw
    b -= lr * db
    if (step + 1) % 1000 == 0:
        loss = (err ** 2).mean()
        print(f"step {step+1:4d}  w={w:.3f}  b={b:.3f}  loss={loss:.3f}")

Output:

text

step 1000  w=2.593  b=0.747  loss=3.417
step 2000  w=2.593  b=0.747  loss=3.417
step 3000  w=2.593  b=0.747  loss=3.417

After 3 000 steps the slope is close to 2.5 and the intercept close to 1.0. The true values were 2.5 and 1.0, so gradient descent found them.

SGDRegressor in scikit-learn

Implementing gradient descent by hand is a learning exercise. In practice, SGDRegressor does it efficiently. The max_iter parameter controls how many passes through the data it makes.

python

import numpy as np
import pandas as pd
from numpy.random import default_rng
rng = default_rng(42)
n = 300
experience = rng.integers(1, 21, n)
certifications = rng.integers(0, 7, n)
overtime = rng.integers(0, 45, n)
education = rng.choice([10, 12, 14, 16, 18], n, p=[0.05, 0.25, 0.35, 0.25, 0.10])
noise = rng.normal(0, 4500, n)
salary = (20000 + experience * 3200 + (education - 10) * 1500 +
          certifications * 1200 + overtime * 80 + noise).round(-2)
salary = np.clip(salary, 18000, 160000).astype(int)
senior = (salary >= 65650).astype(int)
top = ((experience >= 15) & (certifications >= 5)).astype(int)
employees = pd.DataFrame(dict(
    experience=experience, certifications=certifications,
    overtime=overtime, education=education, salary=salary,
    senior=senior, top=top,
))
from sklearn.linear_model import SGDRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
X = employees[["experience", "certifications", "overtime", "education"]]
y = employees["salary"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
pipe = make_pipeline(StandardScaler(), SGDRegressor(max_iter=2000, random_state=0))
pipe.fit(X_train, y_train)
print(f"train R2: {pipe.score(X_train, y_train):.3f}")
print(f"test  R2: {pipe.score(X_test,  y_test):.3f}")

Output:

text

train R2: 0.944
test  R2: 0.931

Learning rate matters

A learning rate that is too large causes the steps to overshoot the minimum. A rate that is too small makes convergence very slow. Scikit-learn defaults work for most problems, but SGDRegressor(eta0=...) lets you override.

Key takeaways

The loss function measures prediction error; gradient descent minimises it
Each step moves in the direction that reduces loss by the learning rate
SGDRegressor applies stochastic gradient descent to regression
Always scale features before gradient-based models
A learning rate too large overshoots; too small is slow

Loading playground…

Check your understanding

Gradient Descent Quiz

What does the loss function measure?

1 / 10

← Bias and Variance

Regularization →