Optimization

loss function

The Loss Function is the heart of Machine Learning.

A Loss Function measures how wrong a model's prediction is.

Think of it as the model's report card.

• Small loss → Good prediction
• Large loss → Poor prediction

Without a loss function, an AI model has no way to know whether it is improving.

Intuition: Your real estate price model

Suppose your ML model predicts house prices.

	Price
Actual	₹80 Lakh
Predicted	₹74 Lakh

The prediction is wrong by ₹6 lakh.

The loss function converts this error into a number the model can minimize.

The model learns by making this error smaller.

Why do we need a loss function?

Imagine a student solving math questions.

Question	Actual	Student
Q1	10	9
Q2	25	22
Q3	50	48

The teacher calculates the mistakes.

Similarly, ML calculates prediction mistakes.

That mistake is called loss.

General idea

A model does three steps repeatedly:

1. Make a prediction
2. Calculate loss
3. Update weights

This loop happens millions of times during training.

Loss vs Error

People often confuse these.

Error	Loss
Difference for one prediction	Mathematical value used for learning
80 − 74 = 6	Depends on loss formula
Human-friendly	Optimizer-friendly

Example:

Actual = 80

Predicted = 74

Error = 6

Loss may be:

• 6 (Absolute Error)
• 36 (Squared Error)

Types of Loss Functions

1. Mean Absolute Error (MAE)

Formula:

Example:

Actual	Predicted	Error
80	74	6
60	63	3
90	87	3

MAE:

Average mistake = 4

Use Case

• House price prediction
• Sales forecasting
• Rental prediction

2. Mean Squared Error (MSE)

Formula:

Same data:

Error	Squared
6	36
3	9
3	9

MSE penalizes larger mistakes more heavily.

Why square?

Suppose:

Error	Squared
2	4
10	100

A 10-unit mistake becomes 100.

This forces the model to avoid large errors.

3. Cross Entropy Loss

Used for classification.

Example:

Cat vs Dog.

Actual	Prediction
Cat	0.95
Dog	0.05

Good prediction → Low loss

Wrong prediction:

Actual	Prediction
Cat	0.10

Loss becomes very large.

This is why ChatGPT and image classifiers use cross entropy.

ML Use Case 1: Real Estate Price Prediction

Input:

• Area
• BHK
• Location

Output:

Predicted Price

The optimizer reduces this loss until predictions improve.

ML Use Case 2: Spam Detection

Actual:

Spam

Predicted probability:

0.98

Loss = Very low

Wrong prediction:

Actual = Spam

Prediction = 0.05

Loss = Very high

Cross entropy teaches the classifier to increase the correct probability.

ML Use Case 3: ChatGPT

Prompt:

Machine learning is...

Target word:

powerful

Model predicts:

Word	Probability
powerful	0.72
useful	0.18
difficult	0.10

Cross entropy compares the predicted distribution with the correct word and computes loss.

Then gradients update billions of parameters.

Training Curve

A good model shows decreasing loss.

Epoch	Loss
1	2.8
2	2.0
3	1.4
4	0.9
5	0.5

As epochs increase:

• Predictions improve
• Loss decreases
• Model learns

Python Example

from sklearn.metrics import mean_absolute_error

actual = [80, 60, 90]
pred = [74, 63, 87]

mae = mean_absolute_error(actual, pred)

print(mae)

Output:

4.0

The model makes an average mistake of 4 units.

Which loss function is used where?

ML Problem	Loss Function
House Price Prediction	MAE / MSE
Sales Forecasting	MAE
Image Classification	Cross Entropy
Spam Detection	Cross Entropy
ChatGPT / LLM	Cross Entropy
Regression	MSE or MAE

Summary

Concept	Meaning
Error	Difference between actual and predicted
Loss	Mathematical measure of prediction error
MAE	Average absolute error
MSE	Average squared error
Cross Entropy	Classification loss

Memory Trick

Prediction → Compare → Loss → Learn

The loss function is the compass that tells the model whether it's moving toward better predictions or away from them.

Interview Question

Q: Why is a loss function essential in Machine Learning?

Answer: A loss function quantifies how wrong a model's predictions are. Optimization algorithms like Gradient Descent use this value and its derivatives to update model parameters, allowing the model to learn from data and improve over time.

MSE

Mean Squared Error (MSE) is the most commonly used loss function for regression problems such as house price prediction, salary prediction, and sales forecasting.

MSE measures the average of the squared differences between actual and predicted values.

In simple words:

1. Find the prediction error.
2. Square the error.
3. Average all squared errors.

Why do we square the error?

Imagine your real estate AI predicts house prices.

House A

Actual: ₹80L

Predicted: ₹78L

Error = 2

House B

Actual: ₹80L

Predicted: ₹70L

Error = 10

If we use normal error:

• 2
• 10

The second error is only 5× larger.

After squaring:

Error	Squared
2	4
10	100

Now the large mistake becomes 100, so the model is punished much more heavily.

This is the main purpose of MSE.

MSE Formula

Symbol	Meaning
y	Actual value
ŷ	Predicted value (from weights)
w	Model weight
b	Bias
n	Number of observations
Σ	Sum
MSE	Mean Squared Error

Where ŷ comes from: weights

The predicted value ŷ is not magic — it comes from the model's weights.

For a simple linear model:

Symbol	Meaning
w	Weight (how much each feature matters)
x	Input feature (e.g. house area)
b	Bias (base offset)
ŷ	Prediction made using current weights

Example (house price):

• Area x = 1000 sq.ft
• Weight w = 0.05 (₹ lakhs per sq.ft)
• Bias b = 10

Then:

ŷ = 0.05 × 1000 + 10 = 60

So the model predicts ₹60L.

How MSE updates weights

MSE tells us how wrong the current weights are.

Gradient Descent then adjusts weights to reduce MSE:

Where:

• η (eta) = learning rate (step size)
• ∂MSE/∂w = how much MSE changes when w changes

Intuition:

1. Weights make a prediction ŷ.
2. MSE measures (y − ŷ)².
3. If MSE is high, update the weights.
4. Repeat until predictions get closer to actual values.

Weights are the knobs the optimizer turns. MSE is the score that says whether those knobs are better or worse.

Step-by-step calculation

Suppose your model predicts 3 house prices.

Actual (y)	Predicted (ŷ)
80	74
60	63
90	87

Step 1: Calculate Error

Actual	Predicted	Error
80	74	6
60	63	-3
90	87	3

Step 2: Square the Error

Error	Squared Error
6	36
-3	9
3	9

Notice:

• Negative disappears.
• Large errors become much larger.

Step 3: Average

Final Answer: MSE = 18

Visual intuition

The model wants to reach the minimum loss.

Gradient Descent tries to reduce MSE from 18 → 10 → 4 → 0.5.

Why MSE is better than MAE?

Consider two models.

Model A

Errors: 2, 2, 2

MSE = 4

Model B

Errors: 1, 1, 6

MSE = 12.7

Although both models have similar average errors, Model B made one very large mistake.

MSE penalizes that strongly.

This helps regression models become more accurate.

ML Use Case 1: House Price Prediction

Features:

• Area
• BHK
• Location

Output:

Predicted Price

The optimizer minimizes MSE until predicted prices match actual prices.

ML Use Case 2: Sales Forecasting

Suppose actual monthly sales:

Month	Actual	Predicted
Jan	120	118
Feb	150	142
Mar	180	176

MSE measures how accurate the forecasting model is across all months.

ML Use Case 3: Temperature Prediction

Weather AI predicts:

City	Actual	Predicted
A	30°	29°
B	28°	25°
C	35°	34°

The model minimizes MSE during training to improve forecasts.

ML Use Case 4: Self-driving Cars

A steering model predicts steering angle.

• Actual angle = 15°
• Predicted = 10°

Error = 5°

MSE penalizes incorrect steering predictions, helping autonomous vehicles learn smoother driving behavior.

Python Example

from sklearn.metrics import mean_squared_error

actual = [80, 60, 90]
pred = [74, 63, 87]

mse = mean_squared_error(actual, pred)

print(mse)

Output:

18.0

MSE vs MAE

Metric	Formula	Penalizes Large Errors?
MAE	Average of |Error|	Less
MSE	Average of Error²	Yes

Example

Error	MAE	MSE
2	2	4
5	5	25
10	10	100

Notice how MSE grows much faster.

Why does Gradient Descent use MSE?

The MSE curve is smooth and differentiable.

Because it's smooth:

• We can compute derivatives.
• Gradient Descent knows which direction to move.
• The model learns efficiently.

Summary

Concept	Meaning
Error	Actual − Predicted
Squared Error	Error²
MSE	Average of squared errors
Goal	Minimize MSE

Memory Trick

M = Mean (average)

S = Squared (penalize big mistakes)

E = Error (prediction mistake)

Interview Question

Q: Why is MSE widely used in regression models?

Answer: MSE averages squared prediction errors, making it differentiable and heavily penalizing large mistakes. This produces a smooth loss surface that works well with Gradient Descent, making it ideal for training regression models such as house price prediction.

Cross Entropy Loss

Cross Entropy is the most important loss function for classification problems. It is used by:

• ChatGPT / LLMs
• Image classification
• Spam detection
• Face recognition
• Sentiment analysis

Cross Entropy measures how wrong a model's predicted probabilities are compared to the correct class.

Unlike MSE (which predicts numbers), Cross Entropy is used when the model predicts classes.

Regression vs Classification

	Regression	Classification
Predict	A number	A category
Example	House Price → ₹82L	Cat / Dog
Loss	MSE	Cross Entropy

Key idea: Cross Entropy compares probabilities, not raw numbers.

Intuition: Cat vs Dog

Suppose an image is actually a Cat.

The model predicts probabilities.

Good prediction

Class	Probability
Cat	0.95
Dog	0.05

The model is confident and correct.

Cross Entropy = Very Low

Bad prediction

Class	Probability
Cat	0.05
Dog	0.95

The model is confidently wrong.

Cross Entropy = Very High

Why probabilities matter

Imagine a spam detector.

Email:

"WIN FREE MONEY NOW!"

Actual class:

Spam

Model 1

Spam	Not Spam
0.98	0.02

Excellent prediction.

Model 2

Spam	Not Spam
0.51	0.49

Correct, but uncertain.

Model 3

Spam	Not Spam
0.02	0.98

Completely wrong.

Cross Entropy distinguishes these three situations.

Cross Entropy Formula

Binary classification:

Don't memorize it yet.

Understand the variables.

Symbol	Meaning
y	Actual label (0 or 1)
p	Predicted probability
L	Loss

For beginners:

Higher probability for the correct class → Smaller loss

Step-by-step Example

Actual:

Cat

Represent it as:

Prediction A

Model says:

Cross Entropy:

≈ 0.105

Very small.

Prediction B

Model says:

Loss:

≈ 1.61

Much larger.

Comparison

Prediction	Loss
0.99	0.01
0.90	0.10
0.70	0.36
0.50	0.69
0.20	1.61
0.01	4.60

Notice:

As the correct probability decreases, the loss rises rapidly.

Visual intuition

As confidence in the correct class increases:

Loss decreases.

ML Use Case 1: Image Classification

Input image:

Actual label:

Cat

Model output:

Class	Probability
Cat	0.92
Dog	0.08

Cross Entropy teaches the CNN to increase Cat probability and decrease Dog probability.

ML Use Case 2: Spam Detection

Input:

Congratulations! Claim your prize!

Actual:

Spam

Output:

Class	Probability
Spam	0.97
Normal	0.03

Loss is very small.

If the model predicted 0.10, the loss becomes large and weights are updated.

ML Use Case 3: ChatGPT

Prompt:

Machine learning is

Target next word:

powerful

The model predicts:

Word	Probability
powerful	0.72
useful	0.18
difficult	0.10

Cross Entropy compares the predicted distribution with the actual next word.

Then backpropagation updates billions of parameters.

This happens for every token during LLM training.

ML Use Case 4: Sentiment Analysis

Review:

"Amazing apartment with great amenities."

Actual sentiment:

Positive

Model prediction:

Sentiment	Probability
Positive	0.94
Negative	0.06

Cross Entropy rewards the model for assigning high probability to the correct class.

Cross Entropy vs MSE

Feature	MSE	Cross Entropy
Used for	Regression	Classification
Predicts	Numbers	Classes
Output	₹80L	Cat / Dog
Compares	Values	Probabilities
Used in LLMs	No	Yes

Training Curve

A good classifier shows decreasing cross entropy.

Epoch	Loss
1	2.4
2	1.7
3	1.1
4	0.6
5	0.2

As training improves:

• Correct probabilities increase.
• Cross Entropy decreases.

Python Example

from sklearn.metrics import log_loss

# Actual labels
y_true = [1, 0, 1]

# Predicted probabilities
y_pred = [0.9, 0.2, 0.8]

loss = log_loss(y_true, y_pred)

print(loss)

Output (approximately):

0.18

A lower value indicates better classification.

Summary

Concept	Meaning
Cross Entropy	Classification loss
Input	Predicted probabilities
Goal	Increase probability of the correct class
Used In	ChatGPT, CNN, Spam Detection, BERT

Memory Trick

MSE asks: "How far is my number?"

Cross Entropy asks: "How confident am I about the correct class?"

Interview Question

Q: Why do LLMs use Cross Entropy instead of MSE?

Answer: LLMs predict the probability distribution of the next token, not a continuous number. Cross Entropy compares the predicted token probabilities with the actual next token and provides gradients that efficiently train the model through backpropagation.

gradient descent

Gradient Descent is the algorithm that teaches an ML model how to learn. It uses the loss function and derivatives to improve predictions step by step.

Gradient Descent = An optimization algorithm that repeatedly adjusts model weights to minimize the loss function.

Think of it as walking downhill until you reach the lowest point.

Intuition: Climbing down a mountain

Imagine you're standing on a hill wearing a blindfold. Your goal is to reach the lowest valley.

You don't know where the valley is.

So you do one thing:

Check which direction slopes downward, then take one step.

That is exactly what Gradient Descent does.

Why do we need Gradient Descent?

Suppose your real estate model predicts:

Actual	Predicted
₹80L	₹74L

Loss = 6

The model asks:

How should I change my weights to reduce this loss?

The answer comes from the gradient.

The Gradient Descent Formula

This is one of the most important equations in ML.

Symbol	Meaning
w	Weight (parameter)
α	Learning Rate
L	Loss Function
∂L / ∂w	Gradient (slope)

Read it as:

New Weight = Old Weight − Learning Rate × Gradient

Understanding each part

Step 1: Weight

A model learns using weights.

Example:

Here:

• 10
• 0.05

are weights.

Gradient Descent changes these numbers.

Step 2: Loss

Suppose:

• Actual = 80
• Predicted = 74

MSE tells us the error.

Loss = 36

The objective is:

Minimize Loss

Step 3: Gradient

The derivative tells the slope.

If:

The slope is positive.

Move left.

If:

Move right.

The gradient tells direction.

Step 4: Learning Rate

Learning rate controls step size.

Learning Rate	Behavior
Very small	Slow learning
Medium	Good learning
Very large	Overshoots minimum

Step-by-step Example

Suppose:

Current weight:

Gradient:

Learning rate:

Apply formula:

The weight becomes 9.5.

The model has taken one learning step.

Visual learning process

Every iteration:

1. Predict
2. Compute loss
3. Compute gradient
4. Update weight

Repeat thousands of times.

ML Use Case 1: House Price Prediction

Features:

• Area
• BHK
• Parking

Initial model:

Loss is high.

Gradient Descent adjusts weights.

After training:

The prediction becomes much more accurate.

ML Use Case 2: Linear Regression

Training data:

Area	Price
1,000	50
1,200	60
1,400	70
1,600	80
1,800	90

The regression line starts randomly.

Gradient Descent moves the line until MSE becomes minimum.

ML Use Case 3: Image Classification

A CNN predicts:

Class	Probability
Cat	0.30
Dog	0.70

Actual = Cat

Cross Entropy computes loss.

Gradient Descent updates filters.

Next iteration:

Class	Probability
Cat	0.80
Dog	0.20

Prediction improves.

ML Use Case 4: ChatGPT

For every token:

Prompt:

Machine learning is

Target:

powerful

Training loop:

This happens billions of times.

Gradient Descent is one of the core learning mechanisms behind LLM training.

Batch, Stochastic & Mini-Batch Gradient Descent

Type	Uses	Speed
Batch	Entire dataset	Slow
Stochastic (SGD)	One sample	Fast but noisy
Mini-Batch	Small batches (32, 64, 128)	Most common

Mini-Batch intuition

Instead of learning from all 1 million properties:

• Batch = 1,000,000 homes
• SGD = 1 home
• Mini-batch = 64 homes

Modern deep learning usually uses mini-batch gradient descent.

Python Example

# One gradient descent update

weight = 10
gradient = 5
learning_rate = 0.1

new_weight = weight - learning_rate * gradient

print(new_weight)

Output:

9.5

The weight moved closer to the optimal value.

Gradient Descent vs Loss vs Derivative

Concept	Purpose
Loss Function	Measures how wrong the prediction is
Derivative (Gradient)	Tells which direction reduces loss
Gradient Descent	Updates weights using that direction

They work together:

1. Model predicts
2. Loss measures error
3. Derivative computes slope
4. Gradient Descent updates weights
5. Repeat until loss is minimized

Summary

Term	Meaning
Weight	Learnable parameter
Loss	Prediction error
Gradient	Slope of the loss function
Learning Rate	Step size
Gradient Descent	Algorithm that minimizes loss

Memory Trick

Loss tells the model how wrong it is.

Gradient tells it which way to move.

Gradient Descent takes the step.

Interview Question

Q: Why is Gradient Descent essential in Machine Learning?

Answer: Gradient Descent is the optimization algorithm that minimizes the loss function. It uses the gradient (derivative of the loss with respect to model parameters) to update weights iteratively, enabling the model to learn from data and improve its predictions over time.