Information Theory

entropy

Before learning LLMs like ChatGPT, you need one foundational idea:

Information Theory studies how information is measured, stored, compressed, and transmitted.

Entropy is the most important concept in Information Theory. It measures uncertainty.

What is Information Theory?

Information Theory was introduced by Claude Shannon in 1948.

Its goal is to answer questions like:

• How much information does a message contain?
• How uncertain is the next word?
• How can data be compressed?
• How do LLMs predict the next token?

Intuition: Guessing game

Imagine someone is thinking of a number.

Easy

Number is always 5.

You already know the answer.

Hard

Number could be 1–100.

Many possibilities.

The second situation contains more information because there is more uncertainty.

Information Theory = Measuring uncertainty.

What is Entropy?

Entropy measures how uncertain or unpredictable a system is.

The more uncertain something is, the higher its entropy.

The more predictable it is, the lower its entropy.

Coin example

A fair coin:

Possible outcomes:

• Heads = 50%
• Tails = 50%

This is highly uncertain.

High entropy.

Predictable coin

Suppose the coin always lands on Heads.

You already know the result.

Low entropy.

Shannon Entropy Formula

Don't memorize it immediately. Understand the parts.

Symbol	Meaning
H(X)	Entropy
p(x)	Probability of an outcome
log₂	Information measured in bits
Σ	Sum over all outcomes

Simple idea

• Equal probabilities → Higher entropy
• One dominant outcome → Lower entropy

Visualizing entropy

Low entropy

Very predictable.

High entropy

Maximum uncertainty.

ML Use Case: Decision Trees

A decision tree chooses the feature that reduces entropy the most.

Suppose you're predicting whether a customer will buy a property.

Before splitting:

Buy	Don't Buy
50	50

Entropy = High

After splitting by Budget:

Budget	Buy	Don't Buy
Under ₹80L	45	5
Above ₹80L	5	45

Now each group is much more predictable.

The tree chooses Budget because it reduces entropy.

Before vs after split

ML Use Case: ChatGPT (Most Important)

When you type:

Machine learning is

The model predicts probabilities for the next word.

Word	Probability
powerful	0.42
useful	0.30
important	0.18
difficult	0.10

These probabilities form a distribution.

Entropy measures how uncertain the model is.

Low entropy prediction

Word	Probability
powerful	0.98
others	0.02

The model is almost certain.

High entropy prediction

Word	Probability
powerful	0.26
useful	0.25
important	0.24
difficult	0.25

The model is uncertain.

LLMs internally work with these probability distributions during token prediction.

ML Use Case: Image Classification

Input image:

Prediction:

Class	Probability
Cat	0.94
Dog	0.06

Low entropy → High confidence

Another image:

Class	Probability
Cat	0.51
Dog	0.49

High entropy → Low confidence

This helps AI estimate uncertainty.

Entropy vs Cross Entropy

This is where many beginners get confused.

Entropy	Cross Entropy
Measures uncertainty of data	Measures prediction error
Information Theory	ML Loss Function
One probability distribution	Two distributions
Decision Trees	LLM Training

Think of it like this:

• Entropy asks: How uncertain is the data?
• Cross Entropy asks: How wrong is my predicted probability?

Python Example

from scipy.stats import entropy

# Fair coin
p = [0.5, 0.5]

print(entropy(p, base=2))

Output:

1.0

A fair coin has 1 bit of entropy.

Summary

Concept	Meaning
Information Theory	Science of measuring information
Entropy	Measures uncertainty
Low Entropy	Predictable outcomes
High Entropy	Uncertain outcomes
Cross Entropy	Classification loss using probabilities

Memory Trick

Entropy = Uncertainty

More uncertainty → Higher entropy

Less uncertainty → Lower entropy

Interview Question

Q: Why is entropy important in AI and ML?

Answer: Entropy quantifies uncertainty in probability distributions. It is used in decision trees to choose the best feature splits, in probabilistic models to measure information content, and as the foundation for cross-entropy loss, which trains classifiers and large language models by comparing predicted probabilities with the correct labels.

information gain

Information Gain is the concept that makes Decision Trees intelligent. It tells the algorithm which feature is the best one to split the data on.

Information Gain = How much uncertainty (entropy) decreases after splitting the data.

Think of it as:

Before Split → High Confusion

After Split → Less Confusion

The feature that reduces the most confusion is chosen.

Intuition: Property Buyers

Suppose you're building an AI model to predict whether a customer will buy a property.

Current dataset:

Customer	Buy?
A	Yes
B	No
C	Yes
D	No
E	Yes
F	No

We have:

• Yes = 3
• No = 3

Very uncertain.

The model doesn't know how to classify well.

Step 1: Calculate Entropy Before Split

We already know:

• Yes = 50%
• No = 50%

Entropy:

This is maximum uncertainty.

Step 2: Split by Budget

Now divide customers.

Under ₹80L

Buy	Count
Yes	3
No	0

Entropy = 0

Perfectly predictable.

Above ₹80L

Buy	Count
Yes	0
No	3

Entropy = 0

Also perfectly predictable.

The split completely removes uncertainty.

Information Gain Formula

Very simple:

Gain = Old uncertainty − New uncertainty

Step-by-step Calculation

Before split:

After split:

Information Gain:

IG = 1

This is an excellent split.

Bad Split Example

Suppose we split by Favorite Color.

Color	Buy	No
Blue	2	2
Red	1	1

Still mixed.

Entropy remains high.

Information Gain is very small.

The decision tree will reject this feature.

Decision Tree chooses the highest Information Gain

Suppose three possible features.

Feature	Information Gain
Budget	1.00
Location	0.42
Property Type	0.18

The tree chooses Budget because it reduces uncertainty the most.

This becomes the root node.

ML Use Case 1: Loan Approval

Dataset:

Income	Loan
High	Yes
High	Yes
Low	No
Low	No

Splitting by Income gives maximum Information Gain.

The tree instantly separates approvals.

ML Use Case 2: Medical Diagnosis

Features:

• Fever
• Blood Pressure
• Age
• Sugar

The algorithm evaluates Information Gain.

Suppose:

Feature	IG
Fever	0.81
Age	0.30
Sugar	0.12

The first question becomes:

Does the patient have fever?

Because it provides the most information.

ML Use Case 3: Spam Detection

Features:

Word	Information Gain
FREE	0.72
OFFER	0.44
HELLO	0.03

The word FREE is much better at separating spam from normal emails.

ML Use Case 4: Real Estate CRM

Predict whether a lead becomes a buyer.

Features:

Feature	IG
Budget	0.65
City	0.40
Occupation	0.21
Favorite Color	0.01

The CRM AI prioritizes Budget because it provides the highest Information Gain.

Python Example

from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(
    criterion="entropy"
)

model.fit(X_train, y_train)

Setting criterion="entropy" tells scikit-learn to use Information Gain for building the tree.

Entropy vs Information Gain

Entropy	Information Gain
Measures uncertainty	Measures reduction in uncertainty
Before split	Improvement after split
Higher = More confusion	Higher = Better feature
Used in decision trees	Chooses the best split

Think of it like cleaning a room

• Entropy = How messy the room is.
• Information Gain = How much cleaner the room becomes after organizing it.

The best action is the one that reduces the most mess.

Summary

Concept	Meaning
Entropy	Uncertainty in data
Split	Divide data using a feature
Information Gain	Reduction in entropy
Goal	Choose the feature with maximum IG

Memory Trick

Entropy = Confusion

Information Gain = Confusion Removed

Decision Tree always asks the question that removes the most confusion first.

Interview Question

Q: Why does a Decision Tree use Information Gain?

Answer: Information Gain measures how much a feature reduces entropy after splitting the data. The decision tree evaluates all candidate features and selects the one with the highest Information Gain, producing the most informative and least uncertain split.