Numerical Methods

floating point precision

This is one of the most important hidden concepts in AI and Deep Learning. Every neural network weight, embedding, gradient, and attention score is stored using floating-point numbers.

Floating Point Precision is the way computers store decimal (fractional) numbers with limited accuracy.

Unlike humans, computers cannot store every decimal exactly.

What is a floating-point number?

A floating-point number is a way to represent very large and very small decimal values.

For example:

Decimal Number	Floating Point
3.14	3.14
0.00052	5.2 × 10⁻⁴
5200000	5.2 × 10⁶

It's similar to scientific notation.

Scientific notation

A computer stores numbers using the same idea.

Why can't computers store decimals exactly?

Consider this simple number:

Humans write it exactly.

A computer stores it approximately in binary.

Human	Computer
0.1	0.10000000149...
0.2	0.20000000298...

The approximation is tiny, but it exists.

Example

0.1 + 0.2

Expected:

Computer often returns:

0.30000000000000004

This is floating-point precision error.

How computers store float values

A floating-point number has 3 parts.

Part	Purpose
Sign	Positive / Negative
Exponent	Size of number
Mantissa	Significant digits

Float32 vs Float64

In AI, these are the two most common types.

Type	Bits	Precision	ML Usage
float16	16	Low	Fast GPU training
float32	32	Medium	Default for deep learning
float64	64	High	Scientific computing

Visual comparison

More bits mean more accurate decimal representation.

Why does ML use float32?

Imagine ChatGPT has 7 billion weights.

If each weight used float64:

• More memory
• Slower GPU
• Higher cost

Using float32:

• Half the memory
• Faster computation
• Nearly identical accuracy

That's why TensorFlow and PyTorch default to float32.

AI Use Case 1: Neural Network Weights

Every connection has a weight.

Example weight:

This is stored as a float32 value.

Millions of these exist inside every model.

AI Use Case 2: Embeddings

Word embedding:

Each number is stored as float32.

A single embedding may contain 768 or 4096 floating-point values.

AI Use Case 3: Images

A pixel is often normalized.

Original RGB:

Normalized:

Another pixel:

Normalized:

These decimals require floating-point precision.

Precision problem example

Suppose we repeatedly add tiny numbers.

value = 1.0

for i in range(1000000):
    value += 0.000001

The expected answer isn't always exact because tiny rounding errors accumulate.

This is called numerical precision loss.

Why this matters for LLMs

Every forward pass computes billions of operations like:

Each multiplication introduces tiny floating-point rounding.

Modern AI libraries are carefully designed so these tiny errors do not grow uncontrollably.

This leads directly to your next lesson:

Overflow and Underflow.

Python Example

import numpy as np

a = np.float32(0.1)
b = np.float32(0.2)

print(a + b)

Output (approximately):

0.30000001

Notice it's not perfectly 0.3.

Summary

Concept	Meaning
Floating Point	Computer representation of decimals
Mantissa	Precision digits
Exponent	Scale of number
float32	Default ML data type
float64	Higher precision, more memory

Memory Trick

Integers are exact.

Decimals are approximate.

AI models learn using billions of approximate floating-point numbers.

Interview Question

Q: Why do deep learning frameworks use float32 instead of float64?

Answer: Float32 provides a strong balance between precision, memory usage, and computational speed. It uses half the memory of float64, runs significantly faster on GPUs, and is accurate enough for training most neural networks, making it the standard numeric type in modern deep learning.

overflow

After understanding Floating Point Precision, the next concept is Overflow.

Overflow happens when a number becomes larger than the maximum value a floating-point type can store.

In AI/ML, overflow is one of the most common causes of NaN, ∞ (Infinity), and failed model training.

What is Overflow?

Imagine a calculator that can display only 3 digits.

It can store:

• 999
• 845
• 1000 ❌

The number exceeds its storage limit.

That is overflow.

Visual intuition

The computer represents this as Infinity.

Maximum value of float32

A float32 has a maximum value of approximately:

Anything larger becomes:

Example:

Number	Result
100	Stored
1,000,000	Stored
10³⁰	Stored
10⁴⁰	Infinity

Why does overflow happen in ML?

The biggest reason is the exponential function.

As x grows, the value explodes.

Example

x	e^{x}
2	7.39
5	148
10	22,026
100	Huge
1000	Overflow

The exponential grows much faster than floating-point storage.

Visual growth of e^{x}

The curve rises so quickly that it exceeds memory limits.

AI Use Case 1: Softmax (Most Important)

Suppose a neural network outputs logits:

Softmax computes:

But:

Now Softmax becomes:

This produces:

NaN (Not a Number)

Your entire neural network breaks.

What happens?

This is exactly why Softmax Stabilization exists (your Lesson 5).

AI Use Case 2: Cross Entropy Loss

Suppose prediction probability becomes:

Cross entropy computes:

Result:

Infinite loss means training becomes unstable.

Modern frameworks prevent this with tiny epsilon values.

AI Use Case 3: Gradient Explosion

During backpropagation:

Gradient:

Next layer:

Now:

Weights become unusable.

This is called gradient explosion.

Python Example

import numpy as np

print(np.exp(1000))

Output:

inf

The exponential overflows and returns Infinity.

How ML prevents overflow

Technique	Purpose
Softmax Stabilization	Prevent exp(1000)
Gradient Clipping	Stop exploding gradients
Float64	Larger numeric range
Log-space Computation	Avoid huge exponentials

You'll learn the first technique in Lesson 5: Softmax Stabilization.

Overflow vs Underflow

Overflow	Underflow
Number becomes too large	Number becomes too small
Result → ∞	Result → 0
exp(1000)	exp(-1000)
Exploding gradients	Vanishing gradients

Summary

Concept	Meaning
Overflow	Number exceeds maximum float value
Symbol	∞ (Infinity)
Main Cause	Exponential growth
ML Impact	NaN, infinite loss, failed training

Memory Trick

Overflow = Too BIG → ∞

Underflow = Too SMALL → 0

Interview Question

Q: Why is overflow dangerous in deep learning?

Answer: Overflow occurs when calculations exceed the representable range of floating-point numbers, producing Infinity or NaN. This commonly happens in exponential operations such as Softmax and can cause unstable loss values, exploding gradients, and failed model training.

underflow

After Overflow (too large) comes the opposite problem: Underflow (too small).

Underflow happens when a number becomes so close to zero that the computer stores it as 0.

This is extremely important in AI because probabilities, gradients, and exponential functions often produce tiny numbers.

What is Underflow?

Imagine your calculator can only display numbers down to 0.001.

Real Value	Calculator Stores
0.1	0.1
0.01	0.01
0.001	0.001
0.0000001	0

The value is not actually zero—it is too small to represent.

Visual intuition

Too small → Stored as zero

Why does Underflow happen?

The main cause is negative exponentials.

As x increases, the value becomes extremely tiny.

x	e^{-x}
2	0.135
5	0.0067
10	0.000045
100	Almost 0
1000	0

The computer cannot represent such tiny values accurately.

Visual growth of e^{-x}

Eventually the value becomes numerically zero.

AI Use Case 1: Softmax

Suppose logits are:

Softmax computes:

The result underflows to:

Probability becomes exactly 0.

This can create problems when computing loss.

AI Use Case 2: Cross Entropy Loss

Cross entropy uses:

Suppose the probability underflows.

Now:

Result:

Even though the original value was merely very small, underflow turns it into zero, producing infinite loss.

AI Use Case 3: Vanishing Gradients

During backpropagation:

Result:

After many layers:

Eventually:

0

The gradient disappears.

This is called Vanishing Gradient.

Older deep neural networks struggled with this problem.

Modern architectures use techniques like:

• ReLU activation
• Batch Normalization
• Residual Connections

to reduce vanishing gradients.

AI Use Case 4: Probability Models

Suppose your language model predicts:

Word	Probability
the	0.80
cat	0.19
zebra	1 × 10⁻⁵⁰

The last probability may underflow to 0.

Instead of storing probabilities directly, many ML libraries store log probabilities, which are much more stable.

Python Example

import numpy as np

print(np.exp(-1000))

Output:

0.0

The value isn't mathematically zero—it underflowed.

Overflow vs Underflow

Overflow	Underflow
Number too large	Number too small
Result = ∞	Result = 0
exp(1000)	exp(-1000)
Exploding gradients	Vanishing gradients

Think of them as opposites:

Why this matters for LLMs

During every forward pass:

• Softmax computes exponentials.
• Cross Entropy computes logarithms.
• Backpropagation computes tiny gradients.

Without handling underflow correctly, training would produce:

• Zero gradients
• Infinite loss
• NaN values

This leads directly to the next lesson:

Numerical Stability — techniques that prevent both overflow and underflow.

Summary

Concept	Meaning
Underflow	Number becomes too small to represent
Result	0
Main Cause	Negative exponentials
ML Impact	Vanishing gradients, zero probabilities

Memory Trick

Overflow = Too BIG → ∞

Underflow = Too SMALL → 0

Numerical Stability prevents both.

Interview Question

Q: Why is underflow dangerous in deep learning?

Answer: Underflow causes extremely small floating-point values to become zero. This can produce zero probabilities, vanishing gradients, and unstable loss calculations, especially in Softmax, Cross Entropy, and deep neural network backpropagation.

numerical stability

You've learned:

• Floating Point Precision → Approximate decimals
• Overflow → Numbers become ∞
• Underflow → Numbers become 0

Now the solution is Numerical Stability.

Numerical Stability is the practice of designing algorithms so they produce accurate results despite floating-point limitations.

In AI, it's not enough for an equation to be mathematically correct—it must also be computationally stable.

Why do we need numerical stability?

Suppose two numbers are almost identical.

Mathematically:

But floating-point precision can lose significant digits.

This is called loss of precision.

Visual intuition

A tiny numerical error appears even though the math is simple.

Stable vs Unstable computation

Unstable method

Problem:

• e¹⁰⁰⁰ → Overflow
• Result → NaN

Stable method

Subtract the maximum first.

Now:

• No overflow
• Same mathematical answer
• Stable computation

This trick is used inside every modern deep learning framework.

Example: Cat vs Dog classifier

Neural network outputs logits:

Class	Logit
Cat	1000
Dog	999

If we directly compute Softmax:

Overflow occurs.

Instead, subtract the largest value.

New logits:

Class	New Logit
Cat	0
Dog	-1

Now exponentials are safe.

Notice:

We changed the numbers but not the probabilities.

AI Use Case 1: Softmax

Softmax is everywhere:

• ChatGPT
• BERT
• Llama
• Image classifiers

Without numerical stability:

With stability:

This prevents NaN during training.

AI Use Case 2: Cross Entropy

Cross entropy contains:

If:

Then:

Stable implementations never allow probability to become exactly zero.

Instead:

The loss remains finite.

AI Use Case 3: Gradient Descent

Suppose gradients become huge.

Iteration	Gradient
1	12
2	45
3	8000
4	∞

Training crashes.

A stable algorithm clips gradients.

Example:

This is called Gradient Clipping.

It prevents exploding gradients.

AI Use Case 4: Attention in Transformers

Attention computes many exponentials.

Formula (simplified):

If attention scores are large:

Token	Score
A	250
B	248
C	245

Direct exponentials overflow.

Transformers stabilize these scores before Softmax.

Without this trick, ChatGPT would fail to train.

Common numerical stability techniques

Technique	Prevents
Subtract Max	Overflow in Softmax
Log-Sum-Exp	Stable logarithms
Gradient Clipping	Exploding gradients
Epsilon (ε)	log(0) errors
Batch Normalization	Unstable activations

These are practical engineering techniques used in every major ML library.

Python Example

Unstable

import numpy as np

x = np.array([1000, 999])

print(np.exp(x))

Output:

[inf inf]

Stable

x = np.array([1000, 999])

x = x - np.max(x)

print(np.exp(x))

Output:

[1.0, 0.367879]

Same mathematical result, but no overflow.

Numerical Stability vs Precision

Floating Point Precision	Numerical Stability
How numbers are stored	How algorithms avoid errors
Hardware limitation	Software solution
Rounding errors	Stable computation tricks
float32 / float64	Softmax, clipping, epsilon

Think of it like this:

• Precision = Quality of the measuring ruler.
• Numerical Stability = Using the ruler correctly.

Summary

Concept	Meaning
Numerical Stability	Reliable computation despite floating-point limits
Overflow	Too large → ∞
Underflow	Too small → 0
Stable Softmax	Subtract maximum value
Gradient Clipping	Limits exploding gradients

Memory Trick

Math says the answer should be correct.

Numerical Stability makes sure the computer actually gets that correct answer.

Interview Question

Q: Why is numerical stability important in deep learning?

Answer: Deep learning performs billions of floating-point operations. Numerical stability techniques prevent overflow, underflow, NaN values, exploding gradients, and unstable loss calculations, ensuring models train reliably and produce accurate results.

softmax stabilization

This is one of the most important numerical tricks in Deep Learning. Every Transformer model (ChatGPT, GPT-5, BERT, Llama, Gemini) uses a stable version of Softmax.

Softmax Stabilization prevents overflow by subtracting the largest logit before computing exponentials.

The key insight is:

Subtracting the same number from every logit does not change the final probabilities.

What is Softmax?

A neural network doesn't directly output probabilities.

It outputs logits (raw scores).

Example:

Animal	Logit
Cat	8
Dog	5
Bird	2

Softmax converts these scores into probabilities.

Output:

Animal	Probability
Cat	95%
Dog	4.7%
Bird	0.3%

The probabilities always sum to 1.

Softmax Formula

Where:

Symbol	Meaning
x_i	Logit
e^{x}	Exponential
P_i	Probability

For beginners:

1. Take exponentials
2. Add them
3. Divide each by the total

Normal Softmax Example

Logits:

Step 1: Exponentials

Logit	e^{x}
2	7.39
1	2.72
0	1.00

Step 2: Sum

Step 3: Probability

Cat:

Final probabilities:

Class	Probability
Cat	66.5%
Dog	24.5%
Bird	9.0%

Everything works perfectly.

The Problem: Overflow

Suppose logits become huge.

Now compute exponentials.

Logit	e^{x}
1000	∞
999	∞
998	∞

Softmax becomes:

Result:

NaN

The model crashes.

The Solution: Subtract the Maximum

Take the largest logit.

Subtract it from every value.

New logits:

Notice the ranking is unchanged.

Now exponentials are safe.

Stable Softmax Calculation

New logits:

Exponentials:

Logit	e^{x}
0	1.000
-1	0.368
-2	0.135

Sum:

Probabilities:

Class	Probability
Cat	66.5%
Dog	24.5%
Bird	9.0%

Amazing observation

The probabilities are exactly the same as before.

We prevented overflow without changing the answer.

Why subtracting max works

Suppose logits:

Subtract 8:

The relative differences remain identical.

Softmax only cares about relative differences, not absolute values.

AI Use Case 1: ChatGPT

When ChatGPT predicts the next token:

Prompt:

Machine learning is

The transformer produces logits.

Token	Logit
powerful	23.8
useful	21.3
difficult	19.7

Before Softmax:

• Subtract maximum (23.8)
• Apply exponentials
• Generate probabilities

This happens for every token.

AI Use Case 2: Image Classification

CNN output:

Class	Logit
Cat	250
Dog	245
Bird	241

Direct exponentials overflow.

Stable Softmax converts them safely into probabilities.

AI Use Case 3: Attention Mechanism

Transformers compute attention scores:

Scores may be:

Token	Score
A	140
B	138
C	137

Before applying Softmax, the implementation stabilizes these scores.

Without it, attention would overflow.

AI Use Case 4: Recommendation System

A recommendation model scores products.

Property	Score
Villa	120
Apartment	118
Plot	110

Stable Softmax converts scores into recommendation probabilities without numerical issues.

Python Example

Unstable

import numpy as np

x = np.array([1000, 999, 998])

np.exp(x)

Output:

[inf inf inf]

Stable

x = x - np.max(x)

exp_x = np.exp(x)

softmax = exp_x / np.sum(exp_x)

print(softmax)

Output:

[0.665, 0.245, 0.090]

Safe and correct.

Softmax vs Stable Softmax

Normal Softmax	Stable Softmax
Uses raw logits	Subtracts maximum logit
Can overflow	Prevents overflow
May produce NaN	Numerically stable
Theoretical formula	Practical implementation

Summary

Concept	Meaning
Softmax	Converts logits into probabilities
Logits	Raw neural network scores
Overflow	exp(1000) = ∞
Stable Softmax	Subtract maximum before exponentials
Used In	ChatGPT, BERT, CNNs, Transformers

Memory Trick

Softmax = Scores → Probabilities

Stable Softmax = Scores − Max → Probabilities

The subtraction changes the numbers, not the final probabilities.

Interview Question

Q: Why do Transformers subtract the maximum logit before Softmax?

Answer: Subtracting the maximum logit prevents exponential overflow while preserving the relative differences between logits. Since Softmax is invariant to adding or subtracting the same constant from all logits, this produces identical probabilities with numerically stable computation.