AI Math Capstone

vector calculator

This is your first AI Math Capstone. Instead of learning a new formula, you'll build the same vector operations that every ML library (NumPy, PyTorch, TensorFlow) performs internally.

Goal: Create a calculator that can perform:

• Vector Addition
• Vector Subtraction
• Dot Product
• Magnitude
• Unit Vector

Why this matters: Every embedding, gradient, and feature vector in AI is represented as a vector. This capstone combines everything you learned in Linear Algebra.

Problem statement

Imagine a house is represented by three features:

Feature	Meaning
x	Area
y	Bedrooms
z	Parking

Two properties become two vectors:

Your calculator will compare these vectors mathematically.

Example: Vector Calculator

Example vectors in 2D:

Live graph:

Blue = Vector A • Red = Vector B

Blue = Vector A • Red = Vector B

Calculated results

Operation	Result	Notes
Addition A + B	(−5, 2)
Subtraction A − B	(−1, 2)
Dot Product A · B	6	(−3)(−2) + (2)(0)
Magnitude of A	3.61	√((−3)² + 2²)
Unit Vector of A	(−0.83, 0.55)	Direction only (length = 1)

How each operation is used in AI

A. Vector Addition

Example:

AI use case: Combining multiple embeddings.

Example in ChatGPT:

• Word embedding
• Position embedding

They are added together before entering the transformer.

B. Vector Subtraction

Example:

AI use case: Similarity between two users or two properties.

If two property vectors are very close after subtraction, they are likely similar.

C. Dot Product

Example:

AI use case: Attention Mechanism in Transformers.

The famous equation:

is simply thousands of dot products.

Dot product measures similarity.

D. Magnitude

Example:

AI use case: Measuring the strength of embeddings or gradients.

E. Unit Vector

Example:

AI use case: Cosine Similarity uses normalized (unit) vectors.

Real-world AI scenario

Property Recommendation Engine

Suppose every property is converted into a vector.

Property	Vector
Apartment A	(1200, 3, 2)
Apartment B	(1250, 3, 2)
Villa	(3000, 5, 4)

When a user searches, the AI computes:

• Subtraction → Difference
• Magnitude → Distance
• Dot Product → Similarity

The closest vectors become recommendations.

This is the same idea used in:

• Netflix recommendations
• Spotify songs
• Amazon products
• Property search

Python implementation

import numpy as np

A = np.array([2, 3])
B = np.array([4, 1])

print("Addition:", A + B)
print("Subtraction:", A - B)
print("Dot:", np.dot(A, B))
print("Magnitude:", np.linalg.norm(A))
print("Unit:", A / np.linalg.norm(A))

Expected output:

Addition: [6 4]
Subtraction: [-2 2]
Dot: 11
Magnitude: 3.6055
Unit: [0.55 0.83]

Capstone Checklist

• Create two vectors
• Perform addition
• Perform subtraction
• Calculate dot product
• Calculate magnitude
• Calculate unit vector
• Visualize vectors on a 2D graph

This is the mathematical foundation behind embeddings and attention in modern AI.

Next Capstone

Project 2: Gradient Descent Visualizer — you'll build an interactive optimizer that visually learns the minimum of a loss function step by step.

gradient descent visualizer

This capstone lets you be the optimizer. Instead of memorizing the formula, you'll watch Gradient Descent move toward the minimum loss step by step.

Objective: Minimize the loss function by updating the weight using Gradient Descent.

Loss function used in this project:

The minimum occurs at w = 3, where the loss is 2.

The algorithm you're implementing

Every training iteration follows the same 4 steps:

1. Make a prediction (represented by the current weight w)
2. Compute the loss
3. Compute the gradient
4. Update the weight

The update rule is:

Where:

• w = model weight
• α = learning rate
• gradient = slope of the loss curve

Example: Gradient Descent step

Start from w = -5 with learning rate α = 0.20.

Live loss landscape:

Blue curve = loss function • Red point = current weight • Orange path = optimization history

Blue curve = loss function • Red point = current weight • Orange path = optimization history

Quantity	Value	Notes
Current weight w	-5.000	Starting point
Current loss L(−5)	66.000
Gradient	−16.000	2 × (−5 − 3)
Learning rate α	0.20

Live calculation

This is exactly the computation PyTorch performs during optimization.

How the graph works

The loss surface is a parabola.

The optimizer always tries to move toward the lowest point.

• Left side: Gradient is negative → move right
• Right side: Gradient is positive → move left
• Center: Gradient becomes zero → stop

Learning Rate Experiment

Try these values in the visualizer:

Learning Rate	What you'll observe
0.05	Very slow movement
0.20	Smooth convergence
0.50	Fast convergence
1.00	Oscillates and may never settle

This is why choosing the learning rate is critical in deep learning.

Real AI Example: House Price Model

Suppose the model is:

Initial weight:

After each gradient descent step:

Iteration	Weight	Loss
0	-5.0	66
1	-1.8	25
2	0.76	7.0
3	2.41	2.3
4	2.88	2.0

The weight gradually approaches the optimal value.

How PyTorch does this

import torch

# Weight
w = torch.tensor(-5.0, requires_grad=True)

# Learning rate
lr = 0.2

for i in range(5):
    loss = (w - 3) ** 2 + 2

    loss.backward()

    with torch.no_grad():
        w -= lr * w.grad

    w.grad.zero_()

    print(w, loss)

This loop is exactly what your visualizer simulates.

Connection to ChatGPT

When training an LLM:

1. Predict next token
2. Compute Cross Entropy Loss
3. Calculate gradients
4. Update billions of weights
5. Repeat for trillions of tokens

Your visualizer uses one weight. ChatGPT uses the same mathematics, just at a vastly larger scale.

Capstone Checklist

• Visualize a loss function
• Calculate the gradient
• Adjust the learning rate
• Update weights using Gradient Descent
• Watch convergence toward the minimum

Next Capstone

Project 3: PCA from Scratch — you'll reduce a 2D/3D dataset into principal components using covariance, eigenvalues, and eigenvectors.

pca from scratch

This is one of the most important mathematical projects in Machine Learning because PCA (Principal Component Analysis) is how AI compresses data while keeping the most useful information.

Goal: Reduce high-dimensional data into fewer dimensions using covariance, eigenvalues, and eigenvectors.

Real AI examples:

• Face recognition → Compress 10,000 pixel features into ~100 features
• Recommendation systems → Reduce thousands of user behaviors into latent features
• Customer analytics → Compress dozens of variables into 2–3 meaningful components
• Computer vision → Remove redundant image information

The problem PCA solves

Imagine each house has many features:

Area	Bedrooms	Parking	Floor	Age
1200	3	2	4	8
1250	3	2	5	6
1300	4	2	5	5

Instead of keeping 5 features, PCA may reduce them to 2 features while preserving most of the information.

The reduced features are called Principal Components.

The 4 mathematical steps

PCA always follows the same pipeline.

We'll build each step from scratch.

Example: PCA visualization

Higher variance makes the principal component more dominant.

Live PCA visualization:

Left: PCA finds the red direction of maximum variance. Right: the same points projected onto one dimension.

Left: PCA finds the red direction of maximum variance. Right: the same points projected onto one dimension.

Quantity	Value
Covariance Matrix	[[4.02, 4.00], [4.00, 3.99]]
Principal Component	PC1 = (0.71, 0.71)
Eigenvalue	λ₁ = 8.00

Step 1 — Center the data

Before PCA, subtract the mean.

Original points:

X	Y
2	3
4	5
6	7

Mean:

Centered data:

X	Y
-2	-2
0	0
2	2

Why? PCA measures variation around the center, not around zero.

Step 2 — Build the Covariance Matrix

Covariance measures how two variables change together.

Example:

Interpretation:

• Large diagonal values → High variance
• Large off-diagonal values → Strong correlation

The covariance matrix is the input to PCA.

Step 3 — Find Eigenvalues & Eigenvectors

From the covariance matrix, compute:

Output	Meaning
Eigenvalue	Amount of variance
Eigenvector	Direction of variance

Visual intuition:

The red line is the first principal component.

It captures the most information.

Step 4 — Project the data

Instead of storing 2 coordinates:

Project onto the principal component.

The 2D points become 1D values.

This is dimensionality reduction.

AI Use Case — Face Recognition

A grayscale face image:

A 100×100 image contains:

PCA reduces those 10,000 features into perhaps 100 principal components.

The compressed representation is much smaller while preserving facial structure.

This was the foundation of the classic Eigenfaces algorithm.

AI Use Case — Customer Segmentation

Imagine customers described by:

• Income
• Spending
• Age
• Visits
• Purchases

PCA finds hidden directions like:

• Luxury buyers
• Budget buyers
• Frequent visitors

Instead of analyzing 20 variables, analysts visualize just 2 principal components.

Python from Scratch

import numpy as np

# Dataset
X = np.array([
    [2, 3],
    [4, 5],
    [6, 7]
])

# Step 1: Center
X_centered = X - X.mean(axis=0)

# Step 2: Covariance
C = np.cov(X_centered.T)

# Step 3: Eigen decomposition
eigenvalues, eigenvectors = np.linalg.eig(C)

# Step 4: Projection
PC1 = eigenvectors[:, 0]
projection = X_centered @ PC1

print(projection)

This is the complete PCA pipeline.

PCA Summary

Step	Purpose
1. Center Data	Remove the mean
2. Covariance	Measure relationships
3. Eigenvalues	Measure variance
4. Eigenvectors	Find best direction
5. Projection	Reduce dimensions

Memory Trick

Covariance asks: How do features move together?

Eigenvector answers: Which direction contains the most information?

PCA keeps that direction and removes the rest.

Interview Question

Q: Why do we use PCA before training some ML models?

Answer: PCA reduces the number of features by projecting data onto directions with the highest variance. This decreases dimensionality, removes redundant information, speeds up training, reduces memory usage, and often improves visualization and noise reduction while preserving most of the important information.

bayes spam detector

This capstone combines everything you've learned about Probability + Conditional Probability + Bayes Theorem to build a simple spam detector.

Objective: Given an email's words, calculate the probability that it is Spam.

Real AI examples:

• Gmail spam filtering
• Email phishing detection
• SMS spam detection
• Document classification

The problem

Suppose you receive this email:

Subject: WIN a FREE iPhone Today!

Claim your reward now. Limited offer.

Question:

Is this Spam or Not Spam?

A machine doesn't understand language first—it starts by calculating probabilities.

Bayes Theorem refresher

Bayes updates our belief after seeing evidence.

• Prior = Probability an email is spam before reading it
• Evidence = Words like FREE or WIN
• Posterior = Updated spam probability

How Naive Bayes works

The model looks for important words.

Word	Spam Likelihood	Ham Likelihood
FREE	80%	10%
WIN	70%	5%
Meeting	10%	80%

Notice:

• FREE strongly suggests spam.
• Meeting strongly suggests a normal email.

The model multiplies these probabilities together.

Example: Spam Detector

Suppose the email contains FREE and WIN.

Class	Probability
Spam	98.7%
Not Spam	1.3%

Classification: Spam

This is a simplified Naive Bayes model using three features.

Try these combinations

Selected words	Expected result
FREE + WIN	Very high spam probability
Meeting only	Very low spam probability
FREE + Meeting	Mixed evidence
None	Falls back to the prior probability

Step-by-step calculation

Suppose the email contains FREE and WIN.

Step 1 — Prior

Step 2 — Likelihood

Step 3 — Ham likelihood

Step 4 — Normalize

Spam score:

Ham score:

Posterior:

The email is almost certainly Spam.

Why is it called Naive Bayes?

It assumes every feature is independent.

Example:

Word	Assumption
FREE	Independent
WIN	Independent
OFFER	Independent

In reality, these words are often related.

The independence assumption is "naive," but it works surprisingly well for text classification.

AI pipeline

Every spam filter follows this basic workflow.

Python from scratch

# Prior probabilities
spam = 0.4
ham = 0.6

# Likelihoods
p_free_spam = 0.8
p_win_spam = 0.7

p_free_ham = 0.1
p_win_ham = 0.05

# Scores
spam_score = spam * p_free_spam * p_win_spam
ham_score = ham * p_free_ham * p_win_ham

# Normalize
posterior = spam_score / (spam_score + ham_score)

print(posterior)

Output:

0.987

Meaning: 98.7% Spam

Where Naive Bayes is used in ML

Application	Example
Email Filtering	Gmail Spam
SMS Detection	Fraud messages
Sentiment Analysis	Positive vs Negative reviews
News Classification	Sports, Politics, Tech
Document Categorization	Legal, Medical, Finance

Capstone Summary

Step	Purpose
Prior	Initial spam probability
Evidence	Words in the email
Likelihood	Probability of seeing each word
Bayes	Update the belief
Posterior	Final spam probability

Memory Trick

Prior = What I believed before reading the email.

Evidence = The words I found.

Posterior = What I believe after seeing the evidence.

This is the exact probabilistic reasoning behind classic spam filters and a foundational example of Bayesian machine learning.