GenAI Course · Module 3 · Session 10

Loss Functions & Gradient Descent

⏱ 14 min read🎯 Intermediate🖼 1 diagram🧪 1 live demo🐍 Python code

A network with random weights makes terrible predictions. To improve, it needs two things: a number that says how wrong it is (the loss), and a method to change the weights so that number goes down (gradient descent). This is the engine of all deep learning, including every LLM.

In this session

Loss functions: MSE, MAE, binary and categorical cross-entropy
Gradient descent: walking downhill on the loss
Learning rate: too small, just right, too big (live demo)
Batch vs stochastic vs mini-batch
Epochs, batch size and how they relate

01Loss functions

A loss function turns “how wrong was the model?” into one number. Lower is better. Which one you use depends on the task:

losses.py

import numpy as np

# Regression: predicted vs actual house price (₹ lakh)
actual = np.array([50, 80, 120]); pred = np.array([55, 70, 150])
print("MSE:", round(np.mean((pred - actual) ** 2), 1))   # squares: big errors hurt a lot
print("MAE:", np.mean(np.abs(pred - actual)))         # plain average error

# Binary classification: probability that a transaction is fraud
def bce(y, p): return -(y * np.log(p) + (1 - y) * np.log(1 - p))
for p in [0.9, 0.6, 0.1]:
    print(f"fraud (y=1), model says p={p}: BCE = {bce(1, p):.2f}")

# Multi-class: 3 complaint categories, the true class is index 2
probs = np.array([0.2, 0.1, 0.7])
print("categorical cross-entropy:", round(-np.log(probs[2]), 3))

MSE: 341.7 MAE: 15.0 fraud (y=1), model says p=0.9: BCE = 0.11 fraud (y=1), model says p=0.6: BCE = 0.51 fraud (y=1), model says p=0.1: BCE = 2.30 categorical cross-entropy: 0.357

Loss	Task	Formula idea	Note
MSE (mean squared error)	regression	average of (prediction − actual)²	big errors are punished a lot (30² = 900)
MAE (mean absolute error)	regression	average of \|prediction − actual\|	more forgiving of outliers
Binary cross-entropy	yes/no classification	−[y log p + (1−y) log(1−p)]	confident and wrong = very high loss
Categorical cross-entropy	pick one of many classes	−log(probability of the true class)	what LLMs minimise for the next token

Look at the cross-entropy numbers: saying 0.9 for a real fraud costs only 0.11, but saying 0.1 costs 2.30. Cross-entropy punishes confident mistakes hardest.

02Gradient descent

Picture the loss as a valley. The weights are your position; the height is the loss. The gradient tells you which way is uphill, so you step the opposite way:

new weight = old weight − learning_rate × gradient

Gradient descent: from a random start, take steps downhill (against the gradient) until the loss stops falling. Steps shrink as the slope flattens.

⛰️

Like coming down a hill in fog. You can’t see the bottom, but you can feel the slope under your feet. Take a step downhill, feel again, repeat. The learning rate is your step size.

We fit a straight line to 100 points (the true line is y = 3x + 2) with four learning rates, 50 steps each:

gd_lr.py

import numpy as np
np.random.seed(0)
x = np.random.rand(100) * 4
y = 3 * x + 2 + np.random.randn(100) * 0.5          # true line: y = 3x + 2

def train(lr, steps=50):
    w, b = 0.0, 0.0
    for _ in range(steps):
        err = (w * x + b) - y
        w -= lr * 2 * np.mean(err * x)               # gradient of MSE w.r.t. w
        b -= lr * 2 * np.mean(err)                   # gradient w.r.t. b
    return w, b, np.mean(((w * x + b) - y) ** 2)

for lr in [0.001, 0.01, 0.05, 0.2]:
    w, b, loss = train(lr)
    print(f"lr={lr:<6} after 50 steps: w={w:8.2f}  b={b:8.2f}  loss={loss:.3g}")

lr=0.001 after 50 steps: w= 1.43 b= 0.60 loss=23.4 lr=0.01 after 50 steps: w= 3.23 b= 1.50 loss=0.35 lr=0.05 after 50 steps: w= 3.09 b= 1.87 loss=0.263 lr=0.2 after 50 steps: w=-488186.78 b=-197613.44 loss=1.57e+12

Demo · Pick a learning rate

03Batch, stochastic and mini-batch

How many examples should we look at before each step?

Variant	Examples per update	Pros	Cons
Batch GD	all of them	smooth, exact gradient	slow, and huge datasets don’t fit in memory
Stochastic GD (SGD)	one	many quick updates; noise can escape bad spots	very noisy; needs a smaller learning rate
Mini-batch GD	a small group (32–512)	best of both; fast on GPUs	one more setting to choose

gd_variants.py

import numpy as np
np.random.seed(1)
N = 10_000
x = np.random.rand(N) * 4; y = 3 * x + 2 + np.random.randn(N) * 0.5

def run(batch, lr=0.05, epochs=3):
    w = b = 0.0; updates = 0
    for _ in range(epochs):
        idx = np.random.permutation(N)
        for s in range(0, N, batch):
            j = idx[s:s + batch]; err = (w * x[j] + b) - y[j]
            w -= lr * 2 * np.mean(err * x[j]); b -= lr * 2 * np.mean(err); updates += 1
    return w, b, updates

for name, bs in [("Batch GD", N), ("Mini-batch (64)", 64), ("Stochastic (1)", 1)]:
    w, b, u = run(bs, lr=0.05 if bs > 1 else 0.005)
    print(f"{name:16s} updates={u:6d}  w={w:.2f}  b={b:.2f}")

Batch GD updates= 3 w=3.07 b=1.26 Mini-batch (64) updates= 471 w=2.94 b=1.97 Stochastic (1) updates= 30000 w=3.04 b=2.04

Same 10,000 points, same 3 epochs. Full-batch made only 3 updates and is still far from b = 2. Mini-batch made 471 updates and is close. Pure SGD made 30,000 tiny noisy updates. In practice almost everyone uses mini-batches; when people say “SGD” today they usually mean mini-batch SGD.

04Learning rate, epochs and batch size

Term	Meaning	Typical value
Learning rate	step size of each update	0.1–0.0001; LLM fine-tuning ~1e-4 to 2e-5
Batch size	examples per update	32–512 (limited by GPU memory)
Epoch	one full pass over the training data	3–100s, depending on data size
Steps (iterations)	number of updates	epochs × (dataset size ÷ batch size)

Quick maths: 10,000 examples, batch size 64 → 157 steps per epoch (the last batch is smaller). 3 epochs → 471 steps, exactly what the code printed.

05Summary

The loss measures how wrong the model is: MSE/MAE for numbers, cross-entropy for classes.
Cross-entropy punishes confident mistakes the most; LLMs are trained with it.
Gradient descent: weight −= learning rate × gradient, repeated.
Learning rate too small = slow; too big = explodes.
Mini-batch gradient descent is the standard; steps = epochs × (data ÷ batch size).

06Quick quiz

1. Which loss is used for yes/no classification?

2. In gradient descent we move the weights…

3. Loss explodes to 10¹². Most likely cause?

4. One epoch is…

5. Why is mini-batch GD the default?

07References

Created by Ajeetkumar