Backpropagation

Gradient descent needs the gradient of the loss with respect to every weight. A modern LLM has billions of weights. Backpropagation computes all of those gradients in one backward sweep using a single idea from school calculus: the chain rule. Let us do it by hand once, so it never feels like magic again.

01The chain rule

If A affects B and B affects C, then how much A affects C = (how much A affects B) × (how much B affects C). In symbols:

A neural network is a long chain of simple steps (multiply, add, activate). So the effect of any weight on the loss is the product of the local effects along the path from that weight to the loss.

02Computational graphs

We write the network as a graph of tiny operations. Our example: one input, one hidden neuron with sigmoid, one output, squared-error loss. With x = 2, target y = 1 and starting weights w₁ = 0.5, b₁ = 0, w₂ = 1.5, b₂ = −0.5:

• Forward pass: compute and save every intermediate value (z, h, ŷ, L).
• Backward pass: start from the loss and multiply local derivatives backwards.
• The saved values are why training needs more memory than inference.

03Backpropagation step by step

Now the full code: forward, backward by hand, a check with PyTorch, and one update step:

import math

# tiny network: x -> z = w1*x + b1 -> h = sigmoid(z) -> y_hat = w2*h + b2 -> loss = (y_hat - y)^2
x, y = 2.0, 1.0
w1, b1, w2, b2 = 0.5, 0.0, 1.5, -0.5

# forward pass (save every intermediate value)
z = w1 * x + b1
h = 1 / (1 + math.exp(-z))
y_hat = w2 * h + b2
loss = (y_hat - y) ** 2
print(f"forward:  z={z:.3f}  h={h:.3f}  y_hat={y_hat:.3f}  loss={loss:.4f}")

# backward pass: chain rule, from the loss back to each weight
dL_dyhat = 2 * (y_hat - y)
dL_dw2 = dL_dyhat * h
dL_db2 = dL_dyhat * 1
dL_dh = dL_dyhat * w2
dL_dz = dL_dh * h * (1 - h)              # sigmoid'(z) = h(1-h)
dL_dw1 = dL_dz * x
dL_db1 = dL_dz * 1
print(f"manual:   dw1={dL_dw1:.4f}  db1={dL_db1:.4f}  dw2={dL_dw2:.4f}  db2={dL_db2:.4f}")

# check with PyTorch autograd
import torch
t = {k: torch.tensor(v, requires_grad=True) for k, v in dict(w1=w1, b1=b1, w2=w2, b2=b2).items()}
L = (t["w2"] * torch.sigmoid(t["w1"] * x + t["b1"]) + t["b2"] - y) ** 2
L.backward()
print("autograd: " + "  ".join(f"d{k}={v.grad.item():.4f}" for k, v in t.items()))

# one gradient-descent step
lr = 0.5
w1, b1, w2, b2 = w1 - lr * dL_dw1, b1 - lr * dL_db1, w2 - lr * dL_dw2, b2 - lr * dL_db2
y_hat = w2 / (1 + math.exp(-(w1 * x + b1))) + b2
print(f"after one step: loss={(y_hat - y) ** 2:.4f}")

Our hand-written gradients match PyTorch exactly. After a single step the loss drops from 0.1627 to 0.1564. Repeat this thousands of times on millions of examples and you have trained a neural network.

04Autograd: why you rarely do this by hand

Frameworks like PyTorch and TensorFlow record the computational graph as your code runs and apply the chain rule automatically when you call loss.backward(). This is called automatic differentiation (autograd).

for x_batch, y_batch in loader:          # mini-batches (Session 10)
    optimizer.zero_grad()                # clear old gradients
    loss = loss_fn(model(x_batch), y_batch)   # forward pass
    loss.backward()                      # backpropagation: fills p.grad for every weight
    optimizer.step()                     # gradient descent update (Session 13)

05Summary

• Chain rule: the effect along a chain = the product of the local effects.
• A computational graph splits the network into tiny operations.
• Forward saves values; backward multiplies local derivatives from the loss to each weight.
• Hand-computed gradients match PyTorch autograd exactly.
• Training loop: zero_grad → forward → loss.backward() → optimizer.step().

06Quick quiz

07References

• 3Blue1Brown: Backpropagation calculus
• CS231n: Backpropagation, intuitions
• PyTorch: Automatic differentiation with autograd
• Learning representations by back-propagating errors (Rumelhart, Hinton & Williams, 1986)