Module 3 cheat sheet: Deep Learning Fundamentals

Neurons, layers, loss, gradients and optimizers: the engine inside every modern AI model.

All modules 📖 Glossary

Key ideas

Neuron = activation(Σ weight × input + bias). S9 →
One perceptron cannot learn XOR; layers can. S9 →
Gradient descent: weight −= learning rate × gradient. S10 →
Backprop = chain rule from the loss back to every weight. S11 →
ReLU + He initialisation keeps gradients alive in deep networks. S12 →
Adam/AdamW is the default optimizer; watch the validation loss. S13 →

Formulas

params per layer = n_in × n_out + n_out

MSE = mean((ŷ − y)²) · BCE = −[y log p + (1−y) log(1−p)]

steps = epochs × (data ÷ batch size)

Pick the output layer

Task	Activation	Loss
Yes / no	sigmoid	binary cross-entropy
One of many classes	softmax	categorical cross-entropy
A number	none (linear)	MSE or MAE
Hidden layers	ReLU (GELU in transformers)	set by the output task

Optimizers

Optimizer	Idea
SGD	plain step down the gradient
Momentum	build up speed in a steady direction
RMSProp	scale each weight’s step by its recent gradient size
Adam	momentum + RMSProp; robust default
AdamW	Adam + proper weight decay; used for LLMs

Code: the training loop

for x, y in loader:
    optimizer.zero_grad()            # clear old gradients
    loss = loss_fn(model(x), y)      # forward pass
    loss.backward()                  # backpropagation
    optimizer.step()                 # update weights
model.eval()                         # before validation: turns dropout off

Common mistakes

Feeding unscaled inputs, so activations saturate.
Learning rate too high: the loss explodes to NaN.
No validation set, so overfitting goes unnoticed.
Forgetting model.eval() before evaluation.
Sigmoid in every hidden layer of a deep network (vanishing gradients).

Created by Ajeetkumar · EasyLearnSpace