Artificial Neural Networks

Every model in this course, from a spam filter to GPT, is built from one tiny unit: the artificial neuron. In this session we build one by hand, connect many into layers, run a forward pass with real numbers, and count the parameters a network learns.

01The perceptron

The perceptron (Frank Rosenblatt, 1958) is the simplest neural network: one neuron. It multiplies each input by a weight, adds a bias, and outputs 1 if the total is above zero.

Here a perceptron learns the AND rule from four examples, using the classic update: if the prediction is wrong, nudge the weights towards the right answer.

import numpy as np

# Learn the AND rule: output 1 only when both inputs are 1
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 0, 0, 1])

w, b, lr = np.zeros(2), 0.0, 0.1
for epoch in range(10):
    errors = 0
    for xi, target in zip(X, y):
        pred = int(xi @ w + b > 0)          # weighted sum, then a step
        update = lr * (target - pred)       # perceptron learning rule
        w += update * xi; b += update
        errors += int(update != 0)
    if errors == 0:
        print(f"learned in {epoch + 1} epochs: w={w}, b={b:.1f}"); break

print("predictions:", [int(x @ w + b > 0) for x in X])

02Multi-layer perceptrons and forward propagation

Stack neurons in layers and the network can draw curved boundaries, solving XOR and much more. A multi-layer perceptron (MLP) has an input layer, one or more hidden layers and an output layer.

Forward propagation is just running the numbers from left to right: each layer computes activation(inputs @ W + b) and passes the result on. Here is a 3 → 4 → 1 network scoring one loan application:

import numpy as np
np.random.seed(0)

def relu(z): return np.maximum(0, z)
def sigmoid(z): return 1 / (1 + np.exp(-z))

# one loan application: [income (lakh/yr), loan amount (lakh), years employed]
x_raw = np.array([12.0, 5.0, 3.0])
x = x_raw / np.array([10.0, 10.0, 10.0])            # scale inputs to a small range first

W1, b1 = np.random.randn(3, 4) * 0.5, np.zeros(4)    # layer 1: 3 inputs -> 4 neurons
W2, b2 = np.random.randn(4, 1) * 0.5, np.zeros(1)    # layer 2: 4 -> 1 output

h = relu(x @ W1 + b1)            # hidden layer
p = sigmoid(h @ W2 + b2)         # probability of approval
print("hidden layer:", h.round(2))
print("approval probability:", p.round(3))
print("same network on unscaled input:", sigmoid(relu(x_raw @ W1 + b1) @ W2 + b2).round(3))
print("parameters:", W1.size + b1.size + W2.size + b2.size)

03Weights, biases, layers and capacity

A layer with n_in inputs and n_out neurons has n_in × n_out weights plus n_out biases. PyTorch counts them for us:

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(3, 64), nn.ReLU(),     # 3*64 weights + 64 biases
    nn.Linear(64, 64), nn.ReLU(),    # 64*64 + 64
    nn.Linear(64, 1), nn.Sigmoid(),  # 64*1 + 1
)
for name, p in model.named_parameters():
    print(f"{name:10s} {tuple(p.shape)}")
print("total:", sum(p.numel() for p in model.parameters()))

Check by hand: (3×64 + 64) + (64×64 + 64) + (64×1 + 1) = 256 + 4,160 + 65 = 4,481. ✓

Capacity is how complex a pattern a network can learn. More neurons and layers mean more capacity, but also more risk of memorising the training data (overfitting, Session 13).

04Summary

• A neuron computes activation(Σ weights × inputs + bias).
• A single perceptron learns straight-line rules (AND, OR) but not XOR.
• An MLP stacks layers; forward propagation runs inputs through them left to right.
• Weights and biases are learned; width and depth are design choices.
• Parameters per layer = n_in × n_out + n_out. Scale inputs before training.

05Quick quiz

06References

• 3Blue1Brown: But what is a neural network?
• PyTorch: Build the neural network
• Michael Nielsen: Neural Networks and Deep Learning, ch. 1