Recurrent Neural Networks

The networks in Module 3 take a fixed-size input. But sentences have different lengths, and order matters: “not good” is different from “good, not bad”. A Recurrent Neural Network (RNN) reads one word at a time and keeps a running memory. Let us build one, unroll it, and find out why it forgets.

01The hidden state and unrolling

An RNN cell takes two things at each step: the current word and its own hidden state from the previous step. It mixes them and produces a new hidden state:

import numpy as np
np.random.seed(0)

words = ["my", "upi", "payment", "failed"]
emb = {w: np.random.randn(4) * 0.5 for w in words}      # 4-number word vectors (random for the demo)

H = 3                                                    # hidden state size
Wx, Wh, b = np.random.randn(4, H) * 0.5, np.random.randn(H, H) * 0.5, np.zeros(H)

h = np.zeros(H)                                          # memory starts empty
for t, w in enumerate(words, 1):
    h = np.tanh(emb[w] @ Wx + h @ Wh + b)                # SAME weights at every step
    print(f"t={t} read {w!r:10s} -> hidden state {h.round(2)}")
print("final h summarises the whole sentence; parameters:", Wx.size + Wh.size + b.size)

Notice there are only 24 parameters, whatever the sentence length, because the same three weight matrices are reused at every step. “Unrolling” just means drawing that one cell once per time step.

02RNNs in PyTorch, and bidirectional RNNs

PyTorch’s nn.RNN processes a whole batch of sequences. A bidirectional RNN runs one RNN left-to-right and another right-to-left, then joins their outputs, so every word sees both its past and its future:

import torch, torch.nn as nn
torch.set_num_threads(1)                    # same results on every run
torch.manual_seed(0)
x = torch.randn(2, 6, 8)                      # batch of 2 sentences, 6 tokens, 8 numbers per token

rnn = nn.RNN(input_size=8, hidden_size=16, batch_first=True)
out, h_n = rnn(x)
print("uni   outputs:", tuple(out.shape), " final hidden:", tuple(h_n.shape))

birnn = nn.RNN(input_size=8, hidden_size=16, batch_first=True, bidirectional=True)
out, h_n = birnn(x)
print("bi    outputs:", tuple(out.shape), " final hidden:", tuple(h_n.shape), " <- forward + backward")

• outputs holds a hidden state for every token: use it for tagging each word (NER, POS).
• h_n is the final hidden state: use it to classify the whole sentence (sentiment, intent).
• Bidirectional doubles the output size (16 + 16 = 32).

03Backpropagation Through Time (BPTT)

To train an RNN we unroll it over the sequence and run normal backpropagation (Session 11) through the unrolled graph. This is Backpropagation Through Time. Because the same weights appear at every step, their gradients from all steps are added up.

The catch: the gradient that reaches early words has been multiplied by the recurrent weights and the tanh derivative once per step. We measure how much the first token still affects the last output:

import torch, torch.nn as nn
torch.set_num_threads(1)                    # same results on every run
torch.manual_seed(0); rnn = nn.RNN(8, 32, batch_first=True)

for T in [5, 20, 50, 100]:
    torch.manual_seed(0)
    x = torch.randn(1, T, 8, requires_grad=True)
    out, _ = rnn(x)
    out[0, -1].sum().backward()                     # loss uses only the last step
    print(f"length {T:3d}: gradient reaching token 1 = {x.grad[0, 0].abs().mean().item():.0e}")

• Exploding gradients also happen in RNNs; clip them with clip_grad_norm_.
• Truncated BPTT backpropagates through only the last k steps to save memory on very long sequences.
• RNNs process tokens one after another, so they cannot use GPU parallelism well. Transformers fix this too (Module 5).

04Summary

• An RNN reads a sequence one step at a time, updating a hidden state (its memory).
• The same weights are reused at every step; unrolling draws one copy per step.
• Bidirectional RNNs read both ways: great when the whole text is available.
• BPTT = backprop through the unrolled steps.
• Gradients vanish over long sequences, so plain RNNs forget early words.

05Quick quiz

06References

• The Unreasonable Effectiveness of RNNs (Karpathy)
• PyTorch nn.RNN
• Learning long-term dependencies with gradient descent is difficult (Bengio et al., 1994)