Module 4 cheat sheet: Sequential Models for NLP

RNNs read text in order with a memory; LSTMs and GRUs make that memory last; seq2seq turns one sequence into another, and its limits lead to attention.

All modules 📖 Glossary

Key ideas

RNN: hₜ = tanh(xₜWₓ + hₜ₋₁Wₕ + b), same weights at every step. S14 →
BPTT = backprop through the unrolled steps; gradients vanish over long sequences. S14 →
LSTM: cell state + forget, input and output gates. S15 →
GRU: 2 gates, no separate cell, ~25% fewer parameters. S15 →
Seq2seq: encoder → one context vector → decoder. S16 →
The context vector is a bottleneck; attention removes it. S16 →

LSTM equations

cₜ = f ⊙ cₜ₋₁ + i ⊙ g

hₜ = o ⊙ tanh(cₜ)

f, i, o = sigmoid gates (0 to 1)
g = tanh candidate information
forget gate near 1 = remember for a long time

Which sequence model?

Model	Parameters (vs RNN)	Use when
RNN	1×	short sequences, teaching
GRU	3×	faster, smaller data
LSTM	4×	longer sequences
Bi-LSTM	8×	whole text available (NER, classification)
Transformer	much larger	almost all NLP today (Module 5+)

Measured in this module

Experiment	Result
Gradient to token 1 after 100 steps	RNN ~10⁻²⁸ · LSTM with open forget gate ~10⁻²
Seq2seq reversing digits	100% at length 4 → under 10% at length 12

Code: a BiLSTM layer

import torch.nn as nn
lstm = nn.LSTM(input_size=128, hidden_size=256,
               batch_first=True, bidirectional=True)
out, (h, c) = lstm(x)     # out: (batch, seq, 512)

Common mistakes

Using a bidirectional model for live next-word prediction.
Not clipping gradients when training RNNs.
Expecting a seq2seq context vector to hold a whole long document.
Confusing teacher forcing (training) with generation (inference).

Created by Ajeetkumar · EasyLearnSpace