Recurrent Neural Networks (RNN): Explained Visually

Why do we need RNNs?

Many kinds of data are sequences, where order matters:

• Text: words in a sentence
• Speech: sound frames over time
• Finance: daily stock prices
• Sensors: temperature every minute

A normal feed-forward neural network has three problems with sequences:

1. Fixed input size. It expects exactly, say, 10 inputs. Sentences can have 3 words or 300.
2. No sense of order. It treats input 1 and input 7 as unrelated features.
3. No sharing. It learns separately what "good" means at position 2 and at position 9.

An RNN fixes all three with one idea: use the same small network at every step, and pass a memory from one step to the next.

How an RNN works

At each time step t, the RNN takes two inputs: the current item x_t (for example, a word) and its previous memory h_t-1. It mixes them and produces a new memory h_t:

If we need an output at that step (for example, the next word), we add:

There are only three weight matrices, and they are reused at every step:

Unrolling the RNN

It is the same cell used again and again. To see what happens over time, we draw one copy per word. This is called unrolling. Type any sentence below and press Play for the forward pass. The four coloured squares in each cell are the hidden state (mint is positive, pink is negative). Then switch to Backward pass to see the training signal travel back.

A worked example by hand

Let us use a tiny RNN with one number as memory. Let W_x = 0.5, W_h = 0.8 and b = 0. Input sequence: x = [1, 0, 0]. That is, one signal at the start, then nothing.

The first input is still remembered at step 3, but it fades a bit every step. Over 30 steps it would be close to zero. Keep that in mind; it is the key to the vanishing gradient problem below.

Types of RNN architectures

Training an RNN: backpropagation through time

An RNN is trained like any neural network: compute a loss, then use gradients to adjust W_x, W_h and W_y. Because the network is unrolled over time, the gradient must travel backwards through every time step. This is called Backpropagation Through Time (BPTT).

Each step back, the gradient is multiplied by W_h and by the derivative of tanh (which is at most 1, often around 0.5–0.8). After k steps, the gradient reaching an early word is roughly:

The vanishing and exploding gradient problem

Raising a number to a large power gives one of two results:

for w in (0.5, 0.9, 1.0, 1.5):
    print(f"factor {w}: after 10 steps = {w**10:.4f}, after 50 steps = {w**50:.2e}")

factor 0.5: after 10 steps = 0.0010, after 50 steps = 8.88e-16
factor 0.9: after 10 steps = 0.3487, after 50 steps = 5.15e-03
factor 1.0: after 10 steps = 1.0000, after 50 steps = 1.00e+00
factor 1.5: after 10 steps = 57.6650, after 50 steps = 6.38e+08

• Vanishing gradients (factor below 1): early words get almost no learning signal. The RNN cannot learn that "France" at the start predicts "French" at the end.
• Exploding gradients (factor above 1): updates become huge, the loss jumps to NaN, and training breaks.

Explore it yourself:

How to fix it

RNN in Python

Forward pass from scratch in NumPy

import numpy as np

np.random.seed(0)
vocab = {"I": 0, "love": 1, "deep": 2, "learning": 3}
x_size, h_size = len(vocab), 3

W_x = np.random.randn(h_size, x_size) * 0.5   # input  -> hidden
W_h = np.random.randn(h_size, h_size) * 0.5   # hidden -> hidden (the "memory" weights)
b = np.zeros(h_size)

def one_hot(word):
    v = np.zeros(x_size); v[vocab[word]] = 1
    return v

h = np.zeros(h_size)                            # empty memory at the start
for word in ["I", "love", "deep", "learning"]:
    h = np.tanh(W_x @ one_hot(word) + W_h @ h + b)
    print(f"{word:>9} -> h = {h.round(3)}")

        I -> h = [ 0.707  0.732 -0.052]
     love -> h = [0.464 0.18  0.069]
     deep -> h = [ 0.599  0.591 -0.02 ]
 learning -> h = [0.881 0.437 0.533]

The final hidden state [0.881 0.437 0.533] is a summary of the whole sentence. A classifier on top of it could decide the sentiment.

Using nn.RNN in PyTorch

import torch
import torch.nn as nn

rnn = nn.RNN(input_size=50, hidden_size=64, batch_first=True)
x = torch.randn(8, 20, 50)        # 8 sentences, 20 words each, 50 numbers per word
output, h_n = rnn(x)

print(output.shape)   # hidden state at every step
print(h_n.shape)      # final hidden state only
print(sum(p.numel() for p in rnn.parameters()))

torch.Size([8, 20, 64])
torch.Size([1, 8, 64])
7424

The parameter count is 64×50 (W_x) + 64×64 (W_h) + 64 + 64 (two biases) = 7,424. The same weights are reused for all 20 words, so a longer sentence does not add parameters.

RNN vs feed-forward network vs LSTM

Applications of RNNs

• Next-word prediction on mobile keyboards
• Speech recognition and voice assistants
• Time-series forecasting: sales, weather, energy demand
• Music generation note by note
• Handwriting recognition from pen strokes
• Video understanding, frame by frame