Long Short-Term Memory (LSTM): Explained Visually

Why do we need LSTM?

Read this sentence: "I grew up in France, moved to Bangalore for work, learned to cook dosa, and after all these years I still speak fluent ____."

You know the answer is French, because you remembered the word France from the start of the sentence. To predict the blank, a model must carry one piece of information across almost 20 words. This is called a long-term dependency.

A plain RNN struggles with this. During training, the error signal has to travel backwards through every word. At every step it is multiplied by a number. If that number is smaller than 1, the signal shrinks again and again until it disappears. This is the vanishing gradient problem. If the number is larger than 1, it grows until training breaks. That is the exploding gradient problem.

Play with the visual below. The pink bars show how much of the signal from word 1 is still alive in a plain RNN. The mint bars show an LSTM with a forget gate close to 1.

The big idea: a memory highway

The LSTM solves this with a simple design trick. It keeps two kinds of memory:

The cell state runs like a straight highway through time. Information can ride along it for many steps almost untouched. Three gates stand beside the highway and control traffic.

LSTM architecture

Every LSTM cell takes three inputs and produces two outputs:

• Inputs: the current word x_t, the previous hidden state h_t-1 and the previous cell state c_t-1.
• Outputs: the new hidden state h_t and the new cell state c_t.

Inside the cell there are four small neural network layers: three gates that use the sigmoid function (output between 0 and 1) and one candidate layer that uses tanh (output between -1 and 1). A gate value of 0 means "let nothing through" and 1 means "let everything through".

Use the interactive cell below. Move the sliders to change the inputs, try the three scenarios, then press Play to watch the data flow one gate at a time. Every number is computed live.

How an LSTM works, step by step

Step 1: Forget gate, what should we erase?

The forget gate looks at the previous hidden state and the current word, and outputs a number between 0 and 1 for every value in the cell state.

• f = 1 means keep this memory completely.
• f = 0 means erase it completely.

Example: in "Ravi is a doctor. Priya is a teacher. She…", when the model reads the new subject Priya, the forget gate can erase "the subject is male" so that she makes sense.

Step 2: Input gate and candidate, what new information should we save?

This step has two parts that work together:

• The candidate g_t is the new information itself (for example, "the subject is now female").
• The input gate i_t decides how much of that candidate to actually write. Not every word is worth remembering. Words like the or is usually get a small input gate value.

Step 3: Update the cell state

Now the old memory and the new information are combined:

Here ⊙ means element-wise multiplication. The first part is "what we kept", the second part is "what we added". This single line is the heart of the LSTM.

Step 4: Output gate, what should we show?

The cell does not reveal its whole memory at every step. The output gate decides which parts are useful right now:

The new hidden state h_t goes to the next time step and to the next layer (for example, a classifier that predicts the next word).

A worked example with real numbers

Let us run one LSTM step with a single memory unit so you can follow every number. Suppose the gates have already been computed from x_t and h_t-1:

Now compute:

1. Kept memory = f × c_t-1 = 0.90 × 0.80 = 0.72
2. New information = i × g = 0.30 × (-0.50) = -0.15
3. New cell state c_t = 0.72 + (-0.15) = 0.57
4. tanh(0.57) = 0.515
5. New hidden state h_t = 0.60 × 0.515 = 0.31

The memory went down a little (from 0.80 to 0.57) because the new word disagreed, but most of it survived. Set the sliders in the visual above to see the same behaviour.

How LSTM solves the vanishing gradient problem

In a plain RNN, the gradient from one step to the previous step is multiplied by the recurrent weights and the tanh derivative every time. In an LSTM, look at how the cell state is updated: c_t = f_t ⊙ c_t-1 + i_t ⊙ g_t. If we take the derivative of c_t with respect to c_t-1 along the highway, we get:

So the gradient is multiplied only by the forget gate. The network can learn to keep f close to 1 for information that matters, so the gradient survives for 100+ steps. This additive path is the same idea later used in ResNet's skip connections.

LSTM in Python

LSTM from scratch in NumPy

This code implements exactly the four equations above. Read it next to the formulas.

import numpy as np

np.random.seed(42)
sigmoid = lambda z: 1 / (1 + np.exp(-z))

input_size, hidden_size = 3, 2
# one weight matrix and bias per gate: forget, input, candidate, output
W = {g: np.random.randn(hidden_size, hidden_size + input_size) * 0.5 for g in "figo"}
b = {g: np.zeros(hidden_size) for g in "figo"}
b["f"] += 1.0  # common trick: start by remembering

def lstm_step(x_t, h_prev, c_prev):
    z = np.concatenate([h_prev, x_t])           # [h_{t-1}, x_t]
    f = sigmoid(W["f"] @ z + b["f"])            # forget gate
    i = sigmoid(W["i"] @ z + b["i"])            # input gate
    g = np.tanh(W["g"] @ z + b["g"])            # candidate memory
    o = sigmoid(W["o"] @ z + b["o"])            # output gate
    c_t = f * c_prev + i * g                     # new cell state
    h_t = o * np.tanh(c_t)                       # new hidden state
    return h_t, c_t, f

sequence = np.random.randn(4, input_size)       # 4 time steps
h, c = np.zeros(hidden_size), np.zeros(hidden_size)
for t, x_t in enumerate(sequence, 1):
    h, c, f = lstm_step(x_t, h, c)
    print(f"t={t}  forget={f.round(2)}  cell={c.round(3)}  hidden={h.round(3)}")

t=1  forget=[0.8  0.77]  cell=[-0.033  0.055]  hidden=[-0.015  0.016]
t=2  forget=[0.46 0.74]  cell=[0.726 0.379]  hidden=[0.387 0.279]
t=3  forget=[0.84 0.7 ]  cell=[0.515 0.127]  hidden=[0.205 0.05 ]
t=4  forget=[0.68 0.54]  cell=[ 0.334 -0.096]  hidden=[ 0.134 -0.076]

Notice two things in the output. First, the forget values start high (about 0.8) because we added 1.0 to the forget bias. This "remember by default" trick is used in real libraries too. Second, the cell state carries value from step 2 into steps 3 and 4 even as new words arrive.

LSTM for sentiment analysis in PyTorch

In practice you use a library. This model reads a review of 50 word IDs and predicts positive or negative.

import torch
import torch.nn as nn

class SentimentLSTM(nn.Module):
    def __init__(self, vocab_size=10_000, embed_dim=100, hidden=128):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden, batch_first=True)
        self.fc = nn.Linear(hidden, 1)

    def forward(self, x):
        out, (h_n, c_n) = self.lstm(self.embed(x))   # h_n: final hidden state
        return torch.sigmoid(self.fc(h_n[-1]))      # probability of "positive"

model = SentimentLSTM()
batch = torch.randint(0, 10_000, (32, 50))          # 32 reviews, 50 words each
print(model(batch).shape)
print(sum(p.numel() for p in model.lstm.parameters()))

torch.Size([32, 1])
117760

Where does 117,760 come from? An LSTM layer has 4 weight blocks (one per gate and candidate). Each block has input weights (128 × 100), recurrent weights (128 × 128) and, in PyTorch, two bias vectors of 128. So the total is 4 × (128×100 + 128×128 + 128 + 128) = 4 × 29,440 = 117,760. That is 4 times more than a plain RNN of the same size.

Types of LSTM

LSTM vs RNN vs GRU vs Transformer

Applications of LSTM

• Time-series forecasting: stock prices, electricity demand, sales and sensor readings.
• Speech recognition: converting audio frames to text (used in early Siri and Google Voice).
• Machine translation: the original Google Neural Machine Translation (2016) used stacked LSTMs.
• Text generation and next-word prediction: mobile keyboards.
• Anomaly detection: fraud in card transactions, faults in machines.
• Healthcare: reading patient vital signs over time.

Advantages and limitations