Motivation & Self-Attention

Every modern LLM, from BERT to GPT to Claude, is built on one simple idea: let every word look at every other word. This idea is called self-attention. In this session, let us see why we needed it, and exactly what it computes.

01The problem with RNNs

Before transformers, language models were mostly RNNs (Recurrent Neural Networks) and their improved versions, LSTMs and GRUs. An RNN reads a sentence one word at a time, and carries a small “memory” vector from one word to the next.

RNNs had three big problems:

• They forget. Information from early words fades as it passes through many steps (the “vanishing gradient” problem). LSTMs helped, but did not fully solve it.
• They are slow to train. Word 20 cannot be processed until words 1 to 19 are done, so GPUs, which love doing many things in parallel, sit mostly idle.
• Long paths. For word 20 to use word 1, the signal must travel through 19 steps. Each step is a chance to lose it.

02The big idea: every word looks at every word

In 2017, the famous paper “Attention Is All You Need” removed recurrence completely. Instead of passing a memory along a chain, each word directly looks at all the other words and decides which ones matter for it.

03Query, Key and Value

How does a word decide which other words matter? Self-attention gives every word three roles. The easiest way to understand them is a YouTube search.

In self-attention, every word is all three at once. Each word creates its own query, key and value by multiplying its embedding with three learned weight matrices:

04What attention actually computes

For one word, attention does three simple things:

1. Score: compare its query with every word’s key using a dot product. High score = good match.
2. Softmax: turn the scores into weights between 0 and 1 that add up to 1.
3. Mix: take a weighted sum of all the value vectors. This is the word’s new, context-aware vector.

Let us do it with real numbers. Our sentence has three words, I love chai, and we compute the new vector for “chai”. Press Next to go step by step:

Now see it on a real sentence. Pick a word and watch where its attention goes:

05Self-attention from scratch in NumPy

Here is the whole idea in about 10 lines. Run it in Colab or Jupyter:

import numpy as np
np.random.seed(42)

X = np.random.randn(4, 8)                     # 4 tokens, each an 8-number embedding
Wq, Wk, Wv = [np.random.randn(8, 4) for _ in range(3)]

Q, K, V = X @ Wq, X @ Wk, X @ Wv              # queries, keys, values
scores = Q @ K.T                              # how well each query matches each key
weights = np.exp(scores - scores.max(axis=1, keepdims=True))
weights /= weights.sum(axis=1, keepdims=True) # softmax, row by row
output = weights @ V                          # weighted mix of values

print(output.shape)
print(weights.round(2))

Each row of weights adds up to 1: it shows how much one token looks at each of the four tokens. Notice how extreme some rows are (0.97, 1.00). With bigger vectors, this gets worse. In Session 18 we fix it by scaling the scores.

06Summary

• RNNs read word by word, so they forget long-range information and are slow to train.
• Self-attention connects every word to every word directly, and runs in parallel.
• Each word makes a query (what I need), a key (what I offer) and a value (what I carry).
• Attention = score (q · k) → softmax → weighted sum of values.

07Quick quiz

08References

• Vaswani et al., Attention Is All You Need (2017)
• Jay Alammar, The Illustrated Transformer