GenAI Course · Module 6 · Session 20

Architecture & Positional Encoding

⏱ 12 min read🎯 Intermediate🖼 2 diagrams🧪 1 live demo🐍 Python code

In Module 5 we learned attention. Now let us zoom out and see the full transformer: how the encoder and decoder work together to translate a sentence, and one surprising problem: attention has no idea about word order. Let us see how we fix that.

In this session

The encoder–decoder transformer, end to end, with a translation example
Why attention alone cannot tell “dog bites man” from “man bites dog”
Sinusoidal positional encoding, with a live demo
Learned positions, and modern methods like RoPE and ALiBi

01The transformer, end to end

The original transformer (2017) was built for translation. It has two halves: an encoder that reads and understands the input sentence, and a decoder that writes the output sentence one token at a time, while looking back at what the encoder understood.

The encoder reads the English sentence once. The decoder writes the Hindi sentence word by word, looking at the encoder’s output each time.

Part	What it does	Original size
Embedding	turns each token into a vector	d_model = 512
Encoder	self-attention + feed-forward, repeated	6 layers
Decoder	masked self-attention + cross-attention + feed-forward	6 layers
Output head	linear layer + softmax over the vocabulary	~37,000 tokens (shared vocab)

🗣️

Like a human interpreter. First they listen to the full sentence and understand it (encoder). Then they speak in the other language word by word, glancing at their understanding each time (decoder).

02Why word order has to be injected

Here is the surprise. Self-attention compares every word with every other word, but the calculation does not care where a word sits. If you shuffle the words, each word gets exactly the same attention scores, just in a different row.

Without position information, attention sees both sentences as the same “bag of words”.

So the transformer must be told the position of each token. We do this by adding a positional encoding vector to each token embedding before the first layer:

input to layer 1 = token embedding + positional encoding

RNNs never needed this, because they read words one by one, so order was built in. Transformers read everything in parallel, so order must be added on purpose.

03Sinusoidal positional encoding

The original paper used fixed sine and cosine waves of different speeds. For position pos and dimension pair i:

PE(pos, 2i) = sin(pos / 10000^2i/d) · PE(pos, 2i+1) = cos(pos / 10000^2i/d)

Early dimensions change quickly from one position to the next (fast waves). Later dimensions change slowly (slow waves). Together, every position gets a unique pattern, like a fingerprint.

🕰️

Like the hands of a clock. The seconds hand moves fast, the minutes hand slower, the hours hand slowest. Reading all three together tells you the exact time. Fast and slow waves together tell the model the exact position.

Demo · Positional encoding for each position (first 8 of 512 dimensions)

positional_encoding.py

import numpy as np

def positional_encoding(n_positions, d_model):
    pe = np.zeros((n_positions, d_model))
    pos = np.arange(n_positions)[:, None]
    div = np.power(10000, np.arange(0, d_model, 2) / d_model)
    pe[:, 0::2] = np.sin(pos / div)      # even dimensions
    pe[:, 1::2] = np.cos(pos / div)      # odd dimensions
    return pe

pe = positional_encoding(50, 512)
print(pe.shape)
print(pe[1, :4].round(3))

(50, 512) [0.841 0.54 0.822 0.57 ]

04Learned positions, RoPE and ALiBi

Sinusoidal encoding is only one option. Here is what popular models use:

Method	How it works	Used in	Good to know
Sinusoidal	fixed sine/cosine waves added to embeddings	Original Transformer	no extra parameters
Learned absolute	a trainable vector for each position 0, 1, 2…	BERT (512), GPT-2 (1,024)	cannot go beyond the trained length
RoPE (rotary)	rotates query and key vectors by an angle that depends on position	LLaMA, Mistral, Qwen, Gemma	encodes relative distance; can be stretched for longer context
ALiBi	adds a penalty to attention scores that grows with distance	BLOOM, MPT	handles longer inputs than seen in training

Why RoPE is popular: what matters most in language is often how far apart two words are, not their exact index. RoPE builds this relative distance directly into the attention scores, and tricks like RoPE scaling let models reach 128K-token contexts.

05Summary

The transformer has an encoder (understand) and a decoder (write).
Attention by itself is order-blind, so positions must be added.
Sinusoidal encodings use fast and slow waves to give each position a unique pattern.
Modern LLMs mostly use RoPE, which encodes relative distance.

06Quick quiz

1. Why does a transformer need positional encoding?

2. How is positional encoding combined with the token embedding?

3. In sinusoidal encoding, the early dimensions…

4. Which positional method do LLaMA and Mistral use?

5. In translation, which part writes the output sentence?

07References

Attention Is All You Need (Vaswani et al., 2017)
RoFormer: Rotary Position Embedding (Su et al., 2021)
ALiBi: Train Short, Test Long (Press et al., 2021)

Created by Ajeetkumar