Module 6 cheat sheet: Transformer Architecture & Variants

A transformer is a stack of identical blocks; the encoder understands, the decoder writes, and positions are added because attention ignores order.

All modules 📖 Glossary

Key ideas

Attention alone is order-blind: “dog bites man” = “man bites dog” without positions. S20 →
Encoder block: self-attention → Add & Norm → feed-forward → Add & Norm. S21 →
Decoder adds masked self-attention and cross-attention (reads the encoder output). S21 →
Training uses teacher forcing (all positions at once); inference writes one token at a time. S21 →
Pick a family by the shape of the output you need. S22 →

Positional encoding

PE(pos, 2i) = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

Type	Used in
Sinusoidal (fixed waves)	Original transformer (2017)
Learned positions	BERT, GPT-2
RoPE (rotates Q and K)	Llama, Mistral, Qwen and most modern LLMs
ALiBi (distance penalty)	BLOOM, MPT

Three families

Family	Examples	Best for
Encoder-only	BERT, RoBERTa, DeBERTa	Classification, NER, search embeddings
Decoder-only	GPT, Llama, Mistral, Qwen	Chat, writing, code, agents
Encoder-decoder	T5, BART	Translation, summarisation, rewriting

Code: one encoder block and a causal mask

import torch, torch.nn as nn
layer = nn.TransformerEncoderLayer(d_model=512, nhead=8,
                                   dim_feedforward=2048, batch_first=True)
x = torch.randn(1, 4, 512)          # 1 sentence, 4 tokens
print(layer(x).shape)               # same shape in and out: [1, 4, 512]
mask = nn.Transformer.generate_square_subsequent_mask(4)   # -inf = cannot look

Common mistakes

Using BERT to generate long text: it is built to understand, not to write.
Thinking the decoder can see future tokens during training: the mask blocks them.
Confusing residual connections (add the input back) with normalisation (rescale numbers).
Choosing a giant decoder for a simple label task a small encoder can do faster and cheaper.

Created by Ajeetkumar · EasyLearnSpace