Module 7 cheat sheet: Pretrained Models & LLMs
Use pretrained models from the Hugging Face Hub, control their output with decoding settings, shrink them with quantization, and measure them properly.
Key ideas
- Autoregressive models predict the next token (GPT); autoencoding models fill blanks (BERT). S23 →
- Read the model card: task, size, licence and limits. S23 →
pipeline()is the fastest start; AutoTokenizer + AutoModel give full control. S23 →- Choose closed APIs or open weights by quality, cost, privacy and control. S24 →
- Adaptation ladder: prompting → RAG → PEFT → full fine-tuning. S25 →
Memory rule of thumb
GPU memory ≈ parameters × bytes per parameter
| Precision | Bytes | 7B model |
|---|---|---|
| FP32 | 4 | ~28 GB |
| FP16 / BF16 | 2 | ~14 GB |
| INT8 | 1 | ~7 GB |
| INT4 (GGUF, AWQ, GPTQ) | 0.5 | ~3.5–4 GB |
Add extra for the KV cache and activations.
Decoding settings
| Setting | What it does | Use for |
|---|---|---|
| Greedy | Always the top token | Short factual answers |
| Beam search | Keeps several candidate sentences | Translation |
| Temperature | Low = focused, high = varied | 0–0.3 facts/code; 0.7–1 creative |
| Top-k / top-p | Limit choices to the likeliest tokens | Creative text without nonsense |
| Repetition penalty | Discourages repeating | Stops loops |
Evaluation metrics
| Metric | Measures | Watch out |
|---|---|---|
| Perplexity | How surprised the model is by real text (lower = better) | Says nothing about truth |
| BLEU | Word overlap with a reference (translation) | Misses correct paraphrases |
| ROUGE | Overlap focused on recall (summaries) | Same as BLEU |
| Human / LLM judge | Helpfulness, correctness, tone | Needs clear rubric |
Code: three lines with Hugging Face
# pip install transformers torch
from transformers import pipeline
sentiment = pipeline("sentiment-analysis")
print(sentiment("The new UPI app is super fast and easy to use!"))No API key needed: the model downloads from the Hub.
Common mistakes
- High temperature for facts or code: more random, more mistakes.
- Judging a model on BLEU/ROUGE alone.
- Ignoring the licence on the model card before using it at work.
- Estimating memory from parameters only and forgetting the KV cache.
Created by Ajeetkumar · EasyLearnSpace