Module 10 cheat sheet: Fine-Tuning Techniques
Fine-tune to change behaviour, not to add facts; LoRA and QLoRA make it cheap enough for a single GPU.
Key ideas
- Climb the ladder: prompt → RAG → fine-tune, measuring at every step. S39 →
- Fine-tune for behaviour (format, tone, labels); use RAG for knowledge. S39 →
- Data: chat or instruction JSONL; clean, de-duplicate, remove PII, keep a held-out test set. S39 →
- LoRA freezes W and learns a small low-rank update B·A. S40 →
- QLoRA = 4-bit NF4 base model + LoRA adapters. S40 →
- Always compare with the base model on the same held-out set. S40 →
LoRA in one line
W′ = W + (α / r) · B · A
- Trainable numbers per matrix = r × (d_in + d_out).
- Example: r = 8 on q and v of a 7B model (4096 wide, 32 layers) → 4.19 M trainable, about 0.06% of the model.
- Merge B·A into W after training for no extra latency, or keep adapters separate to swap per task.
Full vs partial vs PEFT
| Approach | Trains | Hardware | When |
|---|---|---|---|
| Full fine-tuning | All weights | Many GPUs | Big domain shift, lots of data |
| Partial (freeze layers) | Top layers / head | One GPU | Small models, classifiers |
| LoRA | <1% extra weights | One GPU | Default for LLMs |
| QLoRA | LoRA on a 4-bit base | One consumer/Colab GPU | Big models, small budget |
Main settings
| Setting | Typical start |
|---|---|
| r (rank) | 8–16 (up to 64) |
| lora_alpha | about 2 × r |
| target_modules | q, k, v, o projections |
| learning rate | 1e-4 to 2e-4 |
| epochs | 1–3 (watch the eval loss) |
Code: LoRA config
# pip install peft
from peft import LoraConfig
lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"])
# pass to TRL SFTTrainer(peft_config=lora, ...) (see Session 40)Common mistakes
- Fine-tuning to teach new facts that change often (use RAG instead).
- No held-out test set, so you cannot prove the model improved.
- Too many epochs on a small dataset: the model memorises (overfits).
- Training data that ignores the model’s chat template.
- Leaving customer PII in the training data.
Created by Ajeetkumar · EasyLearnSpace