Project 9: Fine-Tune a Small LLM with LoRA / QLoRA

Show that you can adapt a model, not just call an API. You'll fine-tune a small open model (1–8B) with QLoRA on a free Colab GPU for a narrow task, then prove with an evaluation that it beats prompting the base model.

What you'll build

• A task with a clear format, for example support-ticket → JSON (intent, priority, summary), or a domain Q&A style.
• Data prep: 1–3k instruction examples in chat format, with a train/validation/test split.
• QLoRA fine-tuning (4-bit base + LoRA adapters) with PEFT/TRL on Colab.
• An evaluation comparing base zero-shot vs base few-shot vs fine-tuned on accuracy, JSON validity and latency.

Architecture

Raw examples → clean + dedupe → chat template (system/user/assistant) → train/val/test
Base model (4-bit NF4) + LoRA (r=16, alpha=32, target q,k,v,o) → SFTTrainer (2–3 epochs, lr 2e-4)
→ adapter weights (~50 MB) → merge (optional) → evaluate → push to Hugging Face Hub → serve (vLLM / Ollama GGUF)

Tech stack

Hugging Face Transformers, PEFT, TRL (SFTTrainer), bitsandbytes, datasets, Colab T4/L4 GPU, the evaluate library, optionally vLLM or Ollama for serving.

Step by step

1. Pick the task and metric first (e.g. exact-match intent accuracy plus JSON validity).
2. Build the dataset: existing labelled data turned into instructions, or synthetic data from a big model, filtered and spot-checked by you.
3. Format with the model's chat template; hold out a test set.
4. Baselines: prompt the base model (zero-shot and few-shot) and record the scores.
5. Train with QLoRA: watch the validation loss for overfitting.
6. Evaluate on the test set: accuracy, JSON validity, latency, plus general-ability spot checks (catastrophic forgetting).
7. Publish the adapter with a model card covering data, intended use, limitations and metrics.

Starter code

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
import torch
base = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, quantization_config=BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16), device_map="auto")
ds = load_dataset("json", data_files={"train": "train.jsonl", "validation": "val.jsonl"})   # {"messages": [...]}
peft = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], task_type="CAUSAL_LM")
cfg = SFTConfig(output_dir="out", num_train_epochs=2, per_device_train_batch_size=4, gradient_accumulation_steps=4,
                learning_rate=2e-4, logging_steps=20, eval_strategy="steps", eval_steps=100, bf16=True)
SFTTrainer(model=model, args=cfg, train_dataset=ds["train"], eval_dataset=ds["validation"], peft_config=peft, processing_class=tok).train()

Deploy it

Push the adapter to the Hugging Face Hub and add a Gradio demo Space. Optionally convert to GGUF and run it locally with Ollama.

Put it on your resume

• Fine-tuned Qwen2.5-1.5B with QLoRA on 2.4k ticket examples, raising intent accuracy from 71% (few-shot base) to 92% with 99.6% valid JSON, served at about 3× lower latency than a hosted API.
• Published the adapter and model card on Hugging Face, with an evaluation comparing zero-shot, few-shot and fine-tuned variants.

Interview questions you'll get

• When should you fine-tune instead of using prompting or RAG?
• Explain LoRA vs QLoRA. What do r and alpha control?
• How did you avoid overfitting and catastrophic forgetting?
• How did you build and validate the training data?

Practise these in Mock Interview → LLMs & Prompting.

Stretch goals

• DPO preference tuning on chosen/rejected pairs.
• Compare r = 8/16/64 and target modules (an ablation table).
• Serve with vLLM and benchmark the throughput.