GenAI Course · Module 10 · Session 39

The Fine-Tuning Decision

⏱ 12 min read🎯 Intermediate🖼 2 diagrams🧪 1 live demo🐍 Python code

Fine-tuning sounds powerful, and it is, but it is also the most expensive and slowest tool in the box. Many teams fine-tune when a better prompt or RAG would have solved the problem in a day. In this session, let us learn when fine-tuning is the right call, and how to prepare data that actually works.

In this session

A clear decision path: prompt → RAG → fine-tune
Full vs partial fine-tuning (and where LoRA fits)
Dataset formats: chat JSONL and instruction format
How much data you really need
A runnable script to clean and split your dataset

01Fine-tune vs prompt vs retrieve

Always try the cheapest option first, and move to the next only when you have measured that it is not good enough:

Climb this ladder one step at a time, measuring on your evaluation set at each step.

Your problem	Best first tool
Model doesn’t know our policies or data	RAG
Answers must follow a strict format or tone	prompting (few-shot); fine-tune if still unreliable
Need a small, cheap, fast model for one narrow task	fine-tuning a small model
Domain language the model handles poorly (legal Hindi, medical codes)	fine-tuning (often plus RAG)
Facts change every week	RAG (fine-tuning would go stale)

Demo · Should we fine-tune?

👨‍🍳

Like training a new chef. If he doesn’t know today’s menu, give him the menu card (RAG). If he needs to cook in your restaurant’s style every single time, train him in your kitchen (fine-tuning).

02Full vs partial fine-tuning

Approach	What is trained	Cost	When
Full fine-tuning	every weight in the model	highest: needs many GPUs; a full copy per task	large budgets, big domain shift
Partial (freeze layers)	only the top layers	medium	small models, classic transfer learning
PEFT: LoRA / QLoRA	small add-on matrices (well under 1% of weights)	low: often a single GPU	the default for LLMs today (Session 40)

Why parameter-efficient fine-tuning (PEFT) became the standard.

03Preparing the dataset

Fine-tuning learns exactly what your examples show, including their mistakes. The two common formats:

chat_format.jsonl

{"messages": [{"role": "system", "content": "Classify the ticket. Reply in JSON."},
              {"role": "user", "content": "My card was blocked after a failed ATM withdrawal in Pune"},
              {"role": "assistant", "content": "{\"category\": \"card_block\", \"priority\": \"high\"}"}]}

instruction_format.jsonl

{"instruction": "Classify the ticket and reply in JSON.",
 "input": "Need my FD interest certificate for tax filing",
 "output": "{\"category\": \"documents\", \"priority\": \"low\"}"}

How much data?

Goal	Typical size
Format or style	a few hundred high-quality examples
Narrow classification / extraction	1,000 – 10,000 labelled examples
New domain language	tens of thousands of examples or more

Quality beats quantity: 500 clean, consistent examples beat 50,000 noisy ones.
Cover the real mix: all categories, edge cases, Hinglish, typos, long and short inputs.
Remove PII (names, phone numbers, account numbers) before training.
Hold out 10–20% as a test set the model never sees.

This script validates, de-duplicates and splits a dataset. We ran it on a sample file of 1,182 support tickets:

check_and_split.py

import json, random

rows = [json.loads(l) for l in open("train.jsonl", encoding="utf-8")]
ok = [r for r in rows
      if [m["role"] for m in r["messages"]] == ["system", "user", "assistant"]
      and all(m["content"].strip() for m in r["messages"])]
unique = list({json.dumps(r, sort_keys=True): r for r in ok}.values())     # drop exact duplicates

random.seed(42); random.shuffle(unique)
cut = int(0.9 * len(unique))
train, test = unique[:cut], unique[cut:]
print(f"read {len(rows)} | valid {len(ok)} | unique {len(unique)} | train {len(train)} | test {len(test)}")

read 1182 | valid 1180 | unique 1155 | train 1039 | test 116

Two broken rows (missing system message or empty answer) and 25 exact duplicates were removed before training. Small checks like this prevent big surprises.

04Summary

Climb the ladder: prompt → RAG → fine-tune, measuring at each step.
Fine-tune for behaviour; use RAG for knowledge.
PEFT (LoRA/QLoRA) is the default way to fine-tune LLMs.
Use chat or instruction JSONL; clean, de-duplicate, remove PII and hold out a test set.

05Quick quiz

1. The bot quotes old FD rates. Best fix?

2. A good reason to fine-tune is…

3. Roughly how many examples for teaching a format or style?

4. Why hold out a test set?

5. What does LoRA train?

06References

Created by Ajeetkumar