Project 6: LLM & RAG Evaluation Harness

Most candidates build a chatbot; very few can prove it works. This project builds a reusable evaluation harness for RAG and LLM apps: golden datasets, retrieval and generation metrics, LLM-as-judge, regression reports and a CI gate. It's a strong signal for senior and LLMOps roles.

What you'll build

• A dataset format: question, reference answer, gold document IDs, tags.
• Metrics: recall@k, MRR, faithfulness, answer correctness, citation accuracy, latency and cost.
• A CLI: evalrag run --config configs/chunk800.yaml, producing an HTML/Markdown report and a comparison against the previous run.
• A GitHub Actions gate that fails the build if faithfulness drops by more than 3 points.

Architecture

configs/*.yaml (retriever, k, prompt, model) → runner → for each question: retrieve → generate → log
   → metrics: retrieval (gold IDs) + generation (LLM judge with rubric, calibrated on 30 human labels)
   → results.parquet → report.md (scores, deltas vs baseline, worst 10 failures by category)

Tech stack

Python, Pydantic, pandas, an LLM judge (with a fixed rubric and temperature 0), RAGAS or DeepEval (optional), Typer (CLI), Jinja2 (report), GitHub Actions.

Step by step

1. Golden set: write 50–100 realistic questions with domain experts; include unanswerable ones (expected: "I don't know").
2. Retrieval metrics: recall@k and MRR against the gold document IDs.
3. Judge prompt: score faithfulness (are all claims supported by the context?) and correctness (vs the reference) on a 1–5 scale with reasons; output JSON.
4. Calibrate the judge: label 30 answers yourself; check agreement (Spearman/κ) and adjust the rubric.
5. Cache LLM calls (hash of prompt + model) so re-runs are cheap.
6. Report: a table of metrics, deltas vs baseline, and failure buckets (retrieval miss / wrong synthesis / refusal error).
7. CI gate: run on pull requests that change prompts or configs.

Starter code

import json, hashlib, statistics as st
JUDGE = """Rate the ANSWER for faithfulness to CONTEXT (1-5): 5 = every claim supported, 1 = mostly unsupported.
Return JSON {"faithfulness": int, "unsupported_claims": [..]}."""
def recall_at_k(retrieved, gold, k=4): return len(set(retrieved[:k]) & set(gold)) / max(1, len(gold))
def mrr(retrieved, gold): return next((1 / (i + 1) for i, d in enumerate(retrieved) if d in gold), 0.0)
def judge(context, answer, cache={}):
    key = hashlib.sha1((context + answer).encode()).hexdigest()
    if key not in cache: cache[key] = json.loads(llm(system=JUDGE, user=f"CONTEXT:\n{context}\n\nANSWER:\n{answer}", temperature=0))
    return cache[key]
def run(dataset, pipeline):
    rows = []
    for ex in dataset:
        docs, ans = pipeline(ex["question"])
        rows.append({"r@4": recall_at_k([d.id for d in docs], ex["gold_ids"]), "mrr": mrr([d.id for d in docs], ex["gold_ids"]),
                     "faith": judge("\n".join(d.text for d in docs), ans)["faithfulness"]})
    return {k: round(st.mean(r[k] for r in rows), 3) for k in rows[0]}

Deploy it

Publish the harness as a pip-installable package or a GitHub template repo, and add a sample report to the README. Write a LinkedIn post showing "how chunk size changed faithfulness".

Put it on your resume

• Built a RAG evaluation harness (recall@k, MRR, LLM-judge faithfulness calibrated to human labels, κ = 0.71) with cached runs, regression reports and a CI quality gate.
• Used it to compare 6 retrieval configurations, improving faithfulness by +11 points while cutting cost per query by 35%.

Interview questions you'll get

• How do you evaluate an LLM app without ground truth? How do you trust an LLM judge?
• Which metrics catch retrieval problems vs generation problems?
• How do you prevent quality regressions when prompts or models change?

Practise these in Mock Interview → System Design & MLOps.

Stretch goals

• Synthetic question generation from documents (with human filtering).
• Pairwise A/B judging between two pipelines.
• Online evaluation from production traces.