GenAI Course · Module 9 · Session 31

RAG Foundations

⏱ 13 min read🎯 Beginner friendly🖼 3 diagrams🧪 1 live demo🐍 Python code

An LLM knows a lot about the world, but nothing about your company’s documents, and its knowledge stops at a training date. Retrieval-Augmented Generation (RAG) fixes this: first find the right information, then let the model answer from it. It is the most common GenAI pattern in companies today.

In this session

The two halves of RAG: indexing (offline) and answering (online)
Why RAG matters in production: fresh data, private data, citations, access control
RAG vs fine-tuning vs long-context prompting, with a decision helper
The NovaBank policy assistant we build through this module
A complete minimal RAG in LangChain

01The RAG workflow

RAG has two halves. The indexing half runs ahead of time; the answering half runs every time a user asks something.

Indexing (offline): prepare your documents once, then update when they change.

Answering (online): every question goes through these steps in about a second or two.

📚

Like an open-book exam. The student (LLM) is smart, but instead of answering from memory, they first open the right page of the textbook (retrieval) and then write the answer from it (generation).

02Why RAG is critical in production

Problem with a plain LLM	How RAG helps
Does not know private data (policies, tickets, contracts)	retrieves from your own documents
Knowledge has a cut-off date	re-index new documents any time, no retraining
Hallucinates confident answers	answers are grounded in retrieved text
No proof for the answer	every answer can cite its source page
Everyone sees everything	filter documents by the user’s access rights at retrieval time
Fine-tuning is costly for changing facts	updating an index is cheap and fast

03RAG vs fine-tuning vs long context

	RAG	Fine-tuning	Long-context prompting
Best for	facts that change; large document sets	style, format, domain behaviour	one-off questions on a few documents
Update cost	low: re-index	high: retrain	none: paste again
Citations	easy	hard	possible
Cost per question	low (only top chunks sent)	low	high (huge prompts every time)
Main risk	bad retrieval → bad answer	stale facts, forgetting	lost in the middle, latency

Demo · Which approach fits?

Rule of thumb: fine-tune for how the model answers, use RAG for what it knows.

04Reference architecture: NovaBank policy assistant

Through this module we build one realistic project: an internal assistant that answers employee questions from NovaBank’s policy documents, with citations and access control.

Each box is one session in this module.

05A complete minimal RAG in LangChain

About 20 lines from PDF to cited answer. Later sessions improve each step.

minimal_rag.py

# pip install langchain-community langchain-openai langchain-chroma pypdf
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate

docs = PyPDFLoader("fd_policy.pdf").load()                                   # 1. load
chunks = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100).split_documents(docs)   # 2. chunk
db = Chroma.from_documents(chunks, OpenAIEmbeddings(model="text-embedding-3-small"))              # 3-4. embed + store
retriever = db.as_retriever(search_kwargs={"k": 4})

prompt = ChatPromptTemplate.from_template(
    "Answer only from the context. Cite page numbers like [p.4]. "
    "If the answer is not in the context, say you don't know.\n\nContext:\n{context}\n\nQuestion: {question}")
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

question = "What is the penalty for breaking an FD early?"
found = retriever.invoke(question)                                            # retrieve
context = "\n\n".join(f"[p.{d.metadata['page'] + 1}] {d.page_content}" for d in found)
print(llm.invoke(prompt.format(context=context, question=question)).content)  # augment + generate

Breaking an FD early reduces the interest rate by 1% [p.4]. (sample output)

06Summary

RAG = retrieve the right chunks, then generate an answer from them.
Indexing runs offline; answering runs per question.
RAG gives fresh, private, cited answers with access control.
Fine-tune for behaviour; use RAG for knowledge; use long context for one-off documents.

07Quick quiz

1. What does the “R” in RAG do?

2. Your policies change every week. Best approach?

3. Which step happens offline?

4. A key production benefit of RAG is…

5. To teach the model your company’s writing style, you would mainly use…

08References

Retrieval-Augmented Generation for Knowledge-Intensive NLP (Lewis et al., 2020)
LangChain RAG tutorial

Created by Ajeetkumar