Document Loaders
Before an LLM can answer from your data, the data must come into Python in one standard shape. A document loader reads a source (PDF, website, CSV, folder…) and returns Document objects: the text plus metadata such as the file name and page number. This is step 1 of every RAG app, and the metadata is what lets you show citations later.
In this session
- What a Document is: page_content + metadata
- TextLoader, CSVLoader, JSONLoader
- PyPDFLoader: one Document per page
- WebBaseLoader and WikipediaLoader
- DirectoryLoader for whole folders
- lazy_load for very big data
01The big idea: every file becomes the same envelope
Think of India Post. Letters, parcels and documents all go into a standard envelope with the content inside and the address written outside. A LangChain Document is that envelope: page_content is what is inside, metadata is the address label (source, page, row). After loading, splitters, embeddings and vector stores only need to understand one envelope.
02Popular loaders
| Source | Loader | Install |
|---|---|---|
| .txt | TextLoader | langchain-community |
PyPDFLoader, PyMuPDFLoader | pypdf / pymupdf | |
| .csv | CSVLoader | langchain-community |
| .json | JSONLoader(jq_schema=".[].answer") | jq |
| .docx | Docx2txtLoader | docx2txt |
| website | WebBaseLoader, RecursiveUrlLoader | beautifulsoup4 |
| Wikipedia | WikipediaLoader | wikipedia |
| YouTube | YoutubeLoader | youtube-transcript-api |
| folder | DirectoryLoader(glob="*.txt") | langchain-community |
| any file | UnstructuredFileLoader | unstructured |
langchain_community now prints a warning that the package is being sunset; many integrations are moving to their own packages (for example langchain-chroma, langchain-groq). The loaders in this lab still work. When a dedicated package exists, prefer it.03Hands-on: run it here
Same files as the notebook, plus a sample PDF so you can see PyPDFLoader without downloading. Download novabank_handbook.pdf (3 pages, fictional bank) and use it in Colab too: it is the PDF for Labs 7–11.
import json, os, warnings; warnings.filterwarnings("ignore")
from langchain_core.documents import Document
from langchain_community.document_loaders import TextLoader, CSVLoader, JSONLoader, PyPDFLoader, DirectoryLoader
doc = Document(page_content="LangChain helps build LLM apps.", metadata={"source": "my_notes", "author": "Ajeet"})
print("Document :", doc.page_content, "|", doc.metadata)
open("bank_policy.txt", "w").write("NovaBank Card Policy\nReport a lost card within 3 days for zero liability.\nCard-not-present transactions above Rs 5,000 need OTP.")
open("students.csv", "w").write("name,course,city\nAjeet,GenAI,Pune\nPriya,Data Science,Mumbai\nRahul,Python,Nagpur")
json.dump([{"id": 1, "question": "What is AI?", "answer": "Machines that think."},
{"id": 2, "question": "What is ML?", "answer": "Learning from data."}], open("faq.json", "w"))
d = TextLoader("bank_policy.txt").load()
print("\nTextLoader:", len(d), "doc,", len(d[0].page_content), "chars, metadata", d[0].metadata)
print("CSVLoader :")
for d in CSVLoader("students.csv").load(): print(" ", d.page_content.replace("\n", " | "), d.metadata)
print("JSONLoader:", [d.page_content for d in JSONLoader("faq.json", jq_schema=".[].answer", text_content=False).load()])
pages = PyPDFLoader("novabank_handbook.pdf").load() # one Document per page
print(f"\nPyPDFLoader: {len(pages)} pages")
for p in pages: print(f" page {p.metadata['page']}: {p.page_content.splitlines()[0]}")
os.makedirs("kb", exist_ok=True)
open("kb/upi.txt", "w").write("UPI limit for new payee is Rs 25,000 in first 24 hours.")
open("kb/kyc.txt", "w").write("Video KYC is available 9 AM to 9 PM.")
print("\nDirectoryLoader:")
for d in sorted(DirectoryLoader("kb", glob="*.txt", loader_cls=TextLoader).load(), key=lambda d: d.metadata["source"]):
print(" ", d.metadata["source"], "->", d.page_content)
print("\nlazy_load:", [len(p.page_content) for p in PyPDFLoader("novabank_handbook.pdf").lazy_load()], "chars per page, one at a time")04Hands-on: Ajeetkumar’s Colab notebook
groq_key with your free key from console.groq.com, switch on Notebook access, then run the cells from the top.The notebook downloads the “Attention Is All You Need” paper with wget and loads a Wikipedia page, so those cells need Colab’s internet.
05Practice tasks
- Load your own resume PDF and ask the LLM for 5 interview questions.
- Load 3 Wikipedia pages on Indian cities and print their titles.
- Load a website and count the words.
Try these in the same Colab notebook after you finish the lab. Change one thing at a time and look at the output again.
06Summary
- A
Document=page_content+metadata. - Loaders exist for text, CSV, JSON, PDF, web, Wikipedia, YouTube and folders.
- PDF loaders give one Document per page with
pagemetadata for citations. lazy_load()streams documents one by one for big data.- Loading is step 1 of RAG: next come splitting, embedding and storing.
07Quick quiz
1. What are the two parts of a Document?
2. PyPDFLoader on a 10-page PDF returns…
3. CSVLoader creates one Document per…
4. Why keep metadata?
5. Which loads one document at a time to save memory?
08References
- LangChain: Document loaders
- Sample PDF: NovaBank handbook (fictional) (download)
- Ajeetkumar’s notebook: LC_06_Document_Loaders.ipynb (download)