Text Preprocessing

Real text is messy: “Paid”, “paying” and “payments” are the same idea written three ways. Classic NLP cleans and normalises text before feeding it to a model. In this session you will run stemming, lemmatization, stopword removal, POS tagging and NER on real sentences, and learn when not to preprocess.

01Lemmatization vs stemming

Both reduce a word to a base form so that “studies” and “study” count as the same word.

from nltk.stem import PorterStemmer, WordNetLemmatizer
from nltk.corpus import stopwords
# once: nltk.download("wordnet"); nltk.download("stopwords")

words = ["running", "studies", "better", "paid", "payments", "flies"]
stem, lem = PorterStemmer(), WordNetLemmatizer()
print(f"{'word':10s}{'stem':10s}{'lemma (verb)':14s}lemma (noun)")
for w in words:
    print(f"{w:10s}{stem.stem(w):10s}{lem.lemmatize(w, pos='v'):14s}{lem.lemmatize(w, pos='n')}")
print("better as adjective ->", lem.lemmatize("better", pos="a"))

stop = set(stopwords.words("english"))
sentence = "I am not happy with the delay in my refund".split()
print("\nstopwords removed:", [w for w in sentence if w.lower() not in stop])
print("'not' is a stopword?", "not" in stop)

• The stemmer turns “studies” into studi and “flies” into fli: not real words, but consistent.
• The lemmatizer needs to know the part of speech: “paid” → pay as a verb; “better” → good as an adjective.

02Stopword removal

Stopwords are very common words such as “the”, “is”, “in”, “a”. Removing them shrinks the data for keyword search and TF-IDF (Session 7). The output above shows the danger:

03POS tagging and Named Entity Recognition

Part-of-speech (POS) tagging labels each word: noun, verb, adjective and so on. Named Entity Recognition (NER) finds real-world things: people, organisations, places, money, dates. spaCy does both with a small pre-trained model:

# pip install spacy && python -m spacy download en_core_web_sm
import spacy
nlp = spacy.load("en_core_web_sm")

doc = nlp("Priya Sharma transferred ₹25,000 to HDFC Bank in Mumbai on 12 March 2025.")
print("POS tags:")
print("  ", [(t.text, t.pos_) for t in doc][:8])
print("Entities:")
for ent in doc.ents:
    print(f"   {ent.text:16s} {ent.label_:8s} {spacy.explain(ent.label_)}")

NER is the first step in many banking workflows: pull the amount, payee and date out of an email, then fill a form automatically. Note that the small model tagged “₹” as a proper noun; bigger models, or a rule, would fix that.

04Preprocessing in the LLM era

05Summary

• Stemming cuts endings by rule (fast, rough); lemmatization finds the dictionary form (slower, accurate).
• Removing stopwords helps keyword methods but can delete “not” and flip meaning.
• POS tagging labels word types; NER finds people, organisations, places, money and dates.
• For LLMs, skip normalisation but still clean text and remove PII.

06Quick quiz

07References

• NLTK book, Chapter 3: Processing raw text
• spaCy 101
• spaCy: Named entities