Module 2 cheat sheet: NLP Fundamentals
Turn messy text into clean tokens and numbers: tokenization, preprocessing, regex, counts and embeddings.
Key ideas
- Modern models use subword tokens: common words whole, rare words in pieces. S4 →
- English ≈ 4 characters per token; many other scripts need more tokens. S4 →
- Stemming chops endings; lemmatization finds the dictionary form. S5 →
- Removing stopwords can delete “not” and flip the meaning. S5 →
- Redact PII with regex before text reaches an LLM. S6 →
- Embeddings put words used in similar contexts close together. S8 →
Tokenizer algorithms
| Algorithm | Idea | Used by |
|---|---|---|
| BPE | merge the most frequent pair | GPT, Llama 3 |
| WordPiece | likelihood-based merges, ## pieces | BERT |
| Unigram | prune a big vocabulary | T5, ALBERT |
| SentencePiece | BPE/Unigram on raw text, ▁ for spaces | T5, Llama 2, Gemma |
Regex essentials
| Pattern | Meaning |
|---|---|
\d \w \s | digit · word character · space |
[A-Z]{5} | exactly 5 capital letters |
+ * ? | one or more · zero or more · optional |
\b | word boundary |
( ) and | | capture group · or |
re.findall · re.search · re.sub
From words to numbers
| Method | Captures | Limit |
|---|---|---|
| One-hot | which word | no similarity, huge |
| Bag of Words | word counts | order lost |
| N-grams | short phrases | many features |
| TF-IDF | important, rare words | still no meaning |
| Word2Vec / GloVe | meaning from context | one vector per word |
TF-IDF = TF × log(N / DF) · cosine = a·b / (|a||b|)
Code: TF-IDF in four lines
from sklearn.feature_extraction.text import TfidfVectorizer docs = ["UPI payment failed", "card payment failed again", "refund for failed UPI payment"] tfidf = TfidfVectorizer() X = tfidf.fit_transform(docs) # rows = documents, columns = words
Common mistakes
- Stemming or removing stopwords before sending text to an LLM.
- Estimating cost from words instead of tokens.
- Regex without
\b, causing partial matches. - Expecting Word2Vec to tell “river bank” from “bank account”.
Created by Ajeetkumar · EasyLearnSpace