Count-Based Feature Representation

Machine learning models need numbers, not words. The first ideas for turning text into numbers simply count words. They are old, but still power search engines, spam filters and quick baselines today, and they explain why embeddings (Session 8) were such a big step.

01One-hot encoding and its limitations

Give every word in the vocabulary a position. A word becomes a vector of zeros with a single 1 at its position:

import numpy as np
vocab = ["upi", "payment", "failed", "refund", "card", "blocked"]
idx = {w: i for i, w in enumerate(vocab)}

def one_hot(word):
    v = np.zeros(len(vocab), dtype=int); v[idx[word]] = 1; return v

for w in ["payment", "refund"]:
    print(f"{w:8s}", one_hot(w))
print("similarity(payment, refund) =", one_hot("payment") @ one_hot("refund"))

• No meaning: “payment” and “refund” are as unrelated as “payment” and “banana”.
• Huge and sparse: a 50,000-word vocabulary means 50,000 numbers per word, almost all zeros.
• Unknown words: a new word has no position at all.

02Bag of Words (BoW)

Bag of Words represents a whole document by counting how often each vocabulary word appears. Word order is thrown away, like emptying the words into a bag.

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

docs = ["UPI payment failed",
        "UPI payment successful",
        "card payment failed again",
        "refund for failed UPI payment"]

bow = CountVectorizer()
X = bow.fit_transform(docs)
print("vocabulary:", list(bow.get_feature_names_out()))
print("BoW matrix:\n", X.toarray())

bi = CountVectorizer(ngram_range=(1, 2))
bi.fit(docs)
print("\nwith bigrams:", len(bi.get_feature_names_out()), "features, e.g.",
      [f for f in bi.get_feature_names_out() if " " in f][:5])

tfidf = TfidfVectorizer()
T = tfidf.fit_transform(docs)
names = tfidf.get_feature_names_out()
print("\nTF-IDF weights for doc 4:", docs[3])
row = T.toarray()[3]
for i in row.argsort()[::-1]:
    if row[i] > 0: print(f"   {names[i]:10s} {row[i]:.2f}")

Each row of the BoW matrix is one document; each column is a word from the vocabulary. Row 1 (“UPI payment failed”) has 1s under failed, payment and upi.

03N-grams

An n-gram is a run of n neighbouring words. Adding bigrams (n = 2) keeps a little order: “payment failed” and “failed again” become features of their own.

Bigrams of “UPI payment failed”: UPI payment, payment failed.

04TF-IDF

Counting treats every word equally, but a word that appears in every document (like “payment” here) tells us little. TF-IDF multiplies how often a word appears in this document (term frequency) by how rare it is across all documents (inverse document frequency):

In the output above, payment appears in all four documents, so it gets the lowest weight (0.30), while refund appears only once and gets the highest (0.57). Notice for also scores high only because it is rare in this tiny corpus: that is why stopwords are often removed before TF-IDF.

Real weights from the same TF-IDF model as the code above.

05Summary

• One-hot: one 1 per word; no notion of similarity; huge and sparse.
• Bag of Words: count words per document; order is lost.
• N-grams keep short phrases such as “not good”, at the cost of many more features.
• TF-IDF boosts words that are frequent in a document but rare overall.
• Count features are fast baselines and still power keyword search (BM25).

06Quick quiz

07References

• scikit-learn: Text feature extraction
• Stanford IR book: tf-idf weighting