Word Embeddings

Counting words (Session 7) cannot see that “UPI” and “card” are both payment methods. Word embeddings fix this: each word becomes a short list of numbers, learned so that words used in similar contexts get similar numbers. This one idea is the foundation of everything from search to LLMs.

01Meaning from context

Linguist J. R. Firth wrote in 1957: “You shall know a word by the company it keeps.” If “UPI” and “card” keep appearing in the same kinds of sentences (“paid via ___”, “the ___ payment failed”), they probably mean similar things.

02Word2Vec: CBOW and skip-gram

Word2Vec (Google, 2013) trains a very small neural network on a fake task. The word vectors it learns along the way are what we actually want. It comes in two flavours:

We train both on 3,000 short sentences about payments, cities and food. Watch which words the model puts next to “upi”:

# pip install gensim
import random
from gensim.models import Word2Vec

random.seed(0)
methods = ["upi", "card", "netbanking", "wallet"]
cities = ["mumbai", "pune", "delhi", "chennai"]
foods = ["biryani", "dosa", "samosa", "idli"]
templates = ["customer paid via {m} today", "the {m} payment was successful", "refund sent to the {m} account",
             "she lives in {c} city", "flights to {c} are cheap", "the office in {c} is open",
             "he ordered {f} for lunch", "the {f} was hot and tasty", "we ate {f} at the stall"]
sentences = []
for _ in range(3000):
    t = random.choice(templates)
    sentences.append(t.format(m=random.choice(methods), c=random.choice(cities), f=random.choice(foods)).split())

for name, sg in [("CBOW", 0), ("Skip-gram", 1)]:
    model = Word2Vec(sentences, vector_size=30, window=3, min_count=1, sg=sg, epochs=20, seed=1, workers=1)
    print(f"{name}: most similar to 'upi' ->", [w for w, _ in model.wv.most_similar("upi", topn=3)])
    print(f"{name}: similarity(upi, card) = {model.wv.similarity('upi', 'card'):.2f}, "
          f"(upi, dosa) = {model.wv.similarity('upi', 'dosa'):.2f}")
print("vector size:", model.wv["upi"].shape)

Nobody told the model that UPI, card, netbanking and wallet are payment methods. It learned it purely from the sentences around them. “upi” and “dosa” never share a context, so their similarity is low.

03GloVe

GloVe (Stanford, 2014) reaches similar vectors a different way. It first counts, over the whole corpus, how often each pair of words appears near each other (a big co-occurrence table), then learns vectors whose dot products match those counts (in log form).

04Cosine similarity and semantic distance

To compare two embeddings we use cosine similarity: the cosine of the angle between the vectors. 1 = same direction (same meaning), 0 = unrelated, −1 = opposite.

import numpy as np

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

# toy 4-number "embeddings": [royalty, male, female, fruit]  (made up for illustration)
v = {"king":  np.array([0.9, 0.8, 0.1, 0.0]), "queen": np.array([0.9, 0.1, 0.8, 0.0]),
     "man":   np.array([0.1, 0.9, 0.1, 0.0]), "woman": np.array([0.1, 0.1, 0.9, 0.0]),
     "apple": np.array([0.0, 0.1, 0.1, 0.9])}

print(f"king vs queen: {cosine(v['king'], v['queen']):.2f}")
print(f"king vs apple: {cosine(v['king'], v['apple']):.2f}")

target = v["king"] - v["man"] + v["woman"]
best = max((w for w in v if w not in ("king", "man", "woman")), key=lambda w: cosine(target, v[w]))
print("king - man + woman ≈", best, f"(cosine {cosine(target, v[best]):.2f})")

Embeddings can even do “maths with meaning”: king − man + woman lands closest to queen. In real Word2Vec models this works for many relations, such as Paris − France + Italy ≈ Rome, though not perfectly.

05Summary

• Embeddings turn words into dense vectors where similar meaning = nearby vectors.
• Word2Vec: CBOW predicts a word from context; skip-gram predicts context from a word.
• GloVe learns vectors from global co-occurrence counts.
• Cosine similarity measures closeness: 1 same, 0 unrelated.
• Static embeddings give one vector per word; transformers make them context-aware.

06Quick quiz

07References

• Efficient Estimation of Word Representations (Word2Vec) (Mikolov et al., 2013)
• GloVe: Global Vectors for Word Representation (Pennington et al., 2014)
• Gensim Word2Vec tutorial
• The Illustrated Word2vec (Jay Alammar)