Word Embeddings: From Words to Vectors

The Miraculous Addition: King - Man + Woman ≈ Queen

Word2Vec revolutionized NLP with a simple arithmetic operation that seemed almost magical. If you took vectors representing "king," subtracted "man," added "woman," you got something resembling "queen." This wasn't programmed manually—the vectors learned this relationship from analyzing billions of words.

This phenomenon captures something profound: that language has mathematical structure. Words with similar meanings cluster nearby in vector space. Relationships between words—like gender or temporal patterns—become linear transformations. This insight transformed NLP from rule-based systems to deep learning.

From One-Hot to Dense Representations

Before word2vec, NLP used one-hot encoding. Each word gets a unique vector of length V (vocabulary size), with a single 1 and V-1 zeros:

cat = [0, 0, 1, 0, 0, 0, ...]

dog = [0, 0, 0, 0, 1, 0, ...]

Problems:

• Huge vectors: With 1 million words, vectors have 1 million dimensions. Computationally expensive.
• No semantic information: "cat" and "dog" are equally different from every other word. The vectors don't capture that they're both animals.
• No generalization: If you saw "cat" 1000 times but "feline" never, they have separate vectors with no relationship.

Dense word embeddings solve this. Instead of one-hot vectors of length V, use dense vectors of length d (often 100-300). Each dimension captures some aspect of meaning. "Ferocity," "size," "domestication" might be implicit dimensions. Similar words have similar values across dimensions.

Skip-Gram Architecture: Learning from Context

Word2Vec's Skip-Gram model is elegantly simple. The core idea: words appearing in similar contexts have similar meanings. If "cat" and "dog" both appear in sentences about pets, playing, feeding, then they're similar.

Architecture

Input: A target word (e.g., "cat")

Output: Predict context words (words within a window, e.g., "the," "sat," "on," "mat")

Network structure

Input layer: One-hot vector of target word (dimension V)

Hidden layer: Fully connected to embedding dimension d. This projection matrix W is our learned embeddings

Output layer: Softmax over V context words

Training

For each word in text, predict its context. Loss is cross-entropy on context words. Backpropagation updates W (embeddings) and output weights.

Computationally expensive: the softmax over V words (millions in large corpora) is prohibitive. Word2Vec introduced negative sampling to fix this.

Negative Sampling: Making Training Tractable

Softmax requires summing over all V words: p(context | target) = exp(score) / Σ_all exp(scores). With V = 1 million, this is slow.

Negative sampling reformulates as binary classification: given a target-context pair, is it real (from actual data) or fake (random word)?

For real pair (cat, mouse), probability = σ(v_mouse · v_cat) where σ is sigmoid and v are embeddings.

For fake pair (cat, random_word), probability = 1 - σ(v_random · v_cat).

Loss: -log σ(v_context · v_target) - Σ_{k=1}^K log σ(-v_negative · v_target)

Instead of computing softmax over millions of words, compute sigmoid on one positive and K negative examples. With K ≈ 5-20, this is fast.

Why this works

By learning to distinguish real from random contexts, the network learns useful embeddings. Real contexts have semantic relationships (cat appears with mouse). Random contexts don't. The embeddings capture what makes real contexts real.

Negative samples are drawn from unigram distribution raised to 3/4 power. Common words are undersampled (appearing as negatives less often than their frequency would suggest), focusing learning on rare words.

CBOW: Predicting the Target from Context

Continuous Bag of Words (CBOW) reverses Skip-Gram's direction:

Input: Context words

Output: Predict the target word

Architecture: Average context word embeddings, pass through hidden layer, softmax to target word.

Skip-Gram and CBOW learn related but distinct embeddings. Skip-Gram usually produces better embeddings for rare words (more training signal: each word is a target multiple times per context window). CBOW trains faster (averaging is faster than individual predictions).

The choice depends on use case: rare word quality (Skip-Gram) vs. training speed (CBOW).

Subword Information: Handling Unknown Words

Word2Vec treats each word independently. "Playing" and "play" have entirely separate embeddings. This is inefficient: they share obvious semantic connection.

FastText (extension to Word2Vec) represents words as sums of character n-grams. For "playing":

Each n-gram has its own embedding. "Playing" embedding = sum of these 7 embeddings. "Play" embedding shares the first 5 n-grams.

Benefits:

• Morphology: Similar morphological forms share n-grams, their embeddings are similar
• Unknown words: Even unseen words get embeddings (sum of their character n-grams). Word2Vec can't handle unknown words.
• Rare word improvement: Rare words benefit from shared n-gram representations with related words

GloVe: Global Vectors for Word Representation

Word2Vec learns embeddings by predicting local context. What if we also used global frequency information?

GloVe (Global Vectors) optimizes embeddings to reconstruct global co-occurrence statistics. Key insight: word relationships appear in co-occurrence patterns.

Co-occurrence matrix X: X_ij = how often words i and j appear together in a window. For a large corpus, X is dense with frequency information.

GloVe minimizes:

Loss = Σ_ij f(X_ij) (v_i · v_j + b_i + b_j - log X_ij)²

Where:

• v_i · v_j is the dot product (inner product) of word embeddings
• b_i, b_j are bias terms
• log X_ij is the log of the co-occurrence frequency
• f(X_ij) is a weighting function that gives less weight to very rare co-occurrences (often zero)

Intuition: For frequent co-occurrences, the embedding dot product should be large (similar meaning). For rare co-occurrences, the dot product should be small. The log makes this relationship linear, enabling SGD.

GloVe embeddings capture different relationships than Word2Vec. Word2Vec excels at syntactic relationships (king-queen are nearby because they have similar contexts). GloVe captures broader semantic relationships (via global statistics). In practice, both work well.

Mathematical Insight: Why Embeddings Work

Why does the king - man + woman ≈ queen relationship emerge?

Consider word co-occurrence patterns:

"King" appears in contexts like "throne," "rule," "royal."

"Man" appears in contexts like "he," "father," "boy."

"Queen" appears in contexts like "throne," "rule," "royal."

"Woman" appears in contexts like "she," "mother," "girl."

The word sets {throne, rule, royal} are similar for king and queen. The sets {he, father, boy} are specific to man, {she, mother, girl} to woman.

During training, embeddings capture these patterns. King and queen have similar values in dimensions capturing "royalty." Man and woman have similar values in dimensions capturing "humanness" but different values in dimensions capturing "gender."

So: king (royalty + male) - man (humanness + male) + woman (humanness + female) ≈ queen (royalty + female).

The gender transformation is learned implicitly from context statistics.

Bias in Word Embeddings: A Critical Issue

Word embeddings learn from human-generated text, which contains biases. These biases get encoded into embeddings:

Gender Bias Example: In many corpora, "nurse" co-occurs more with female pronouns, "doctor" with male. Embeddings learn this. When used in downstream tasks (resume screening, hiring recommendations), they perpetuate gender stereotypes.

Racial Bias: Names with ethnic associations cluster with stereotypical terms. "Latanya" might be near words associated with danger (due to biased source text), affecting bias in applications.

Socioeconomic Bias: Words associated with poverty might cluster with negative sentiment, regardless of context.

Debiasing techniques:

1. Identifying bias directions: Find linear subspaces capturing gender, race, etc. In gender, you can find a direction where man-woman, king-queen, etc. align.

2. Projecting out bias: Subtract bias components from embeddings. Remove gender direction from all embeddings.

3. Regularization during training: Add loss term penalizing bias directions.

4. Fairness-aware embedding methods: Train embeddings while explicitly constraining bias.

Debiasing is imperfect. Completely removing bias is impossible (and sometimes undesirable—some bias reflects reality). The goal is understanding and controlling bias, not eliminating it entirely.

Visualization and Interpretability

Embedding spaces are high-dimensional (usually 100-300 dimensions). How do we visualize them?

t-SNE (t-Distributed Stochastic Neighbor Embedding): Nonlinear dimensionality reduction. Projects high-dimensional data to 2D while preserving local structure. Words with similar embeddings appear close in t-SNE plots.

t-SNE reveals clustering: animal words cluster together, occupation words together, etc. This confirms embeddings capture semantic structure.

PCA (Principal Component Analysis): Linear projection to top 2 principal components. Less flexible than t-SNE but faster, enabling interactive exploration.

Visualizations reveal embeddings' semantic organization, validate training, and enable understanding what the model learned.

Indian Languages and Multilingual Embeddings

Most pre-trained embeddings (Word2Vec, GloVe) focus on English. Indian languages (Hindi, Tamil, Telugu, etc.) have limited resources. Solutions:

Monolingual embeddings: Train Word2Vec on Hindi/Tamil/Telugu text. Quality depends on available text volume. Wikipedia dumps, government documents, crawled web text provide some resources but less than English.

Multilingual embeddings: Train jointly on multiple languages using shared spaces. Aligned embeddings enable transfer learning: train a task in English, apply to Hindi with pre-trained embeddings.

Cross-lingual transfer: Shared embeddings enable zero-shot transfer. A sentiment classifier trained on English can directly apply to Tamil sentences if embeddings are aligned.

Challenges for Indian languages: limited pre-training data, diverse scripts, morphological complexity, code-mixing (mixing multiple languages in single sentences). Research continues addressing these challenges.

Contextual Embeddings: Beyond Static Vectors

Word2Vec and GloVe produce static embeddings: each word has one vector regardless of context. "Bank" has the same embedding in "river bank" and "bank account," yet means different things.

Contextual embeddings (ELMo, BERT, GPT) produce different representations depending on context:

"Bank" in "river bank" → one representation

"Bank" in "bank account" → different representation

These use bidirectional transformers processing the entire sentence, generating context-dependent representations. This enables capturing polysemy (multiple meanings) more effectively.

Static embeddings remain useful (faster, simpler) for some applications. Contextual embeddings dominate high-performance NLP systems.

Connection to Modern Language Models

Word2Vec was a crucial stepping stone. Modern large language models (GPT, BERT, Claude) build on embedding foundations:

1. Learned representations: Instead of one-hot vectors, all words are dense representations (learned during pre-training)

2. Context-aware: Transformer attention combines word embeddings with positional information, generating context-sensitive representations

3. Transfer learning: Pre-train on massive text corpora (like Word2Vec), fine-tune on specific tasks

The shift from discrete one-hot vectors to continuous embeddings, enabled by Word2Vec, remains fundamental. All modern NLP rests on this foundation.

Practical Implications and Lessons

Embedding quality matters: Using pre-trained embeddings (trained on billions of words) usually beats training from scratch on your data.

Dimensionality tradeoff: Higher dimensions capture more semantic nuance but require more training data and computation. 100-300 is typical.

Domain adaptation: Generic embeddings (trained on Wikipedia) might not suit domain-specific tasks (medical text, legal documents). Fine-tuning on in-domain data helps.

Interpretability: Unlike deep neural networks, embeddings are relatively interpretable. Examining nearest neighbors reveals semantic structure. This interpretability aids debugging and understanding model behavior.

Conclusion: Bridging Language and Geometry

Word2Vec revolutionized NLP by revealing that language has mathematical structure. Words embed in continuous vector spaces where semantics become geometry.

Skip-Gram and CBOW learn these embeddings from local context. Negative sampling makes training tractable. Extensions like FastText handle subword information. GloVe leverages global statistics. All share the insight: words are similar if they appear in similar contexts.

The arithmetic relationship (king - man + woman ≈ queen) exemplifies this structure. It isn't programmed—it emerges from analyzing billions of words. This emergence of structure from raw text is powerful and surprising.

Modern embeddings are more sophisticated (contextual, multilingual, bias-aware), but Word2Vec's core ideas remain relevant. Understanding these foundations—why embeddings work, their mathematical basis, their limitations—is essential for modern NLP.

The journey from discrete one-hot vectors to continuous embeddings to transformers shows how understanding language as geometry opens new possibilities for machines to understand and generate human language at remarkable levels of sophistication.