Word Embeddings: From Words to Vectors

Search for "chappal" on a large Indian e-commerce app and the results include listings titled "sandals," "slippers," and "flip-flops." None of those product titles contain the string "chappal." A keyword-matching search engine — one that represents every word as a distinct, unrelated symbol — has no way to know these words belong together. It would have to be told, by hand, that "chappal" maps to "sandals," and then told the same for every other Hindi-English code-switch, regional synonym, and misspelling a few hundred million users might type. That approach does not scale. What does scale is a representation in which "chappal" and "sandals" are simply close together as points in space, learned automatically from how people actually use these words in millions of product titles, reviews, and search logs. That representation is a word embedding, and this chapter builds one from first principles: why the old representation (one-hot vectors) fails, how Word2Vec learns dense vectors by predicting context, how GloVe learns them from global co-occurrence counts instead, and what the resulting vector space actually encodes.

Why One-Hot Vectors Cannot Represent Meaning

Before embeddings, the standard way to feed a word into a machine-learning model was one-hot encoding. Fix a vocabulary of V distinct words, assign each one an index, and represent word i as a vector of length V that is 0 everywhere except a single 1 at position i. If "chappal" is word 4,021 and "sandals" is word 17,558 in a 50,000-word vocabulary, both vectors are 50,000-dimensional, each with exactly one nonzero entry, and those entries sit in completely different positions.

Now compute the cosine similarity between any two distinct one-hot vectors x and y. Cosine similarity is (x · y) / (‖x‖ ‖y‖). Since the 1s sit at different positions, the dot product x · y is exactly 0 for every pair of distinct words — synonyms, antonyms, or completely unrelated terms alike. "Chappal" and "sandals" get similarity 0. So do "chappal" and "satellite." The representation is mathematically incapable of expressing that two words are related, because it encodes nothing but an arbitrary index. It also wastes space: a 50,000-word vocabulary needs 50,000 dimensions to represent one word, almost all of them permanently zero.

What we want instead is a dense vector — typically 50 to 300 real-valued dimensions, far smaller than V — where the geometry of the space carries information: words used in similar ways end up as nearby points, and the directions between points encode relationships. Building that space is the whole problem this chapter solves.

The Distributional Hypothesis

The idea that makes this possible predates neural networks by decades. Linguist J. R. Firth wrote in 1957: "You shall know a word by the company it keeps." This is the distributional hypothesis: words that occur in similar contexts tend to have similar meanings. "Chappal" and "sandals" rarely appear next to each other, but they appear surrounded by the same kinds of neighboring words — "size," "leather," "comfortable," "pair," "₹," "buy." A model that learns to predict a word's neighbors, or that tracks which words co-occur with which, ends up placing "chappal" and "sandals" near each other purely as a side effect of them sharing a distributional signature — no bilingual dictionary required.

Two families of algorithms operationalize this hypothesis: Word2Vec (Mikolov et al., 2013, "Efficient Estimation of Word Representations in Vector Space," arXiv:1301.3781), which predicts context words from a local sliding window, and GloVe (Pennington, Socher & Manning, 2014, "GloVe: Global Vectors for Word Representation," EMNLP), which factorizes a global co-occurrence matrix built over the entire corpus. Both produce dense embeddings; they get there by different routes.

Skip-Gram: Learning Embeddings by Predicting Context

Word2Vec has two architectures, CBOW (predict the center word from its context) and skip-gram (predict the context words from the center word). Skip-gram is the one worth tracing in detail, because its mechanics are exactly what the diagram below depicts.

Take a cleaned, stopword-stripped token sequence — say the four-word toy corpus ice cold sun hot — and a window size w. For every position t, the word at that position is the center word, and every word within w positions on either side is a context word. With w = 1 and center word "ice" at position 0, the only in-window neighbor is "cold" at position 1 (there is no position −1), so training generates one input-output pair: (center = ice, context = cold). Scanning the whole corpus with this sliding window produces the full training set of (center, context) pairs.

The network that learns from these pairs is deliberately simple. There are two weight matrices: an input embedding matrix W of shape V × d (one row per vocabulary word, one column per embedding dimension) and an output embedding matrix W′ of the same shape. Given a one-hot center-word vector x, the hidden layer is h = Wᵀx — because x is one-hot, this multiplication just selects the row of W corresponding to the center word. No activation function is applied. The hidden vector h is fed to the output layer: a score u_j = W′_j · h is computed for every vocabulary word j, and a softmax turns the scores into a probability distribution p_j = exp(u_j) / Σ_k exp(u_k) over "which word is the context word." Training adjusts W and W′ by gradient descent so that this distribution assigns high probability to the words that actually appeared in the window.

Shallow, Not Deep: A Common Misconception

Students who have already met deep convolutional or transformer networks often assume Word2Vec must also be "deep" — after all, it is a neural network that produces impressively structured representations. It is not. Skip-gram has exactly one weight matrix between input and hidden layer (a lookup, not a transformation with an activation function) and one between hidden and output. There is no nonlinearity anywhere in the forward pass shown in the diagram above; mathematically, the whole thing is a log-linear model. This is not an oversight — it is the entire point. Earlier neural language models, notably Bengio, Ducharme, Vincent & Jauvin's 2003 neural probabilistic language model, did use a nonlinear hidden layer, and paid for it in training time that made scaling to web-sized corpora impractical. Mikolov et al.'s 2013 paper is literally titled "Efficient Estimation of Word Representations in Vector Space" — the efficiency comes specifically from stripping out the nonlinearity, so that training reduces to fast matrix lookups and a softmax that can itself be approximated cheaply (see negative sampling, below). Depth is not what makes these embeddings good; the enormous scale of the training corpus, funneled through a computationally cheap architecture, is.

Worked Example: One Skip-Gram Training Step, Fully Traced

Continue the toy example above: vocabulary {ice, cold, hot, sun} indexed 0–3, embedding dimension d = 2, training pair (center = ice, context = cold). Initialize the two weight matrices arbitrarily:

W  (input embeddings, rows = ice, cold, hot, sun):
[[ 0.10,  0.20],
 [-0.10,  0.30],
 [ 0.05, -0.25],
 [ 0.20,  0.05]]

W' (output embeddings, same row order):
[[ 0.15, -0.10],
 [ 0.25,  0.05],
 [-0.20,  0.10],
 [ 0.05,  0.30]]

Forward pass. The one-hot vector for "ice" selects row 0 of W, so the hidden vector is h = W[ice] = [0.10, 0.20] — exactly the lookup the diagram shows, no activation applied. Score every word against h using W′:

u_ice  = 0.15·0.10 + (-0.10)·0.20 = -0.005
u_cold = 0.25·0.10 +   0.05·0.20  =  0.035
u_hot  = -0.20·0.10 +  0.10·0.20  =  0.000
u_sun  = 0.05·0.10 +   0.30·0.20  =  0.065

Apply softmax, p_j = exp(u_j) / Σ exp(u_k). With Σ exp(u_k) ≈ 4.0978:

p_ice  = 0.2428
p_cold = 0.2527   ← target
p_hot  = 0.2440
p_sun  = 0.2604

The four probabilities are all close to 0.25 — unsurprising, since the weights started small and near-random, so the untrained network is close to a uniform guess over four words. The loss on this step is cross-entropy against the true label "cold": L = −log(p_cold) = −log(0.2527) ≈ 1.375 nats.

Backward pass. For softmax + cross-entropy, the gradient with respect to the score vector has the clean closed form ∂L/∂u_j = p_j − t_j, where t is the one-hot target ([0,1,0,0] for "cold"):

∂L/∂u = [0.2428, -0.7473, 0.2440, 0.2604]

The gradient with respect to each row of W′ is (∂L/∂u_j)·h; the gradient with respect to h (and hence row "ice" of W) is Σ_j (∂L/∂u_j)·W′_j. Running these through the arithmetic (verified numerically) gives, in particular:

∂L/∂W'_cold = [-0.0747, -0.1495]
∂L/∂h       = [-0.1862,  0.0409]

With learning rate η = 0.1, gradient descent updates W′_cold ← W′_cold − η·∂L/∂W′_cold and W[ice] ← W[ice] − η·∂L/∂h:

W'_cold_new = [0.25, 0.05] - 0.1·[-0.0747, -0.1495] = [0.2575, 0.0649]
W[ice]_new  = [0.10, 0.20] - 0.1·[-0.1862, 0.0409]  = [0.1186, 0.1959]

Both moves push in the same direction: "cold"'s output embedding grows to better match "ice"'s hidden vector, and "ice"'s input embedding shifts to better match the direction that scores "cold" highly. Repeat this over millions of (center, context) pairs, and words that repeatedly co-occur end up with embeddings that point the same way.

This entire forward/backward computation is exactly what the following code reproduces — no random initialization, so its printed output is deterministic and matches the arithmetic above line for line:

import numpy as np

W = np.array([[0.10, 0.20], [-0.10, 0.30],
              [0.05, -0.25], [0.20, 0.05]])
Wp = np.array([[0.15, -0.10], [0.25, 0.05],
               [-0.20, 0.10], [0.05, 0.30]])
center, target = 0, 1  # ice -> cold

h = W[center]
u = Wp @ h
p = np.exp(u) / np.exp(u).sum()
loss = -np.log(p[target])

t = np.zeros(4); t[target] = 1.0
dLdu = p - t
dLdWp = np.outer(dLdu, h)
dLdh = dLdu @ Wp

eta = 0.1
Wp_new = Wp - eta * dLdWp
W_ice_new = W[center] - eta * dLdh

print(np.round(p, 4))        # [0.2428 0.2527 0.244  0.2604]
print(round(loss, 4))        # 1.3754
print(np.round(Wp_new[1], 4))    # [0.2575 0.0649]
print(np.round(W_ice_new, 4))    # [0.1186 0.1959]

Negative Sampling: Making the Softmax Affordable

The softmax above summed over a 4-word vocabulary. Real corpora have vocabularies in the hundreds of thousands to millions — Word2Vec's own published Google News vectors cover roughly 3 million words and phrases at 300 dimensions. Computing the full softmax denominator means one multiply-add per dimension per vocabulary word, so a single training step costs on the order of V·d operations. At V = 3,000,000 and d = 300, that is roughly 9×10⁸ operations — for one (center, context) pair, repeated billions of times over the corpus. This is computationally prohibitive, and it is the second half of why the original 2003-style neural language models did not scale.

Mikolov et al.'s follow-up paper (2013, "Distributed Representations of Words and Phrases and their Compositionality," NeurIPS) replaces the exact softmax with negative sampling: instead of scoring every word in the vocabulary, score the true context word plus a small number k of randomly sampled "negative" words (drawn from a noise distribution that is the unigram frequency raised to the 0.75 power, which the paper found outperformed sampling by raw frequency or uniformly), and turn the objective into k+1 independent binary classifications ("is this pair a real context pair or not?") using the logistic sigmoid σ:

log σ(w'_context · w_center) + Σ_{i=1}^{k} E[log σ(-w'_i · w_center)]

The cost per step drops to roughly (k+1)·d. With k = 10 and d = 300, that is 3,300 operations instead of 9×10⁸ — a speed-up of about 900,000,000 / 3,300 ≈ 273,000×. This is the concrete complexity trade that makes training embeddings on web-scale text tractable: negative sampling turns an O(V)-per-step algorithm into an O(k)-per-step one, at the cost of an approximate rather than exact objective.

GloVe: Learning from Global Co-occurrence Counts Instead

Skip-gram is a local, predictive method: it never looks at the corpus as a whole, only at one sliding window at a time. GloVe takes the opposite route: it is global and count-based. First, scan the entire corpus once to build a word-word co-occurrence matrix X, where X_ij counts how often word j appears in word i's context window, summed over the whole corpus. Then find vectors w_i, w̃_j and scalar biases b_i, b̃_j that minimize the weighted least-squares objective

J = Σ_{i,j} f(X_ij) · (w_i · w̃_j + b_i + b̃_j − log X_ij)²

where f(x) = (x / x_max)^α for x < x_max and 1 otherwise (Pennington et al. used x_max = 100, α = 0.75 in their experiments). This weighting matters: without it, extremely frequent pairs like ("the", "of") would dominate the loss and drown out the informative, rarer co-occurrences that actually distinguish meanings.

The intuition for why ratios of co-occurrence probabilities (which the log form of the objective is built around) carry more signal than raw counts is worth deriving on toy numbers. Suppose a corpus yields these (invented, illustrative) co-occurrence counts for two words, "ice" and "steam," against three probe words:

"Solid" and "gas" have ratios far from 1 in opposite directions — exactly what you would expect, since solid relates strongly to ice and gas relates strongly to steam. "Water" relates to both roughly equally, so its ratio sits near 1 and it contributes little discriminating signal. GloVe's objective is built directly around fitting log X_ij, so these ratios (differences of logs) become the quantity the learned vectors are shaped to reproduce, which is precisely the behavior that separates a genuinely discriminating context word from a merely frequent one.

What the Vector Space Encodes: Linear Structure

Both methods converge on the same striking empirical property: simple vector arithmetic on the learned embeddings tracks semantic relationships. The canonical example is king − man + woman ≈ queen. Using illustrative embedding values (not literal trained output, but representative of the geometry real embeddings exhibit):

king  = [0.90, 0.80, 0.10]
man   = [0.80, 0.10, 0.05]
woman = [0.85, 0.15, 0.90]
queen = [0.88, 0.90, 0.80]

king - man + woman = [0.95, 0.85, 0.95]
cosine(result, queen)   = 0.9957   ← highest
cosine(result, woman)   = 0.9029
cosine(result, king)    = 0.8485
cosine(result, man)     = 0.6950

The arithmetic result is closer to "queen" than to any other word in this set, including the two words that were literally added into it. A word with no semantic connection to any of the four — "cricket," say — would score lower still on this same cosine measure, precisely because it shares none of the distributional context that ties "king," "man," "woman," and "queen" together; no specific value is given here since no vector for such a word has been defined. The reading is that "man → woman" and "king → queen" share an approximately parallel offset direction in the embedding space — a direction that correlates with the gender distinction learned purely from how these words are distributed across the training corpus. This same mechanism is a double-edged property: because the direction is learned from raw text statistics rather than from any notion of fairness, embeddings trained on real corpora reproduce whatever associations are present in that text, including social biases (Bolukbasi et al., 2016, documented this concretely for occupation words). The vector space encodes co-occurrence statistics, not ground truth about the world — a distinction worth keeping in mind every time an embedding-based system is deployed on human-generated text.

Active Recall

Attempt each question before reading its answer.

1. A search engine represents "chappal" and "sandals" as one-hot vectors over a 50,000-word vocabulary. What is their cosine similarity, and why does that number not mean the words are unrelated?
2. True or false: skip-gram's hidden layer applies a nonlinear activation function, making it a deep network. Justify your answer.
3. In the toy corpus ice(0) cold(1) sun(2) hot(3), window = 1 around center word "ice" generates the single pair (ice, cold). If the window size is increased to 2, what new training pair (or pairs) involving center word "ice" get added — and which word stays excluded even under window = 2?
4. Using the same forward pass as the worked example (so ∂L/∂W′_cold = [-0.0747, -0.1495] and ∂L/∂h = [-0.1862, 0.0409] are unchanged), recompute the updated W′_cold and W[ice] if the learning rate is η = 0.01 instead of η = 0.1. How does the size of the update compare to the η = 0.1 case?
5. A full softmax over a 3,000,000-word vocabulary with 300-dimensional vectors costs about 9×10⁸ operations per training step. Negative sampling with k = 10 costs about (k+1)·d operations. Compute the speed-up factor.
6. Given toy co-occurrence counts X(ice, solid) = 50, X(steam, solid) = 5, X(ice, gas) = 5, X(steam, gas) = 50, compute the ratio X(ice, w) / X(steam, w) for each probe word and explain what a ratio close to 1 versus far from 1 tells you about that probe word's usefulness for distinguishing "ice" from "steam."

Answers

1. The cosine similarity is exactly 0. Any two distinct one-hot vectors have their single 1s at different positions, so their dot product is 0, and cosine similarity — dot product divided by the product of norms — is 0 / 1 = 0 regardless of whether the two words are synonyms or completely unrelated. The 0 is a property of the encoding scheme (an arbitrary index with no shared structure), not a measurement of the words' actual relatedness. Fixing this requires a representation, like a Word2Vec or GloVe embedding, whose geometry is shaped by how the words are actually used in text.

2. False. The hidden layer in skip-gram is a pure lookup — because the input is one-hot, h = Wᵀx simply selects one row of W, with no activation function applied anywhere in the forward pass. Skip-gram is a shallow, effectively log-linear model: one linear projection layer, one linear scoring layer, then softmax. Depth was deliberately avoided; Mikolov et al. (2013) removed the nonlinear hidden layer used by earlier neural language models specifically to make training fast enough to scale to enormous corpora.

3. Window = 2 covers positions within distance 2 of the center word, i.e. positions 1 and 2 relative to "ice" at position 0 — "cold" (distance 1, already included) and "sun" (distance 2, newly included). So the new pair is (ice, sun). "Hot," at position 3 (distance 3), remains excluded even under window = 2, since 3 > 2.

4. W′_cold_new = [0.25, 0.05] − 0.01·[-0.0747, -0.1495] = [0.2507, 0.0515]. W[ice]_new = [0.10, 0.20] − 0.01·[-0.1862, 0.0409] = [0.1019, 0.1996]. Both updates move in exactly the same direction as the η = 0.1 case, but the size of each step (Δ = η · gradient) is exactly one-tenth as large, since the gradient itself does not depend on η — only the step size scales with the learning rate.

5. 900,000,000 / 3,300 ≈ 272,727 — roughly a 273,000× reduction in per-step cost.

6. Ratio for "solid" = 50 / 5 = 10; ratio for "gas" = 5 / 50 = 0.1. Both ratios sit far from 1, in opposite directions, which is exactly what makes a probe word useful: "solid" co-occurs with "ice" much more than with "steam," and "gas" co-occurs with "steam" much more than with "ice," so both ratios carry a strong, unambiguous signal about which word each probe is associated with. A probe word whose ratio sits near 1 (co-occurring with both words at roughly the same rate, like "water" would with both "ice" and "steam") contributes little discriminating information — the ratio close to 1 says "this context word doesn't help tell the two apart," which is precisely why GloVe's objective, built around fitting log X_ij, is shaped by exactly these ratios rather than by raw co-occurrence counts.

Think About It

Think about this: How would you explain word embeddings: from words to vectors to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where word embeddings: from words to vectors is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting word embeddings: from words to vectors to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind word embeddings: from words to vectors, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.