A search bar that has to understand, not match
Type "sasta mobile" into an e-commerce search bar and it should surface budget smartphones — even though not one product listing contains the Hindi word "sasta." A keyword-matching engine, the kind built on an inverted index of exact tokens, fails here by construction: "sasta" and "budget" share zero characters, so a Boolean AND/OR query over tokens returns nothing useful. Widening to fuzzy string matching does not save you either, since "sasta" and "cheap" aren't spelling variants of each other — they're different words in different languages that happen to mean almost the same thing in this context. What the search engine needs is a representation of "sasta mobile" that sits geometrically close to "budget smartphone under 15000" and geometrically far from "expensive flagship phone," even though none of these phrases share a single token. That representation is a dense embedding, and the geometry that makes closeness meaningful is cosine similarity in a learned vector space.
Concretely: suppose a query and three candidate listings have been mapped, by some embedding model, into a (deliberately tiny, illustrative) 3-dimensional space —
query "sasta mobile" = [0.80, 0.10, 0.60]
doc A "budget smartphone under 15000" = [0.75, 0.15, 0.55]
doc B "sasta chappal" (cheap sandals) = [0.70, 0.65, 0.10]
doc C "expensive flagship phone" = [0.20, 0.10, 0.90]Cosine similarity between vectors u and v is (u·v) / (‖u‖ ‖v‖) — the cosine of the angle between them, ranging from −1 (opposite) to 1 (identical direction), insensitive to vector length so that a long product description and a short query can still be compared fairly. Computing it for all three candidates gives cos(query, A) = 0.9981, cos(query, B) = 0.7097, cos(query, C) = 0.7618. Doc A wins decisively, despite sharing no tokens with the query at all. Doc B, which literally contains the query's own word "sasta," ranks last — because the embedding space has learned that "cheap footwear" and "cheap phone" are semantically distant even when a surface token overlaps. This is the entire point of dense representations: they encode meaning as geometry, so that "closeness in meaning" becomes "closeness in space," computable with a dot product instead of a rule engine.
From one-hot vectors to the distributional hypothesis
Every NLP system needs to turn words into numbers before any arithmetic can happen. The naive scheme is one-hot encoding: for a vocabulary of size V, word i becomes a length-V vector of zeros with a single 1 at position i. This is exactly the input representation used at the far left of the diagram below. One-hot vectors have two fatal properties. First, dimensionality equals vocabulary size — a realistic vocabulary of 50,000–100,000 tokens means every word is a 50,000-dimensional vector that is 99.998% zeros, wasteful to store and multiply. Second, and more damaging, every pair of distinct one-hot vectors is orthogonal: for any two different words i ≠ j, their dot product is exactly 0, because the 1 never lands on the same coordinate. "Cricket" and "IPL" are exactly as similar as "cricket" and "samosa" under this encoding — zero. There is no way to recover the fact that cricket and IPL are related from the one-hot vectors themselves; all the relevant structure has been thrown away by the encoding.
The fix has a name in linguistics older than any neural network: the distributional hypothesis, traced to Zellig Harris's 1954 paper "Distributional Structure," and crystallised by J.R. Firth in 1957 into the line every NLP course eventually quotes — "You shall know a word by the company it keeps." The claim is that a word's meaning is largely determined by the contexts it tends to appear in. "Cricket" co-occurs with "stadium," "over," "wicket," and "IPL" far more often than with "samosa" or "train." If you build a numeric representation whose training objective is literally "be good at predicting a word's neighbours" (or equivalently, "be predictable from context"), words that share contexts will be pulled toward similar vectors purely as a side effect of the objective. That is the mechanism embedding models exploit, and the rest of this chapter is different ways of turning that one idea into trainable arithmetic.
word2vec: skip-gram with negative sampling
Mikolov, Chen, Corrado, and Dean introduced two shallow architectures for this in "Efficient Estimation of Word Representations in Vector Space" (2013): continuous bag-of-words (CBOW), which predicts a center word from its surrounding context, and skip-gram, which predicts each context word from the center word. A companion paper by Mikolov, Sutskever, Chen, Corrado, and Dean, "Distributed Representations of Words and Phrases and their Compositionality" (NeurIPS 2013), made skip-gram trainable at web scale by replacing an expensive softmax over the full vocabulary with negative sampling, and it is this version — skip-gram with negative sampling (SGNS) — that is worth deriving by hand, because every later contrastive embedding method (including the sentence-embedding techniques later in this chapter) is a direct descendant of its loss function.
The model keeps two embedding matrices, both of shape V × d: W_in, which gives each word its vector when it plays the role of center word, and W_out, which gives each word its vector when it plays the role of context word. For a center word c and an observed context word o (one that actually appeared inside the sliding window around c in the training corpus), skip-gram wants the dot product v_c · u_o to be large — meaning the two vectors point in a similar direction. To prevent the trivial solution "make every vector enormous," negative sampling also draws k random "noise" words from the vocabulary (words that did not appear in that context) and pushes their dot products with v_c down. The per-step loss, to be minimized, is
L(v_c, u_o, u_k) = -log σ(v_c · u_o) - Σ_k log σ(-v_c · u_k)where σ(z) = 1 / (1 + e^-z) is the logistic sigmoid. The first term is small (good) exactly when σ(v_c · u_o) is close to 1, i.e. when the true context word's dot product is large and positive. The second term is small when σ(-v_c · u_k) is close to 1, i.e. when the negative word's dot product is large and negative. One gradient step nudges v_c toward every true context vector and away from every sampled noise vector — a purely local, cheap-to-compute alternative to softmax over 50,000 classes.
Worked example: one full SGNS training step
Take a toy 5-word vocabulary — ipl, cricket, stadium, train, samosa — with embedding dimension d = 2 so every step can be done by hand. Suppose training has progressed partway and the two matrices currently hold:
W_in (center vectors) W_out (context vectors)
ipl = [ 0.10, 0.30] ipl = [ 0.20, 0.10]
cricket = [ 0.50, -0.20] cricket = [-0.10, 0.30]
stadium = [-0.10, 0.40] stadium = [ 0.40, -0.10]
train = [ 0.05, 0.05] train = [ 0.00, 0.20]
samosa = [-0.30, 0.10] samosa = [ 0.15, -0.25]The center word in this window is "cricket," so v_c = [0.50, -0.20]. The corpus places "stadium" inside the window, so it is the positive context: u_o = [0.40, -0.10]. One negative sample is drawn — "samosa": u_k = [0.15, -0.25].
Step 1 — dot products. v_c · u_o = 0.50(0.40) + (-0.20)(-0.10) = 0.20 + 0.02 = 0.22. v_c · u_k = 0.50(0.15) + (-0.20)(-0.25) = 0.075 + 0.05 = 0.125.
Step 2 — sigmoids. σ(0.22) = 1/(1+e^-0.22) = 0.5548. For the negative term we need σ(-v_c·u_k) = σ(-0.125) = 1/(1+e^0.125) = 0.4688. Note this also implies σ(0.125) = 0.5312, a quantity needed shortly for the gradient.
Step 3 — loss. L = -log(0.5548) - log(0.4688) = 0.5892 + 0.7576 = 1.3468 nats. Both terms are far from zero, telling us the model currently distinguishes this positive pair from this negative pair only weakly — exactly what you'd expect this early in training, since 0.22 and 0.125 are close together rather than one being clearly high and the other clearly low.
Step 4 — gradients. Differentiating the loss with respect to each vector (a short exercise in the chain rule through the sigmoid, whose derivative is σ'(z) = σ(z)(1-σ(z))) gives three clean update rules:
∂L/∂v_c = (σ(v_c·u_o) - 1)·u_o + σ(v_c·u_k)·u_k
∂L/∂u_o = (σ(v_c·u_o) - 1)·v_c
∂L/∂u_k = σ(v_c·u_k)·v_cPlugging in numbers: ∂L/∂v_c = (0.5548-1)[0.40,-0.10] + 0.5312[0.15,-0.25] = [-0.1781, 0.0445] + [0.0797,-0.1328] = [-0.0984, -0.0883].
Step 5 — update, η = 0.1. Gradient descent moves every vector opposite its gradient: v_c ← v_c - η·∂L/∂v_c = [0.50,-0.20] - 0.1[-0.0984,-0.0883] = [0.5098, -0.1912]. The same rule applied to the other two gradients gives u_o ← [0.4223, -0.1089] and u_k ← [0.1234, -0.2394]. All three vectors moved in the direction that makes the true pair's dot product larger and the noise pair's dot product smaller — you can verify this by recomputing v_c·u_o with the updated vectors and checking it exceeds 0.22. Over millions of such steps across a real corpus, this local nudging is the entire mechanism by which "cricket" ends up near "IPL" and far from "samosa" in the final embedding space — no word ever gets a hand-labeled similarity score; the geometry is a byproduct of getting very good at next-word-in-context prediction.
The following code reproduces steps 1–3 exactly, so you can check the arithmetic yourself:
import math
W_in = {
"ipl": [0.10, 0.30],
"cricket": [0.50, -0.20],
"stadium": [-0.10, 0.40],
"train": [0.05, 0.05],
"samosa": [-0.30, 0.10],
}
W_out = {
"ipl": [0.20, 0.10],
"cricket": [-0.10, 0.30],
"stadium": [0.40, -0.10],
"train": [0.00, 0.20],
"samosa": [0.15, -0.25],
}
def dot(a, b):
return a[0]*b[0] + a[1]*b[1]
def sigmoid(z):
return 1 / (1 + math.exp(-z))
v_c = W_in["cricket"] # center word
u_o = W_out["stadium"] # true context word (positive)
u_k = W_out["samosa"] # sampled noise word (negative)
x = dot(v_c, u_o)
y = dot(v_c, u_k)
loss = -math.log(sigmoid(x)) - math.log(sigmoid(-y))
print(round(x, 4), round(y, 4)) # 0.22 0.125
print(round(sigmoid(x), 4), round(sigmoid(-y), 4)) # 0.5548 0.4688
print(round(loss, 4)) # 1.3468Running this prints exactly 0.22 0.125, then 0.5548 0.4688, then 1.3468 — matching every number derived by hand above.
One consequence of this objective is worth naming because it produces word2vec's most famous demo: since the loss only ever cares about relative geometry (dot products, not absolute positions), the vector space ends up encoding relationships as consistent offsets. The offset from "man" to "woman" turns out to be approximately the same direction and magnitude as the offset from "king" to "queen," so vec(king) - vec(man) + vec(woman) ≈ vec(queen) — reported in the original Mikolov et al. (2013) papers as an emergent property of the training objective, not something explicitly engineered in.
GloVe: embeddings from global co-occurrence counts
Skip-gram is a local, predictive method — it only ever looks at one small context window at a time and never directly sees corpus-wide statistics. Pennington, Socher, and Manning's GloVe ("Global Vectors for Word Representation," EMNLP 2014) takes the opposite route: first build one large word-by-word co-occurrence count matrix X across the entire corpus, where X_ij counts how often word j appears in word i's context window, summed over the whole training set. Then fit embeddings so that dot products approximate log co-occurrence counts, via a weighted least-squares objective:
J = Σ_ij f(X_ij) · (v_i·u_j + b_i + b_j - log X_ij)²where f is a weighting function that down-weights extremely frequent pairs (like "the" with almost anything) so they don't dominate the loss. GloVe and skip-gram look architecturally different — one is a regression over a static count matrix, the other a predictive neural net trained with stochastic gradient steps like the one worked above — but Levy and Goldberg's "Neural Word Embedding as Implicit Matrix Factorization" (NeurIPS 2014) proved they are closer than they appear: skip-gram with negative sampling is, implicitly, also factorizing a matrix — one whose entries are a shifted form of pointwise mutual information between word and context. Both families end up solving structurally similar factorization problems from different entry points; the practical difference is mainly engineering (GloVe needs the full co-occurrence matrix in memory or on disk before training starts, skip-gram streams through the corpus once).
Misconception: "embeddings capture the meaning of a word"
Every embedding produced by word2vec or GloVe is static — one fixed vector per vocabulary entry, looked up from a table and never altered by the sentence it appears in. This is precisely what the worked example above did: "cricket" gets exactly one row in W_in, reused identically whether the sentence is about a cricket match or, hypothetically, an insect. The common misconception is treating that single vector as "the meaning of the word" — but a fixed vector cannot represent a genuinely ambiguous word correctly, because it has to average over every sense the word can take across the whole training corpus. "Bank" appears near "river," "erosion," and "flood" in some sentences and near "loan," "interest," and "deposit" in others; a static embedding for "bank" is pulled toward some blended compromise of both neighbourhoods, useful for coarse similarity but wrong for either specific sense.
Transformer-based contextual embeddings fix exactly this. In BERT (Devlin, Chang, Lee, and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," NAACL 2019), there is no static per-word lookup table producing the final representation — the initial token embedding is only the first layer's input. Every subsequent transformer layer applies self-attention, letting each token's representation be recomputed as a weighted combination of every other token in the same sequence. The vector assigned to "bank" in "the bank raised interest rates" and the vector assigned to "bank" in "the river overflowed its bank" start from the same input embedding but diverge through the network, because attention pulls in different neighbouring tokens in each sentence. Illustrating the contrast numerically: under a static scheme, cosine similarity between "bank"-in-sentence-1 and "bank"-in-sentence-2 is trivially 1.0, since it is the literal same stored vector both times. Under a contextual scheme, that similarity typically comes out well below 1 — the two occurrences are related (they're both grammatically nouns, both English) but no longer identical, because the surrounding words genuinely changed what "bank" is doing in each sentence. If a student's mental model of "embedding" is "each word gets one point in space, forever," that model is describing word2vec/GloVe specifically, not embeddings in general — and it silently breaks the moment BERT-style contextualization enters the picture.
From token embeddings to sentence embeddings for retrieval
The search example that opened this chapter needs a vector for an entire query or an entire product description, not one vector per token. The naive way to get one from BERT is pooling: either take the special [CLS] token's final-layer vector, or average every token's final-layer vector. Reimers and Gurevych's Sentence-BERT paper (EMNLP 2019) showed this naive pooling performs surprisingly poorly for semantic similarity tasks — on some benchmarks, even worse than simply averaging plain GloVe vectors — because BERT's training objectives (masked language modeling, next-sentence prediction) never explicitly taught it that cosine similarity between pooled sentence vectors should track sentence-level meaning. The fix is a further fine-tuning stage that directly optimizes for that property: a siamese network encodes two sentences with a shared BERT, and a contrastive or triplet loss pulls semantically similar sentence pairs together in cosine-similarity terms while pushing dissimilar pairs apart — reshaping the embedding space for exactly the geometric property the search example relies on.
Gao, Yao, and Chen's SimCSE (EMNLP 2021) pushed this further with an almost absurdly simple recipe: pass the same sentence through the encoder twice, relying only on dropout's randomness to produce two slightly different embeddings, and treat that pair as a "positive" while every other sentence in the training batch serves as a negative. The training objective is InfoNCE (van den Oord, Li, and Vinyals, "Representation Learning with Contrastive Predictive Coding," 2018), a direct generalization of the SGNS loss derived above from "one positive, k negatives" to "one positive, all other in-batch examples as negatives," scored with cosine similarity instead of a raw dot product and sharpened by a temperature parameter. This is the same distributional-hypothesis machinery from the start of the chapter, one abstraction level up: instead of "predict the neighbouring word," the objective becomes "recognize which of these candidate sentences means the same thing" — and the resulting sentence vectors are what production retrieval systems, including retrieval-augmented generation (RAG) pipelines, store and search over.
Production tradeoffs: dimension, storage, and search
Deploying embeddings at scale surfaces engineering constraints that never show up in a two-dimensional toy example. Storage is the first one: a corpus of 100 million product listings embedded at 768 dimensions in 32-bit floats costs 100,000,000 × 768 × 4 bytes ≈ 307 GB just for the vectors, before any index overhead — which is why many production systems quantize to 8-bit integers (a 4× reduction) or even binary embeddings (a 32× reduction), accepting a small recall loss for a large memory and bandwidth win. Search is the second: finding the single nearest neighbour to a query among 100 million vectors by brute-force cosine similarity means 100 million dot products per query, far too slow for an interactive search bar. Production systems instead use approximate nearest neighbour (ANN) indexes — HNSW (Malkov and Yashunin, "Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs," IEEE TPAMI 2018) builds a multi-layer navigable graph over the embedding space so that search descends through progressively finer layers, touching a tiny fraction of the full dataset while still finding a neighbour close to the true nearest one with high probability. For an Indian platform serving queries across Hindi, Tamil, Kannada, and code-mixed Hinglish simultaneously, the embedding model itself also has to be multilingual — general English-trained embeddings underrepresent Indic scripts and transliterated text, which is the specific gap Google's MuRIL embeddings (Khanuja et al., 2021) were trained to close by including transliteration pairs directly in pretraining. Every one of these choices — dimension, quantization, index structure, and language coverage — trades retrieval quality against latency and cost, and the tradeoff has to be made explicitly rather than inherited by default from whichever pretrained model was easiest to download.
Diagram: one SGNS training step, end to end
Active recall
Q1. Why is the dot product between any two distinct one-hot word vectors always exactly 0, and why does that make one-hot encoding useless for measuring word similarity?
Q2. Using the W_in/W_out matrices from the worked example, compute one SGNS training step where the center word is "train," the positive context is "ipl," and the negative sample is "stadium." Report the two dot products, the two sigmoid values, and the total loss.
Q3. True or false, with justification: "BERT assigns one fixed embedding vector to each word, the same way word2vec does."
Q4. In the chapter's main worked example (center = cricket, positive = stadium, negative = samosa), the learning rate is raised from η = 0.1 to η = 0.5 for that same single step. Recompute v_c_new, and state what happens to u_o and u_k under the same higher rate — not just v_c.
Q5. In the e-commerce search example, "sasta chappal" scored a lower cosine similarity to the query "sasta mobile" than "expensive flagship phone" did (0.7097 vs 0.7618), even though "sasta chappal" literally shares a token with the query. Explain why a well-trained embedding space produces this ranking rather than the reverse.
Q6. Is GloVe trained with backpropagation through a neural network the way skip-gram is? What did Levy and Goldberg (2014) show about the relationship between the two methods?
Answers
A1. A one-hot vector for word i has a 1 only at coordinate i and 0 everywhere else. For two different words i ≠ j, computing the dot product means summing, over every coordinate, the product of the two vectors' entries at that coordinate — and at every coordinate, at least one of the two vectors is 0 (the 1s sit at different positions), so every term in the sum is 0, and the total is 0. This holds regardless of how semantically related the two words are — "cricket"·"IPL" and "cricket"·"samosa" are both exactly 0. Since the encoding provides no numeric signal that varies with semantic relatedness, no amount of downstream arithmetic on one-hot vectors alone can recover similarity; the representation itself has to change, which is exactly what training an embedding matrix accomplishes.
A2. v_train = [0.05, 0.05], u_ipl = [0.20, 0.10], u_stadium = [0.40, -0.10]. Positive dot product: x = 0.05(0.20) + 0.05(0.10) = 0.010 + 0.005 = 0.0150. Negative dot product: y = 0.05(0.40) + 0.05(-0.10) = 0.020 - 0.005 = 0.0150 (coincidentally equal to x for these particular numbers). Sigmoids: σ(0.0150) = 0.5037, σ(-0.0150) = 0.4963. Loss: L = -log(0.5037) - log(0.4963) = 0.6857 + 0.7007 = 1.3864. Both dot products sit near zero, meaning this particular center/context/negative triple is currently almost indistinguishable to the model — the loss is close to the theoretical maximum-uncertainty value of 2·log(2) ≈ 1.386, confirming the model has learned almost nothing yet about how "train" relates to "ipl" versus "stadium."
A3. False. Word2vec stores one row per vocabulary word in a lookup table, and that row is returned unchanged no matter what sentence the word appears in. BERT's output representation for a token is instead the result of passing an initial embedding through many self-attention layers, where each layer recomputes every token's vector as a function of every other token in the same input sequence. The same word therefore gets different final vectors in different sentences — "bank" in a finance sentence and "bank" in a river sentence start from the same input embedding but diverge through the network, which is precisely the property static embeddings lack and the one that makes BERT-style models handle polysemy correctly.
A4. The gradient computed in the worked example was ∂L/∂v_c = [-0.0984, -0.0883]. With η = 0.5: v_c_new = [0.50,-0.20] - 0.5[-0.0984,-0.0883] = [0.50+0.0492, -0.20+0.0442] = [0.5492, -0.1559] — a much larger jump than the η = 0.1 result of [0.5098, -0.1912]. Critically, the gradient step is not applied to v_c in isolation: the same learning rate scales the updates to u_o and u_k too, since all three vectors are parameters updated by the same optimizer step. Using ∂L/∂u_o = [-0.2226, 0.0890] and ∂L/∂u_k = [0.2656, -0.1062] computed earlier: u_o_new = [0.40,-0.10] - 0.5[-0.2226,0.0890] = [0.5113, -0.1445], and u_k_new = [0.15,-0.25] - 0.5[0.2656,-0.1062] = [0.0172, -0.1969]. A student who only recomputes v_c and stops has missed that raising η reshapes every vector touched by that training step, and with a large enough η the negative vector u_k can be pushed to overshoot past where it would stabilize with careful tuning — which is exactly why learning rates are annealed downward over training rather than held at a large fixed value throughout.
A5. A well-trained embedding space organizes vectors by what the surrounding context of each phrase tends to be across the training corpus, not by shared characters. "Sasta chappal" and "sasta mobile" do share the literal token "sasta," but the words that typically surround "chappal" (footwear, size, sole, strap) and the words that typically surround "mobile" (RAM, camera, battery, processor) come from almost entirely disjoint distributions, so the distributional hypothesis pulls their embeddings apart despite the shared token — token overlap is not what the model was ever trained to reward. "Expensive flagship phone" shares no tokens with "sasta mobile" but shares almost the entire distributional neighbourhood (both are phone-category text, differing only on the price attribute), so the embeddings land closer together. This is the same mechanism as the chapter's opening example: geometry tracks meaning, not spelling.
A6. No — GloVe's training procedure is weighted least-squares regression over a precomputed word-by-word co-occurrence count matrix; there is no sequential prediction step and no need to stream through the corpus with a sliding window during optimization, unlike skip-gram's per-window forward/backward passes. Despite this architectural difference, Levy and Goldberg (2014) showed that skip-gram with negative sampling is implicitly performing a matrix factorization of its own — specifically of a matrix of shifted pointwise mutual information values between words and contexts — which is conceptually the same family of problem GloVe solves explicitly and directly. The two methods arrive at similar embedding geometries from different computational routes: one factorizes global counts directly, the other factorizes an equivalent quantity implicitly through local stochastic updates.
Think About It
Think about this: How would you explain embedding models: learning dense representations to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind embedding models: learning dense representations, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.