The search bar that can't tell an iPhone from a fruit
Type "apple" into a large e-commerce search bar in India and the system has to decide, in milliseconds, whether you mean the fruit, the phone, or the laptop stand that says "for Apple devices" in its description. A recommendation engine built on the word-embedding techniques from a first NLP course — word2vec or GloVe — cannot make that decision, and the reason is structural, not a bug you can patch. Those models assign every word in the vocabulary exactly one vector, computed once during training and stored in a lookup table. "Apple" gets one row in that table. It does not matter whether the sentence around it is "apple slices for a fruit chaat" or "apple charger not included" — the model retrieves the identical 300-dimensional vector both times, because retrieval is all a lookup table can do.
This chapter is about the two ideas that fix this. First, how dense word embeddings are learned at all — from one-hot vectors, through word2vec's skip-gram objective, to GloVe's co-occurrence statistics. Second, why static lookup tables hit a hard ceiling on ambiguous language, and how the self-attention mechanism inside BERT produces a genuinely different vector for the same word depending on what sentence it sits in. Every number in this chapter is one you can recompute by hand or in five lines of NumPy — that is the point: contextual embeddings are not magic, they are a specific, traceable arithmetic operation repeated many times.
Why not just use one-hot vectors?
The naive way to give a neural network a word is a one-hot vector: a vector as long as the vocabulary, all zeros except a single 1 at that word's index. For a 50,000-word vocabulary, "cat" and "dog" are both 50,000-dimensional vectors that are orthogonal to each other and to every other word — their dot product is exactly 0, and their cosine similarity is exactly 0, regardless of how related the words actually are. The representation carries zero information about meaning; it is just an ID number wearing a vector costume. It is also wasteful: a network's first layer for a 50,000-word vocabulary needs a 50,000-wide input, most of which is zero on every single example.
Dense embeddings replace this with a short, learned vector — typically 100 to 768 dimensions — where geometric closeness is trained to mean semantic closeness. The question is how you get vectors with that property without a human hand-labeling "cat is 73% similar to dog."
Word2Vec: predicting context from a center word
Word2vec's skip-gram formulation makes a specific, checkable bet: a word can be represented by the company it keeps. Slide a window of size k across a huge corpus. At each position, treat the center word as input and the surrounding words within the window as prediction targets. Train an embedding for "bank" by making it good at predicting "river," "water," and "flows" when those words appear near it in the corpus, or good at predicting "loan," "interest," and "account" when those appear near it instead. The embedding matrix itself — not any downstream task — is the product; once training is done, the prediction task is thrown away and the vectors are kept.
Computing a full softmax over a 50,000-word vocabulary for every single context word, on every window, on a billion-word corpus, is too slow to be practical. Skip-gram with negative sampling reframes the problem as a set of small binary classification problems: for the true pair (center word, real context word), push their dot product up; for a handful of randomly sampled "negative" pairs (center word, some unrelated word that did not actually appear nearby), push their dot product down. This needs only a sigmoid, not a 50,000-way softmax, per pair.
Worked example: one step of skip-gram with negative sampling
Take a toy 2-dimensional embedding space (real word2vec uses 100–300 dimensions; 2 is chosen here purely so every multiplication is checkable by hand). Suppose training reaches a window where the center word is "sat," the true context word is "cat," and a randomly drawn negative sample is "dog." Current embeddings mid-training:
v(sat) = [ 0.5, -0.2] (center-word vector)
v(cat) = [ 0.1, 0.3] (context-word vector, positive pair)
v(dog) = [-0.4, 0.2] (context-word vector, negative sample)
The skip-gram negative-sampling loss for this one center/context pair, with one negative sample, is:
L = -log( sigma( v(sat)·v(cat) ) ) - log( sigma( -v(sat)·v(dog) ) )
Step through it. The positive dot product: v(sat)·v(cat) = 0.5(0.1) + (-0.2)(0.3) = 0.05 - 0.06 = -0.01. Passed through the sigmoid: sigma(-0.01) = 1 / (1 + e^0.01) = 1 / 2.01005 ≈ 0.4975. That is close to 0.5 — the model currently has almost no opinion on whether "sat" and "cat" co-occur, which is the correct state early in training.
The negative dot product: v(sat)·v(dog) = 0.5(-0.4) + (-0.2)(0.2) = -0.2 - 0.04 = -0.24. The loss wants this pair pushed apart, so it uses sigma(-(-0.24)) = sigma(0.24) = 1 / (1 + e^-0.24) = 1 / 1.78663 ≈ 0.5597.
Total loss: L = -ln(0.4975) - ln(0.5597) ≈ 0.6982 + 0.5804 = 1.2785 nats. Gradient descent on this loss moves v(sat) toward v(cat) (increasing their dot product, since the model was under-confident about a real co-occurrence) and away from v(dog) (decreasing that dot product, since "dog" did not actually appear near "sat" in this window). Repeat this update tens of billions of times across a real corpus and words that share contexts — "sat," "stood," "lay" — converge to nearby points in the embedding space, purely as a side effect of getting better at this prediction task.
GloVe reaches a similar destination by a different route: instead of sampling windows one at a time, it first builds a full word-by-word co-occurrence count matrix for the corpus, then factorizes it so that the dot product of two word vectors approximates the log of how often those words co-occur. Word2vec is a local, online estimate of the same statistic GloVe computes globally and directly. Both output one static vector per word.
Where the static picture breaks: the same "apple" problem, with numbers
Return to the search-bar hook with concrete (still toy, but representative) vectors. Suppose training has placed "apple" close to fruit words like "banana" and far from tech words like "iphone" — because in the training corpus, "apple" appeared near "banana" more often than near "iphone," in aggregate, across every sentence that ever used the word:
import numpy as np
def cosine_similarity(u, v):
return np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v))
apple = np.array([0.90, 0.80, 0.10]) # one vector, used for every sentence
banana = np.array([0.85, 0.75, 0.05])
iphone = np.array([0.10, 0.05, 0.95])
print(cosine_similarity(apple, banana)) # 0.999
print(cosine_similarity(apple, iphone)) # 0.19
Both print statements use the identical vector apple. Cosine similarity with "banana" comes out to (0.9·0.85 + 0.8·0.75 + 0.1·0.05) / (‖apple‖·‖banana‖) = 1.37 / (1.2083 × 1.1347) ≈ 0.999 — the model is confident "apple" means the fruit. Cosine similarity with "iphone" comes out to (0.9·0.1 + 0.8·0.05 + 0.1·0.95) / (1.2083 × 0.9566) = 0.225 / 1.1560 ≈ 0.19 — the same model is confident "apple" has almost nothing to do with phones. Both statements are trying to describe the exact same word using the exact same vector, and they cannot both be right for a query like "apple charger" versus "apple juicer." A ranking algorithm built on this vector will systematically misrank one of the two senses, because word2vec and GloVe average every sense of a word into a single point during training and never separate them again at inference time.
Contextual embeddings: let the sentence build the vector
The fix is to stop looking words up in a static table and instead compute each word's vector on the fly, as a function of every other word in the sentence it actually appears in. ELMo (2018) took a step in this direction using bidirectional LSTMs reading a sentence left-to-right and right-to-left and combining both. BERT (Bidirectional Encoder Representations from Transformers, 2018) replaced the recurrent machinery with the self-attention mechanism from the Transformer architecture, which lets every token look directly at every other token in the sentence in a single step, with no left-to-right bottleneck. This is the mechanism worth tracing by hand, because it is exactly how BERT turns one starting vector for "bank" into two different output vectors depending on sentence context.
Self-attention, traced by hand
Self-attention computes, for each token, a weighted average of every token's value vector, where the weights come from how strongly that token's query matches every token's key. In the simplified case used here, the query, key, and value projection matrices are all set to identity, so a token's query, key, and value are just its raw embedding — this isolates the attention mechanism itself without extra matrix algebra, exactly the way real BERT layers behave once you factor out the learned projections.
Take the sentence "river bank flows," with toy 2-dimensional embeddings:
river = [1, 0]
bank = [0, 1]
flows = [1, 1]
Compute the contextual vector for "bank" (the query). Its dot product against every key, scaled by 1/sqrt(d_k) with d_k = 2:
score(bank, river) = (bank · river) / sqrt(2) = 0 / 1.4142 = 0
score(bank, bank) = (bank · bank) / sqrt(2) = 1 / 1.4142 = 0.7071
score(bank, flows) = (bank · flows) / sqrt(2) = 1 / 1.4142 = 0.7071
Softmax those three scores: exp(0)=1, exp(0.7071)=2.0281, exp(0.7071)=2.0281, sum = 5.0562. That gives attention weights river: 0.198, bank: 0.401, flows: 0.401. The new contextual vector for "bank" is the weighted sum of the value vectors: 0.198·[1,0] + 0.401·[0,1] + 0.401·[1,1] = [0.599, 0.802]. In this sentence, "bank" has pulled 40% of its identity from "flows," the river sense.
Now run the identical mechanism on "cash bank loan," starting from the same raw vector for "bank" (this is the crucial point — the input embedding is unchanged; only the neighbors differ):
cash = [1, -1]
bank = [0, 1]
loan = [-1, 1]
score(bank,cash) = -1/1.4142 = -0.7071, score(bank,bank) = 0.7071, score(bank,loan) = 0.7071. Softmax: exp(-0.7071)=0.4931, exp(0.7071)=2.0281 (twice), sum=4.5493, giving weights cash: 0.108, bank: 0.446, loan: 0.446. New contextual vector: 0.108·[1,-1] + 0.446·[0,1] + 0.446·[-1,1] = [-0.337, 0.783].
Both computations start from the identical vector [0,1] for "bank." One produces [0.599, 0.802]; the other produces [-0.337, 0.783] — pulled in the opposite x-direction because "cash" and "loan" sit on the opposite side of the space from "river" and "flows." This is verifiable directly in code:
import numpy as np
def softmax(x):
e = np.exp(x - np.max(x))
return e / e.sum()
def attend(tokens, query_idx):
keys = np.stack(tokens)
query = tokens[query_idx]
d_k = keys.shape[1]
scores = (keys @ query) / np.sqrt(d_k)
weights = softmax(scores)
return weights, weights @ keys
river, bank, flows = np.array([1.,0.]), np.array([0.,1.]), np.array([1.,1.])
weights1, context1 = attend([river, bank, flows], query_idx=1)
print(weights1, context1) # [0.198 0.401 0.401] [0.599 0.802]
cash, bank2, loan = np.array([1.,-1.]), np.array([0.,1.]), np.array([-1.,1.])
weights2, context2 = attend([cash, bank2, loan], query_idx=1)
print(weights2, context2) # [0.108 0.446 0.446] [-0.337 0.783]
This is the whole trick, stacked twelve times (BERT-base) or twenty-four times (BERT-large), with the query/key/value projections learned rather than fixed to identity, and with 768 or 1024 dimensions rather than 2. Every layer lets every token re-average itself against every other token, so meaning genuinely propagates across the whole sentence rather than being fixed at input time.
BERT: how the pretraining actually forces this to happen
Self-attention alone does not guarantee good contextual vectors — it needs a training objective that forces the network to actually use context correctly. BERT uses two objectives together during pretraining on raw, unlabeled text.
Masked Language Modeling (MLM): randomly replace about 15% of input tokens with a special [MASK] token (or, some of the time, a random wrong word, to stop the model from only ever seeing the literal [MASK] symbol) and train the network to predict the original word at each masked position, using the entire surrounding sentence — words before and after the mask — simultaneously. This is what makes BERT bidirectional in a way a left-to-right language model like GPT is not: predicting a masked word in the middle of a sentence requires attending to tokens on both sides at once, which is exactly the mechanism traced above with "bank" attending to every other token in its sentence, forward and backward, in one pass.
Next Sentence Prediction (NSP): feed the model two sentences separated by a [SEP] token, prefixed with a special [CLS] token, and train it to predict whether the second sentence genuinely follows the first in the source document or is a random sentence swapped in. The final hidden state at the [CLS] position is used for this binary decision, and after pretraining that same [CLS] vector is reused as a whole-sentence summary for downstream classification tasks — sentiment, entailment, and so on.
Before either objective sees a token, BERT tokenizes with WordPiece rather than whole words, splitting rare words into known subword pieces (e.g. "unaffordability" might become "un", "##afford", "##ability"). This keeps the vocabulary a manageable ~30,000 pieces while still being able to represent essentially any word, including ones never seen during training, by composing it from familiar fragments. Each token's input representation is the sum of three learned vectors: a token embedding (which piece it is), a position embedding (where it sits in the sequence — attention on its own has no notion of order, so this has to be added explicitly), and a segment embedding (whether it belongs to sentence A or sentence B, for the NSP task).
Once pretrained on a large unlabeled corpus, BERT is fine-tuned: the same weights are lightly adjusted, with a small task-specific head added on top, for a specific downstream job — question answering, named-entity recognition, sentiment classification — usually needing far less labeled data than training a model from scratch, because the contextual representations already encode a great deal about how language works.
Common misconception
The mistake students consistently make after learning word2vec first is assuming BERT still hands you "the embedding for a word" as a fixed lookup — that somewhere there is a table with one row for "bank" that you can retrieve independent of the sentence, just a fancier version of the word2vec table. There is no such table. Every BERT vector is the output of the self-attention computation traced above, run fresh on the specific input sequence you gave it; ask for the vector of "bank" in ten different sentences and you get ten different vectors, as demonstrated numerically in the worked example — [0.599, 0.802] in one sentence, [-0.337, 0.783] in another, from the identical starting embedding. If your code calls a BERT model once per unique word to "cache the embedding," you have thrown away the entire reason to use BERT over word2vec and are paying transformer-level compute for word2vec-level output.
Active recall
Attempt each of these before reading the answers below.
- Why is a one-hot vector a poor word representation for a 50,000-word vocabulary, beyond just being long?
- What expensive computation does negative sampling let skip-gram avoid, and roughly how?
- Two toy word vectors are v_a = [1, 2] and v_b = [2, 1]. Compute their cosine similarity by hand.
- In self-attention, why divide the query-key dot products by sqrt(d_k) before applying softmax?
- True or false: BERT is trained the same left-to-right way as GPT, just with a bigger dataset. Justify your answer.
- In the worked "bank" example, if a third sentence surrounded "bank" with neighbor tokens even further in the [cash, loan] direction than [-1,1] and [1,-1], would you expect the resulting contextual vector to move further from [0.599, 0.802] or closer to it? Why?
Answers.
1. Beyond length, every pair of one-hot vectors is orthogonal by construction — their dot product is always exactly 0, so "cat" and "dog" are exactly as "similar" as "cat" and "spreadsheet." The representation encodes identity (which row is 1) but zero information about meaning, and there is no way for a downstream network to generalize from having seen "cat" to handling "dog" in a similar way, since nothing in the input signals they are related.
2. It avoids computing a softmax over the entire vocabulary (tens of thousands of terms) for every single training pair. Instead of asking "which of 50,000 words is the correct context word," it asks a handful of much cheaper yes/no questions: "is this the real context word?" (one positive pair) and "is this random word NOT the context word?" (a few sampled negative pairs), scored with a sigmoid rather than a full softmax.
3. cosine(v_a, v_b) = (v_a · v_b) / (‖v_a‖ ‖v_b‖) = (1×2 + 2×1) / (sqrt(1²+2²) × sqrt(2²+1²)) = 4 / (sqrt(5) × sqrt(5)) = 4/5 = 0.8.
4. As embedding dimension d_k grows, the VARIANCE of the dot product of two random vectors grows roughly in proportion to d_k (it is a sum of d_k independent terms), so its typical magnitude — the standard deviation — grows roughly in proportion to √d_k, not d_k itself; raw dot products can still become large as d_k grows. Large values pushed into softmax saturate it — the output collapses toward a one-hot distribution with vanishing gradients everywhere except the largest score. Dividing by sqrt(d_k) rescales the dot products back down to a range where softmax stays well-behaved and gradients keep flowing during training.
5. False. GPT is trained autoregressively, left-to-right, predicting each next token using only the tokens before it. BERT's Masked Language Modeling objective predicts masked tokens using context from both sides simultaneously in a single forward pass — that is the entire meaning of the "Bidirectional" in its name, and it is why BERT cannot be used to generate text one token at a time the way GPT can, but is well-suited to tasks that need full-sentence understanding.
6. It would move further from [0.599, 0.802], not closer. The contextual vector is a weighted average pulled toward whichever neighboring tokens dominate the attention weights; the [-0.337, 0.783] result already came from moderately opposed neighbors (cash=[1,-1], loan=[-1,1]). Neighbors placed even further in that same direction increase the magnitude of the negative scores and dot products driving the attention weights toward those tokens, pulling the weighted average even further toward their region of the space and away from the river-context result.
Think About It
Think about this: How would you explain advanced nlp: word embeddings to bert to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where advanced nlp: word embeddings to bert is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting advanced nlp: word embeddings to bert to at least 3 other topics you have studied.