Word2Vec and GloVe: Word Embeddings

The Search Bar That Understands You

Open any shopping app on your phone and type chappal into the search box. Along with results for chappals, you will often see sandals and slippers too — even listings where the seller never typed the word chappal at all. No human sat down and wrote a rule saying "chappal means the same thing as sandal." The app's search system learned this on its own, from millions of past searches, clicks, and product listings, by noticing that these words tend to show up in the same kind of surroundings: the same product categories, the same customer questions, the same reviews. That single idea — judging a word by the company it keeps — is the entire engine behind this chapter's topic: word embeddings, and the two techniques that made them practical at scale, Word2Vec and GloVe.

Before embeddings, a computer had no real way to know that "chappal" and "sandal" were related. To a program, they were just two different strings of characters — as different from each other as "chappal" and "umbrella." Word embeddings fix this by representing every word as a list of numbers — a vector — positioned in space so that words used in similar ways end up physically close together. Getting there means first seeing why the obvious way to turn words into numbers fails, then how Word2Vec and GloVe solve it in two different but related ways — arriving, along the way, at one of the most famous demonstrations in the history of NLP, where simple arithmetic on word vectors recovers a real-world relationship.

Why One-Hot Encoding Falls Apart

The most obvious way to turn words into numbers is one-hot encoding. Take your vocabulary — every distinct word your app has ever seen, perhaps 50,000 of them — and give each word its own index. The word "cricket" might be word #4,521. Its vector is 50,000 numbers long: all zeros, except a single 1 at position 4,521. "Football" might be word #4,522, represented by 50,000 numbers with the 1 at position 4,522 instead.

This works as an identifier, but it destroys any notion of meaning. Take the dot product of the one-hot vectors for "cricket" and "football" — multiply matching positions together and add up the results. Since the two vectors have their single 1 in different positions, every term in that sum is zero, so the dot product is exactly 0, and so is the cosine similarity between them. Now try "cricket" and "umbrella." Same result: 0. One-hot encoding treats every pair of distinct words as equally, maximally unrelated — "cricket" is exactly as similar to "football" as it is to "umbrella," which is to say, not similar at all, ever, by construction. It also does not scale: real vocabularies run into hundreds of thousands of words, so every word becomes a huge vector that is almost entirely empty.

What we actually want is a much shorter vector — perhaps 100 to 300 numbers instead of 50,000 — where the numbers are not mostly zero, and where words that behave alike in real sentences end up with vectors pointing in similar directions. That is what an embedding is: a dense, learned, low-dimensional vector for each word, positioned by meaning rather than by an arbitrary index.

The Distributional Hypothesis

The idea that makes learning such vectors possible is almost a century old in linguistics, long before any computer could use it. In 1957, the linguist J.R. Firth summed it up in a line every NLP course eventually quotes: "You shall know a word by the company it keeps." This claim, known as the distributional hypothesis, holds that words appearing in similar contexts tend to have similar meanings. "Cricket" and "football" both regularly appear near words like "match," "stadium," "score," and "team." "Cricket" and "umbrella" almost never share that kind of company. A model does not need to be told what "cricket" means — it only needs to notice which words habitually surround it, across enough text, and it can infer a great deal about its meaning purely from that pattern.

Word2Vec and GloVe are two different engineering answers to the same question: given a large body of text, how do you turn "which words keep company with which" into an actual vector for every word? Word2Vec answers it by training a model to predict company from a word, or a word from its company. GloVe answers it by directly counting company kept, across the whole corpus, and fitting vectors to match those counts. Word2Vec was introduced by Tomas Mikolov and colleagues at Google in 2013; GloVe was introduced by Jeffrey Pennington, Richard Socher, and Christopher Manning at Stanford in 2014. Both are still worth understanding in detail, because the ideas underneath them reappear throughout modern NLP, right up to the embedding layers inside today's large language models.

Word2Vec: Learning Vectors by Playing a Prediction Game

Word2Vec never explicitly counts anything. Instead, it sets up a prediction task, trains a small neural network to get good at that task, and then — almost as a side effect — keeps the network's internal number-lists as the word vectors, discarding the prediction task itself once training is done. The task is never the real goal; it exists only to force the network to learn something useful.

Every sentence in the training corpus is broken into overlapping windows. Pick a window size — say 2 — and slide it across each sentence. For every word in the sentence (now called the center word), the words up to 2 positions to its left and right become its context words. Word2Vec comes in two architectures, differing only in which direction the prediction runs:

• CBOW (Continuous Bag of Words) feeds in the context words and asks the network to predict the missing center word. It trains faster and tends to work well for frequent words.
• Skip-gram runs the prediction backwards: feed in the center word, and ask the network to predict each surrounding context word. It trains more slowly than CBOW but is generally the better choice on smaller datasets and for learning good vectors for rare words, since every occurrence of a rare word generates several training examples — one per context position — instead of just one.

Under the hood, both architectures maintain two vectors for every word in the vocabulary: an "input" (center) vector and an "output" (context) vector, stored as two separate matrices. Once training finishes, the input vectors are what everyone means by "the Word2Vec embeddings"; the output vectors are typically discarded. The rest of this section works through skip-gram, the architecture behind the widely distributed pretrained Word2Vec vectors and the analogy results discussed later in this chapter.

Why Full Softmax Doesn't Scale

Skip-gram's raw training signal, for one (center word, context word) pair, is a probability: p(context word | center word). The natural way to compute that probability is a softmax over the entire vocabulary — take the dot product of the center word's input vector with every word's output vector, exponentiate each score, and divide by the sum of all of them. That normalizing sum is the problem. If the vocabulary has 100,000 words, every single training example requires touching all 100,000 output vectors just to compute one probability, and touching them again during the gradient update. Multiply that by a corpus with billions of word occurrences, and full-softmax training becomes impractical.

Mikolov and colleagues solved this in their second Word2Vec paper (2013) with negative sampling. Instead of asking "which of these 100,000 words is the correct context word," negative sampling turns the problem into a much smaller yes/no question repeated several times: "does this (center, context) pair actually occur together, or not?" For every real pair pulled from the corpus, the algorithm also manufactures a handful of fake pairs by keeping the same center word and swapping in a small number k of randomly sampled "negative" words — words that were not actually in that window. A typical choice is k between 5 and 20 for smaller training sets, and as low as 2 to 5 for very large ones. The network is trained like logistic regression: push the real pair's score up, push each fake pair's score down. The random words are not drawn uniformly — Word2Vec samples them from the corpus's word-frequency distribution raised to the power 0.75, a detail the original paper found worked distinctly better than sampling uniformly or sampling in exact proportion to frequency, since it moderately boosts the chances of rarer words being picked as negatives instead of letting a handful of extremely common words dominate every batch.

The effect is dramatic: instead of updating 100,000 output vectors per training example, the model now updates only k+1 of them — one positive, k negative. This single change is what made training embeddings on web-scale text corpora feasible.

Worked Example: One Skip-gram Update by Hand

The mechanics are easiest to see on a corpus small enough to compute by hand. Take three toy sentences:

virat plays cricket
sachin plays cricket
virat plays chess

The vocabulary has five words: virat, sachin, plays, cricket, chess. Use a window size of 1, so only immediately adjacent words count as context. Focus on one training pair drawn from the first sentence: center word plays, true context word cricket.

Give every word a made-up 2-dimensional starting vector (real systems use 100 to 300 dimensions; 2 keeps the arithmetic visible). Each word has an input vector v, used when it is the center word, and an output vector u, used when it is a candidate context word:

          input vector v          output vector u
virat     ( 0.60,  0.20)          ( 0.55,  0.15)
sachin    ( 0.50,  0.10)          ( 0.45,  0.25)
plays     ( 0.10,  0.40)          ( 0.20,  0.35)
cricket   ( 0.30,  0.50)          ( 0.40,  0.50)
chess     (-0.40,  0.30)          (-0.30,  0.25)

Step 1 — score every word as a candidate context for "plays." Compute the dot product of v(plays) = (0.10, 0.40) with every word's output vector u:

score(virat)   = 0.55(0.10) + 0.15(0.40) = 0.115
score(sachin)  = 0.45(0.10) + 0.25(0.40) = 0.145
score(plays)   = 0.20(0.10) + 0.35(0.40) = 0.160
score(cricket) = 0.40(0.10) + 0.50(0.40) = 0.240
score(chess)   = -0.30(0.10) + 0.25(0.40) = 0.070

Step 2 — turn the scores into probabilities with softmax. Exponentiate each score and divide by their sum:

e^0.115 = 1.1219    e^0.145 = 1.1560    e^0.160 = 1.1735
e^0.240 = 1.2712    e^0.070 = 1.0725
sum Z = 5.7952

p(virat)   = 1.1219 / 5.7952 = 0.1936
p(sachin)  = 1.1560 / 5.7952 = 0.1995
p(plays)   = 1.1735 / 5.7952 = 0.2025
p(cricket) = 1.2712 / 5.7952 = 0.2194
p(chess)   = 1.0725 / 5.7952 = 0.1851

Before any training, this tiny model already assigns "cricket" the highest probability of the five (21.9%, against a 20% baseline if it had no opinion at all) — a coincidence of the starting vectors chosen here, not a general property, but useful for seeing the mechanics clearly.

Step 3 — measure the loss. The training signal is cross-entropy loss on the true context word, cricket: L = −ln(p(cricket)) = −ln(0.2194) ≈ 1.517. For comparison, a model with no information at all — a uniform 1/5 guess for every word — would score −ln(0.2) = ln(5) ≈ 1.609. The gap between 1.517 and 1.609 is the small amount this random initialization already "knows."

Step 4 — compute the gradient and take one step. For a softmax layer with cross-entropy loss, the gradient of the loss with respect to the center vector has a clean closed form: sum, over every vocabulary word w, of (p(w) − y(w)) · u(w), where y(w) is 1 for the true context word and 0 for every other word. Each word's output vector is pulled into the gradient, weighted by how far that word's predicted probability was from its correct value.

(p - y):  virat 0.1936   sachin 0.1995   plays 0.2025
          cricket -0.7806   chess 0.1851

gradient = sum over w of (p(w)-y(w)) * u(w)
  x: 0.1936(0.55)+0.1995(0.45)+0.2025(0.20)-0.7806(0.40)+0.1851(-0.30) = -0.1310
  y: 0.1936(0.15)+0.1995(0.25)+0.2025(0.35)-0.7806(0.50)+0.1851(0.25) = -0.1943

gradient ≈ (-0.1310, -0.1943)

Gradient descent moves the center vector in the opposite direction, scaled by a learning rate η. With η = 0.1:

v(plays)_new = v(plays) - eta * gradient
             = (0.10, 0.40) - 0.1 * (-0.1310, -0.1943)
             = (0.10 + 0.0131, 0.40 + 0.0194)
             = (0.1131, 0.4194)

Notice the structure of that gradient: cricket is the only word with a negative (p−y) coefficient, because it is the only word being pulled toward instead of pushed away from. Every other word contributes a small positive coefficient. Repeat this update across every window in a real corpus — millions or billions of times, over every center/context pair the text contains — and the accumulated effect is exactly what the distributional hypothesis predicts: words that repeatedly show up as each other's context get their vectors pulled together, step after step, while words that never share context drift apart. Nobody hand-designs the final coordinates; they fall out of many small, mechanical updates like the one above.

What Falls Out: Analogies in Vector Space

Trained on a large enough corpus — the original public release trained on roughly 100 billion words from Google News, producing 300-dimensional vectors for 3 million words and phrases — the resulting space captures far more than "similar words are close together." It captures relationships as directions. The single most famous demonstration from Mikolov's original papers: take the vector for "king," subtract the vector for "man," add the vector for "woman," and search for the closest vector in the whole vocabulary to the result. It lands on "queen." The direction from "man" to "woman" behaves, consistently, like a "make this royal title feminine" operation, and adding that same direction to other words recovers other real male-female word pairs. This was not hand-engineered; it emerged purely from the prediction task and the geometry it produced.

Using a library such as gensim, training a (toy) Word2Vec model takes only a few lines:

from gensim.models import Word2Vec

sentences = [
    ["virat", "plays", "cricket"],
    ["sachin", "plays", "cricket"],
    ["virat", "plays", "chess"],
]

model = Word2Vec(
    sentences,
    vector_size=50,   # dimensions per word vector
    window=2,          # context words on each side
    min_count=1,        # keep even single-occurrence words (toy corpus only)
    sg=1,               # 1 = skip-gram, 0 = CBOW
    epochs=100,
)

print(model.wv["cricket"].shape)      # (50,)
print(model.wv.most_similar("cricket"))

With a corpus of three sentences, the resulting vectors mostly reflect noise rather than real meaning — Word2Vec needs a training set several orders of magnitude larger before its geometry becomes trustworthy. The code above is correct and will run; it is included to show the interface, not to claim three sentences can teach a model what "cricket" means.

GloVe: Starting from Counts Instead of Predictions

GloVe — Global Vectors for word representation — approaches the identical goal from the opposite direction. Instead of sliding a window across text and training a network to predict one word from another, GloVe's first step is to count, once, across the entire corpus: how many times does each pair of words appear within some window of each other? This produces a co-occurrence matrix, with one row and one column per vocabulary word.

Take the same three-sentence corpus and a window of 1 (immediate neighbours only). Counting how often each pair of words sits next to each other gives:

          virat  sachin  plays  cricket  chess
virat       0      0       2      0        0
sachin      0      0       1      0        0
plays       2      1       0      2        1
cricket     0      0       2      0        0
chess       0      0       1      0        0

Two things are already visible before any vector is fitted. First, "plays" co-occurs with everything — it is a hub, exactly the behaviour of a common word that glues a sentence together. Second, "cricket" and "chess" never co-occur with each other, but they share a neighbour: both co-occur only with "plays." This kind of pattern — not raw counts, but which words share which neighbours, and in what proportion — is what GloVe's objective is built to exploit. The original GloVe paper illustrates the idea with "ice" and "steam": both words co-occur with "water" about equally often, so raw counts alone don't distinguish them; but "ice" co-occurs with "solid" far more than "steam" does, and "steam" co-occurs with "gas" far more than "ice" does. It is the ratio of co-occurrence probabilities across many such probe words, not any single count, that cleanly separates related meanings from unrelated ones — and GloVe's training objective is built directly around fitting those ratios.

Formally, GloVe learns a word vector w and a context vector w̃ for every word (plus a scalar bias for each, combined at the end into one final vector per word — a two-vector-per-word setup similar in spirit to Word2Vec's input/output split), by minimizing:

J = sum over pairs i,j of  f(X_ij) * (w_i . w~_j + b_i + b~_j - log(X_ij))^2

summed only over pairs of words i, j whose co-occurrence count X_ij is greater than zero — pairs that never co-occur are simply skipped, since log(0) is . The squared term measures how far the model's current dot product is from the log of the actual count: for the ("plays", "cricket") pair above, X_ij = 2, so log(X_ij) ≈ 0.693, and training nudges w(plays)·w̃(cricket), plus the bias terms, toward that value. f(X_ij) is a weighting function — the original paper uses f(x) = (x / 100)^0.75 for counts below 100, capped at 1 above that — which prevents two problems at once: rare co-occurrences, which are mostly noise, get down-weighted, and extremely frequent pairs, mostly involving function words like "the" or "of," don't dominate the loss just because they are common. Where Word2Vec learns its vectors as a by-product of a prediction task repeated over sliding windows, GloVe fits its vectors directly to global counts in essentially one pass, through what is a form of weighted matrix factorization.

Loading and using pretrained GloVe vectors is similarly simple:

import gensim.downloader as api

glove = api.load("glove-wiki-gigaword-100")   # 100-dimensional vectors

print(glove.similarity("cricket", "football"))
print(glove.most_similar("cricket", topn=5))

similarity() returns the cosine similarity between two word vectors — the cosine of the angle between them, ranging from -1 to 1, where values near 1 mean the two words are used in strongly overlapping contexts.

Two Routes to the Same Destination

Word2Vec and GloVe look like they come from different worlds — one a small neural network trained example by example with stochastic gradient descent, the other a global count matrix fit with weighted least squares — yet the vector spaces they produce end up strikingly similar in what they capture: the same clustering of related words, the same rough shape of analogy relationships. This is not a coincidence. In 2014, researchers Omer Levy and Yoav Goldberg showed that skip-gram with negative sampling is, mathematically, implicitly factorizing a matrix built from pointwise mutual information between words — a count-based statistic very close in spirit to what GloVe fits directly and explicitly. Two approaches that look unrelated on the surface turn out to be solving close variants of the same underlying arithmetic problem.

In practice, the two are usually chosen on more practical grounds. GloVe needs the entire co-occurrence matrix built before training starts, which makes it awkward for streaming or constantly growing corpora, though its explicit global counting can make efficient use of statistics for words that occur only a few times in any single local window. Word2Vec never needs to materialize that matrix — it processes one sentence at a time — which makes it a natural fit for very large or continuously updated text streams.

The Limit Both Share: One Vector Per Word

Word2Vec and GloVe both assign exactly one fixed vector to each word, no matter how that word is used in a sentence. This is a real limitation. The word "bank" gets a single vector, blending its river-bank sense and its financial-institution sense together, because both uses were mixed into the same training statistics. A sentence like "he sat on the bank and opened his bank account" cannot be represented properly by either model — the two "bank"s are forced to share one point in vector space. These are called static embeddings for exactly this reason: the vector is fixed the moment training finishes and never adapts to context at use time.

This limitation is what motivated the shift to contextual embeddings later in the decade — models such as ELMo in 2018, and BERT not long after, which generate a different vector for the same word depending on the specific sentence it appears in, computed on the fly by a deep network rather than looked up in a fixed table. A second, separate gap — Word2Vec and GloVe have no vector at all for a word they never saw during training — was addressed by FastText in 2017, which builds word vectors out of smaller character-level pieces so that even unfamiliar words can be assigned a reasonable vector from their spelling alone. Both later ideas are direct descendants of the embedding idea introduced here: neither would exist without Word2Vec and GloVe first proving that "meaning as geometry" was a workable foundation for representing language.

Back to the Search Bar

Return to the shopping app from the start of this chapter. Once every word in its catalogue and its customers' queries has been mapped to a vector — trained by exactly the mechanics traced by hand above, just at a scale of millions of listings and queries instead of three toy sentences — search stops being a string-matching problem and becomes a geometry problem. A query like "chappal" is converted into its vector, and the app looks for product listings whose word vectors sit nearby in that same space: "sandals," "slippers," even related product types the original listing writer never typed. None of this required a human to write a synonym dictionary by hand. It required only a large corpus, a training objective — predict the company a word keeps, or count it directly — and enough repetitions of a simple update rule for meaning to crystallize into geometry. That, in one line, is what Word2Vec and GloVe contributed to the field: a way to turn the distributional hypothesis, a 1950s idea about language, into coordinates a machine could compute with.

Think About It

Think about this: How would you explain word2vec and glove: word embeddings to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind word2vec and glove: word embeddings, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.