A Search Bar That Understands "Sasta"
Open Flipkart or Amazon on your phone and type "sasta mobile under 15000" into the search bar. Within a fraction of a second, the results fill with listings titled "Best Budget Smartphones Under ₹15,000," "Affordable 5G Mobiles," and "Value-for-Money Android Phones." Look closely at those titles: not one of them contains the word "sasta." Yet the search engine treated "sasta" as though it meant almost exactly what "budget," "affordable," and "value-for-money" mean.
A search system built on plain keyword matching could never do this — it would look for the literal letters s-a-s-t-a inside a product title and find nothing. For the search to work, something in the system has to know, in some computable sense, that "sasta" and "affordable" point at the same idea. That "something" is a word embedding: a list of real numbers — a vector — assigned to every word in the vocabulary, built so that words with similar meaning end up close together when plotted in that number space. Two techniques, published a year apart, taught the field how to learn these vectors automatically from raw text, with no dictionary and no human labelling involved: Word2Vec, released by a team at Google in 2013, and GloVe, released by researchers at Stanford in 2014. This chapter builds both from first principles, traces their arithmetic by hand on small examples, and ends by returning to that search bar to see exactly why it works.
Why Computers Can't Just Use Words
Every machine learning model, however sophisticated, ultimately multiplies and adds numbers together. It cannot take the string "phone" as input the way a person reads it. So the first question any NLP system must answer is: how do you turn a word into numbers?
The most direct answer is one-hot encoding. Fix a vocabulary — say, every distinct word across an e-commerce catalog, 50,000 words in total. Give each word an index from 0 to 49,999. The vector for a word is then 50,000 numbers long: all zeros, except a single 1 sitting at that word's own index. "Phone" might become [0, 0, 1, 0, ..., 0] with its 1 at position 2; "mobile" might have its 1 sitting far away at position 41,238.
This works in the narrow sense that every word now has a distinct numeric vector. But it fails in two ways that matter enormously in practice:
- It is wildly wasteful. Every single word needs a vector as long as the entire vocabulary, almost all of it zeros. A catalog with 50,000 unique words needs a 50,000-dimensional vector just to represent one word — and real-world vocabularies, once you count named entities, brand names, and spelling variants, routinely run into the hundreds of thousands.
- It encodes zero information about meaning. Take the dot product of the one-hot vectors for "phone" and "mobile": since their 1s sit in different positions, every term in the product is 0 times something or something times 0, so the dot product is exactly 0. Now take the dot product of "phone" and "banana." Also exactly 0. One-hot encoding treats "phone" and "mobile" as precisely as unrelated as "phone" and "banana," because it was never given any way to represent relatedness in the first place.
What we actually want is a dense, low-dimensional vector — perhaps 100 to 300 numbers instead of 50,000 — where the geometry itself carries meaning, so that words which tend to mean similar things end up pointing in similar directions. That is exactly what Word2Vec and GloVe learn.
The Company Words Keep
Both algorithms rest on one linguistic idea, stated by the linguist J.R. Firth in 1957: "You shall know a word by the company it keeps." This is the distributional hypothesis — the claim that words which occur in similar contexts tend to have similar meanings.
Consider the sentence "The chef added a pinch of ___ before serving." The blank could be filled by "salt," "jeera," or "haldi" — and precisely because these words are interchangeable in this kind of sentence, across thousands of recipes, they must denote related things (spices and seasonings). Nobody has to tell a model that "jeera" and "haldi" are both spices; the fact falls directly out of how these words get used around other words, over and over, across a large amount of text.
Word2Vec and GloVe both exploit this idea, but through different mechanisms. Word2Vec trains a small neural network on a prediction task built from local context windows and keeps the weights it learns along the way. GloVe instead counts, upfront, how often every pair of words occurs together across the whole corpus, then factorizes that count table directly. We will build both.
Word2Vec: Two Ways to Predict Context
Word2Vec was introduced by Tomas Mikolov and colleagues at Google in 2013, in the paper "Efficient Estimation of Word Representations in Vector Space." Its central trick is almost sleight of hand: set up an easy, free-to-generate supervised learning problem — predicting a word from the words around it — train a small neural network to solve that problem, and then throw away the predictions themselves. What you keep is the network's weight matrix. Each row of that matrix turns out to be a dense vector for one word, and because the network was forced to get good at predicting real context from millions of sentences, those vectors end up organized by meaning.
Word2Vec comes in two architectures that differ only in which direction the prediction runs:
- CBOW (Continuous Bag of Words) looks at the surrounding context words and predicts the missing word in the middle. Given "the ___ hits the ball," it tries to predict "batsman." The context words are averaged together before prediction, so their order inside the window does not matter — hence "bag of words."
- Skip-gram runs the task in reverse: given a single center word, it predicts each of the words around it. Given the center word "six," it tries to predict that "batsman," "hits," "boundary," and "crowd" are likely to appear nearby.
CBOW trains faster, since it makes one prediction per window position, and tends to do slightly better on frequent words. Skip-gram makes one prediction per center-context pair, so a single window produces several training examples instead of one — slower per pass through the corpus, but it exposes rare words to training far more often, so it typically represents infrequent words better. Both architectures share the same internal shape: a one-hot input, a linear hidden layer with no activation function in between (this is what keeps Word2Vec cheap enough to train on billions of words), and an output layer scored against the full vocabulary with a softmax. The pretrained vectors Google eventually released publicly — 300 dimensions across roughly three million words and phrases, trained on about 100 billion words of Google News text — became one of the most widely reused resources in NLP for years afterward, and give a useful sense of the scale these methods were built for.
Worked Example: One Skip-gram Step
To see exactly what gets computed, shrink everything to a toy vocabulary of five words drawn from cricket commentary: {bat, ball, six, wicket, bowler}, with a tiny embedding size of d = 2 so every number can be traced by hand.
Every word has two vectors during training: an input (center-word) embedding in matrix W, and an output (context-word) embedding in matrix W'. Suppose training has proceeded partway and the matrices currently hold these values:
W (center embeddings) W' (context embeddings)
bat: [ 0.10, 0.30] bat: [ 0.15, 0.35]
ball: [ 0.20, 0.25] ball: [ 0.50, 0.05]
six: [ 0.40, 0.10] six: [ 0.05, 0.45]
wicket: [-0.30, 0.20] wicket: [-0.25, 0.10]
bowler: [-0.20, -0.10] bowler: [-0.35, -0.05]
Take the center word "six." Its one-hot vector is [0, 0, 1, 0, 0]; multiplying it against W simply selects the "six" row, giving the hidden vector h = [0.40, 0.10] — this is the entire hidden layer, a plain lookup with no nonlinearity applied.
To score every vocabulary word as a candidate context word, take the dot product of h with each row of W':
u_bat = (0.15)(0.40) + (0.35)(0.10) = 0.095
u_ball = (0.50)(0.40) + (0.05)(0.10) = 0.205
u_six = (0.05)(0.40) + (0.45)(0.10) = 0.065
u_wicket = (-0.25)(0.40) + (0.10)(0.10) = -0.090
u_bowler = (-0.35)(0.40) + (-0.05)(0.10) = -0.145
These raw scores become probabilities through softmax, P(j) = exp(u_j) / Σ exp(u_k). Exponentiating each score gives approximately 1.0997, 1.2275, 1.0672, 0.9139, and 0.8650, which sum to about 5.1733. Dividing each by that sum:
P(bat) ≈ 0.2126
P(ball) ≈ 0.2373
P(six) ≈ 0.2063
P(wicket) ≈ 0.1767
P(bowler) ≈ 0.1672
If this training example came from the sentence "batsman hits the ball for a six" — so the true context word is "ball" — the network's loss is the negative log of the probability it assigned to the correct answer: -ln(0.2373) ≈ 1.44. Backpropagation now nudges every row of W and W' slightly, in whichever direction would have made P(ball) larger and the other four smaller. Repeat this single step for every center-context pair across a corpus of millions of sentences, and the "six" row of W gets pulled, over and over, toward whatever direction makes it good at predicting words like "ball," "boundary," and "hits" — and away from directions associated with "bowler" or "wicket." That pull, repeated billions of times, is the entire learning mechanism. No one ever tells the model what "six" means; its final vector is simply wherever prediction accuracy on real sentences pushed it.
Negative Sampling and the Analogy That Convinced Everyone
There is a problem with the softmax step above: it required computing a score against every single word in the vocabulary just to predict one context word. With a five-word toy vocabulary that is nothing, but Word2Vec is meant to run on vocabularies of hundreds of thousands of words, and that softmax denominator has to be recomputed on every single training step. At real scale this is far too slow to be practical.
Mikolov and colleagues solved this in a 2013 follow-up paper, "Distributed Representations of Words and Phrases and their Compositionality," with a technique called negative sampling. Instead of asking "which of these five hundred thousand words is the context word?" — one very expensive multi-class classification — the task is rewritten as several cheap yes-or-no questions. For the real pair (six, ball), the model is trained to output "yes, these co-occur." Then a handful of random words are drawn from the vocabulary that did not appear in that context — say, "aeroplane" or "biryani" — paired with "six" as negative examples, which the model is trained to label "no." The paper found roughly 5 to 20 negative samples per positive example useful for smaller corpora, dropping to as few as 2 to 5 for very large ones. A five-way, or even twenty-way, yes-or-no comparison is dramatically cheaper than a five-hundred-thousand-way softmax, and this single optimization is a large part of why Word2Vec could be trained on web-scale corpora at all. One further refinement: negative words are not drawn in proportion to their raw frequency, but to their frequency raised to the power 0.75, which stops extremely common words like "the" from being picked as a negative example almost every single time.
What made the field take these vectors seriously was not just that they clustered similar words together, but that they captured relationships as consistent directions in space. Trained on a large enough corpus, the vector arithmetic king - man + woman lands closest to the vector for queen — not because anyone encoded gender or royalty as a rule, but because the direction separating "man" from "woman" in the embedding space turns out to be nearly the same direction that separates "king" from "queen." The same regularity holds for country-capital pairs, one of the standard relationship types the original papers tested: the direction from a country's vector to its capital's vector stays roughly consistent across many countries, so a model that has learned this pattern will place Delhi - India + Japan close to Tokyo. This was the result that convinced the NLP community these were not just similarity scores, but a genuine, if rough, geometry of meaning.
GloVe: Counting Instead of Predicting
A year after Word2Vec, Jeffrey Pennington, Richard Socher, and Christopher Manning at Stanford published GloVe ("Global Vectors for Word Representation") at the 2014 Conference on Empirical Methods in Natural Language Processing, built on a different premise. Word2Vec only ever looks through a small local window as it slides across the corpus; it never directly uses the fact that, say, "ice" and "cold" co-occur across the entire corpus far more often than "ice" and "fashion" do. Older techniques like Latent Semantic Analysis did use such global counts, by factorizing a matrix built from the whole corpus, but tended to perform worse on tasks like analogy completion than Word2Vec's local-window approach. GloVe's premise was to combine both strengths: build one global co-occurrence matrix that counts, across the entire corpus, how often every word appears near every other word, and then learn vectors directly from that matrix.
GloVe's key theoretical move is to argue that raw co-occurrence probabilities are not the most useful signal on their own — the ratios between them are. A probe word strongly associated with one target word and not the other will have a probability ratio far from 1; a probe word associated with both, or with neither, will have a ratio close to 1.
Worked Example: Why Ratios Reveal Meaning
Suppose we scan a large corpus of cricket commentary and count how often each of four words — BATSMAN, BOWLER, CROWD, and STADIUM — appears within a small window around the words SIX and WICKET:
near SIX near WICKET
BATSMAN 40 8
BOWLER 5 45
CROWD 20 22
STADIUM 15 15
------ ------
total 80 90
Convert counts to probabilities by dividing by the column total — for instance, P(BATSMAN | SIX) = 40 / 80 = 0.5. Doing this for every cell and taking the ratio of the two columns:
P(·|SIX) P(·|WICKET) ratio
BATSMAN 0.5000 0.0889 5.625
BOWLER 0.0625 0.5000 0.125
CROWD 0.2500 0.2444 1.023
STADIUM 0.1875 0.1667 1.125
The ratio column is where the signal lives. BATSMAN's ratio sits far above 1 — it is heavily pulled toward SIX, which fits: batsmen hit sixes. BOWLER's ratio sits far below 1 — it is heavily pulled toward WICKET, since bowlers take wickets. CROWD and STADIUM both land close to 1, because a crowd and a stadium are present in roughly equal measure whether a six is struck or a wicket falls — neither word discriminates between the two events. This pattern would be invisible if you only looked at raw probabilities: knowing that P(STADIUM | SIX) = 0.1875 doesn't by itself tell you whether "stadium" is distinctive of "six" or just generically frequent — but the ratio does, immediately. This is precisely the pattern GloVe is built to exploit.
GloVe formalizes this by learning a word vector w_i and a context vector w̃_j for every word, so that their dot product approximates the log of the co-occurrence count between them, weighted so the fit is not dominated by extremely rare or extremely frequent pairs:
J = Σ f(X_ij) · (w_i · w̃_j + b_i + b̃_j - log X_ij)^2Here X_ij is how many times word j appears in the context of word i across the whole corpus, b_i and b̃_j are per-word bias terms, and f is a weighting function that rises smoothly for small counts and flattens to a constant above a cutoff (the original paper used a cutoff of 100 co-occurrences), so that a pair like "the" and "is," which co-occurs enormously often, does not dominate the loss simply by being frequent, while pairs that were never observed together contribute nothing at all. In effect, GloVe performs weighted matrix factorization on the log co-occurrence counts rather than training a predictive classifier — a genuinely different optimization problem that, in practice, lands on embeddings with very similar properties to Word2Vec's.
Word2Vec vs GloVe
The two methods start from different data structures and different objectives, but converge on the same kind of output:
- Signal. Word2Vec learns from local context windows as it streams through text, one prediction at a time. GloVe first builds a global co-occurrence matrix over the entire corpus, then factorizes it in one batch process.
- Objective. Word2Vec is predictive — a classification loss, either full softmax or negative-sampling binary classification, over whether a context word appears near a center word. GloVe is count-based — a weighted regression loss that reconstructs log co-occurrence counts.
- Scaling. Word2Vec never needs to store anything larger than its embedding matrices, so it scales naturally to streaming, arbitrarily large corpora. GloVe must build and hold a co-occurrence matrix first — large, though sparse, since most word pairs never co-occur — and then trains only over its non-zero entries.
- Reported performance. The original GloVe paper reported it matching or outperforming Word2Vec's skip-gram model on standard word-analogy benchmarks of the time. In practice, the two tend to produce embeddings of broadly comparable quality, and the better choice often depends on the corpus and downstream task rather than one method being categorically superior.
Both belong to a family called static embeddings — one fixed vector per word, regardless of context, so "bank" gets exactly the same vector whether the sentence is about a river bank or a savings account. Since 2018, contextual embedding models such as BERT compute a different vector for a word depending on the sentence it appears in, and have overtaken Word2Vec and GloVe for state-of-the-art accuracy on most NLP benchmarks. But Word2Vec and GloVe remain in active use wherever a lightweight, fast, easily inspected embedding is enough, and every embedding technique built since — contextual or not — is still answering the exact question these two methods answered first: what numbers should a word become, so that its meaning survives the translation?
Embeddings in Code
Once vectors exist, comparing meanings becomes pure geometry. The standard measure is cosine similarity — the cosine of the angle between two vectors, which ignores their length and asks only whether they point in the same direction:
import numpy as np
def cosine_similarity(u, v):
return np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v))
# the toy vectors from the skip-gram example above
six = np.array([0.40, 0.10])
bat = np.array([0.10, 0.30])
wicket = np.array([-0.30, 0.20])
print(round(cosine_similarity(six, bat), 3)) # 0.537
print(round(cosine_similarity(six, wicket), 3)) # -0.673
Even in this rough, partially trained toy example, "six" already sits closer to "bat" than to "wicket" — a positive cosine similarity against a negative one. In practice you would never hand-build these vectors; you would learn them from real text using a library such as gensim:
from gensim.models import Word2Vec
corpus = [
"virat kohli hits the ball for a six".split(),
"rohit sharma hits the ball for a four".split(),
"bumrah bowls a fast yorker".split(),
"the bowler celebrates after taking a wicket".split(),
"the batsman walks back after losing his wicket".split(),
]
model = Word2Vec(
sentences=corpus,
vector_size=50, # dimensionality of each word vector
window=3, # words of context on each side
min_count=1, # keep every word (toy corpus)
sg=1, # 1 = skip-gram, 0 = CBOW
epochs=200,
)
print(model.wv.most_similar("wicket", topn=3))
print(model.wv.similarity("kohli", "sharma"))
With only five sentences this particular model will not learn much — real training runs on corpora with millions or billions of words. But scale this same code up to years of match commentary, and model.wv.most_similar("wicket") would surface words like "bowler," "lbw," and "stumped," while model.wv.similarity("kohli", "sharma") would return a high score, since both names tend to appear in similar batting contexts. That is the distributional hypothesis, computed automatically.
Back to the Search Bar
Return to Flipkart's search box. Somewhere in its pipeline sits an embedding model — quite possibly a descendant of Word2Vec or GloVe, or a more modern contextual embedding trained on the same underlying principle — trained on enormous volumes of Indian e-commerce text: product listings, reviews, and past user queries. Across that text, "sasta," "budget," "affordable," and "value-for-money" kept turning up in the same kinds of sentences, next to the same kinds of products, in the same kinds of comparisons. The distributional hypothesis guarantees that any algorithm exploiting co-occurrence — whether by predicting context word by word like Word2Vec, or by factorizing a global co-occurrence matrix like GloVe — will place their vectors close together in the embedding space. Nobody wrote a rule mapping "sasta" to "budget"; the geometry emerged on its own, purely from how people actually write and search.
That is the real inheritance Word2Vec and GloVe left behind. Before 2013, turning text into numbers meant one-hot vectors that treated every pair of words as equally unrelated. After them, it meant dense vectors whose distances and directions genuinely tracked meaning — close together for synonyms, consistent directions for analogies, searchable with nothing more than a dot product and a square root. Every embedding technique built since, including the transformer-based models behind today's most capable language systems, is still answering the exact question Word2Vec and GloVe answered first: what numbers should a word become, so that its meaning survives the translation?
Think About It
Think about this: How would you explain word2vec and glove: learning word embeddings to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind word2vec and glove: learning word embeddings, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.