Word Embeddings: Word2Vec, GloVe, and FastText

IRCTC's support chatbot gets thousands of messages a day about the same problem, worded a thousand different ways: "train is late," "mera train delayed hai," "kitna late chal raha hai gaadi," "running behind schedule." A keyword-matching bot built on exact string comparison treats every one of these as a different, unrelated sentence — because to a computer, the strings "late" and "delayed" share not a single character in common at the representation level it actually reasons over. Meanwhile the same bot will confidently treat "train" and "biryani" — two words that never co-occur in any complaint about delays — as equally unrelated to "delayed," which is correct, but for the wrong reason: it has no notion of relatedness at all. It cannot tell you that "late" is closer to "delayed" than "biryani" is, because in its internal representation, every word is equally far from every other word. This chapter is about fixing exactly that: building numeric vectors for words such that geometric distance in the vector space tracks something real about how the words are used.

Why one-hot vectors fail

The naive fix for "words need numbers" is one-hot encoding: build a vocabulary of size V, and represent word i as a vector of length V with a 1 in position i and 0 everywhere else. This is exactly what you used for categorical features in earlier machine learning work. It has two fatal problems at NLP scale. First, dimensionality: a real vocabulary is 50,000–3,000,000 words, so every word vector is a spike in a space with millions of dimensions, and any downstream matrix multiply against it is enormous and wasteful. Second, and more damaging: every pair of distinct one-hot vectors has cosine similarity exactly 0 and Euclidean distance exactly $\surd 2$. "delayed" and "late" are precisely as similar as "delayed" and "biryani." One-hot vectors carry an identity, not a meaning.

The fix comes from a 1957 idea in linguistics, J.R. Firth's distributional hypothesis: a word is characterized by the company it keeps. Words that appear in similar contexts tend to have similar meanings. "delayed" and "late" both show up near "train," "flight," "arrival," "schedule." If you build a numeric representation from context statistics instead of from identity, similar-context words land near each other in vector space automatically. Word2Vec, GloVe, and FastText are three different engineering routes to the same statistical idea — dense, low-dimensional (typically 100–300 dimensions, not millions), and geometrically meaningful.

Word2Vec: predicting context from a neural network

Word2Vec (Mikolov et al., 2013) turns the distributional hypothesis into a supervised learning problem with no human-labeled data, using the text itself as the label. Slide a window across the corpus; at each position, you get a center word and its neighboring context words. Word2Vec comes in two architectures built on this same window:

• CBOW (Continuous Bag of Words): given the context words, predict the center word. Faster to train, tends to work better on frequent words.
• Skip-gram: given the center word, predict each context word. Slower, but tends to work better on rare words — which matters for a support bot where the informative word ("delayed," "cancelled," "refund") is often the less frequent one in the sentence.

Every word gets two vectors during training: an input (center) vector, from the rows of a matrix W of shape V×N, and an output (context) vector, from the columns of a matrix W′ of shape N×V, where N is the chosen embedding dimension. Training nudges these matrices so that center-context pairs that actually occur in the corpus get high dot products, and pairs that don't get low ones. After training, only W (the input vectors) is kept as "the" word embeddings — this is what a downstream cosine-similarity search or a translation system uses.

A fully worked skip-gram step

Take a miniature vocabulary of four words drawn from the IRCTC inbox — train, late, delayed, biryani (the last one is a stray food-delivery message that leaked into the support queue, included deliberately as an unrelated control word) — indexed 0–3, with embedding dimension N=2. Suppose training has already nudged the matrices to these (illustrative, not yet converged) values:

Input matrix W (row = center-word vector)      Output matrix W' (row = context-word vector)
train:    [ 0.5, -0.2]                         train:    [ 0.3,  0.1]
late:     [ 0.1,  0.3]                         late:     [ 0.2, -0.1]
delayed:  [ 0.4,  0.4]                         delayed:  [ 0.5,  0.2]
biryani:  [-0.6,  0.5]                         biryani:  [-0.2,  0.4]

Consider the training pair (center = train, true context = delayed) drawn from the sentence "train ... delayed" within the window. Look up the center vector: v_c = [0.5, −0.2]. For every word j in the vocabulary, score it by the dot product with its output vector, u_j = v_c · v′_j:

u_train   = 0.5(0.3)  + (-0.2)(0.1)  =  0.13
u_late    = 0.5(0.2)  + (-0.2)(-0.1) =  0.12
u_delayed = 0.5(0.5)  + (-0.2)(0.2)  =  0.21
u_biryani = 0.5(-0.2) + (-0.2)(0.4)  = -0.18

Turn the scores into a probability distribution over the vocabulary with softmax, p_j = e^u_j / Σ_k e^u_k. The exponentials are e^0.13=1.1388, e^0.12=1.1275, e^0.21=1.2337, e^−0.18=0.8353, summing to Z=4.3353. Dividing through:

p_train   = 0.2627   (26.3%)
p_late    = 0.2601   (26.0%)
p_delayed = 0.2846   (28.5%)  ← true context word
p_biryani = 0.1927   (19.3%)

Before any training the model is nearly guessing — 28.5% is barely above the 25% baseline four equiprobable words would give it. That's expected: the vectors above are close to their random initialization, and this is the very first step. The loss is cross-entropy against the true label, which for a one-hot target collapses to a single term: L = −log(p_delayed) = −log(0.2846) = 1.2568 nats.

The gradient step moves every context vector's contribution toward or away from v_c depending on the error e_j = p_j − y_j (where y is 1 for the true context word, 0 otherwise). The gradient with respect to the center vector is a weighted sum of the output vectors, weighted by these errors:

∂L/∂v_c = Σ_j e_j · v'_j
        = (0.2627)(0.3,0.1) + (0.2601)(0.2,-0.1) + (-0.7154)(0.5,0.2) + (0.1927)(-0.2,0.4)
        = (-0.2654, -0.0658)

With a learning rate η=0.1, the update is v_c ← v_c − η·∇: [0.5, −0.2] − 0.1·(−0.2654, −0.0658) = [0.5265, −0.1934]. Every quantity above — the four scores, the softmax, the loss, the gradient, and the updated vector — was independently computed and cross-checked in Python before being written here; you can reproduce the exact same numbers:

import numpy as np

W  = {"train": np.array([0.5,-0.2]), "late": np.array([0.1,0.3]),
      "delayed": np.array([0.4,0.4]), "biryani": np.array([-0.6,0.5])}
Wp = {"train": np.array([0.3,0.1]), "late": np.array([0.2,-0.1]),
      "delayed": np.array([0.5,0.2]), "biryani": np.array([-0.2,0.4])}

vc = W["train"]
scores = {w: vc @ v for w, v in Wp.items()}
exp = {w: np.exp(s) for w, s in scores.items()}
Z = sum(exp.values())
probs = {w: e / Z for w, e in exp.items()}
print(probs)               # {'train': 0.2627, 'late': 0.2601, 'delayed': 0.2846, 'biryani': 0.1927}

target = "delayed"
err = {w: probs[w] - (1 if w == target else 0) for w in probs}
grad = sum(err[w] * Wp[w] for w in Wp)
print(vc - 0.1 * grad)     # [0.5265 -0.1934]

Notice the direction of the move: v_c shifted toward larger dot products with delayed's output vector and smaller ones with the others. Repeat this over millions of (center, context) pairs sampled from real text, and words that keep landing in each other's context windows — "late" and "delayed," both near "train," "arrival," "status" — get pulled toward each other in the embedding space, while "biryani" gets pulled elsewhere. That is the entire mechanism: no dictionary of synonyms was ever consulted, no human labeled anything as similar. Similarity is an emergent property of shared context statistics.

Real vocabularies have hundreds of thousands of words, so computing the full softmax denominator on every training step (a sum over the entire vocabulary, exactly as done above) is too slow. Production Word2Vec uses negative sampling: instead of normalizing over every word, it turns each step into k (typically 5–20) independent binary classifications — "is this the real context word?" (yes) versus k randomly sampled "negative" words (no) — drawn from a noise distribution biased toward frequent words (raised to the power 0.75, empirically found to balance frequent and rare word sampling). This replaces an O(V) softmax with an O(k) computation per step, which is what makes training on billion-word corpora feasible at all.

GloVe: count the corpus once, factor the counts

Word2Vec never looks at the whole corpus at once — it walks through it with a sliding window, one local prediction at a time. GloVe (Pennington, Socher, Manning, 2014 — "Global Vectors") takes the opposite route: first build one global co-occurrence matrix X, where X_ij counts how often word j appears in word i's context window across the entire corpus, then factorize that matrix into low-dimensional vectors. The key insight GloVe's authors made is that raw co-occurrence probabilities are noisy, but ratios of co-occurrence probabilities are informative. Extending the IRCTC inbox example, suppose across the whole corpus delayed co-occurs with train 80 times out of 150 total co-occurrences, and biryani co-occurs with train only 5 times out of 120:

P(train | delayed) = 80/150  = 0.5333
P(train | biryani) =  5/120  = 0.0417
ratio = 0.5333 / 0.0417 = 12.8

A ratio far from 1 means "train" strongly discriminates between the two words — it's relevant to complaints (near "delayed") and irrelevant to food orders (near "biryani"). Now probe with a word that's mildly irrelevant to both, say sasta ("cheap"), co-occurring 3/150 times with delayed-context text and 4/120 times with biryani-context text:

P(sasta | delayed) = 3/150 = 0.0200
P(sasta | biryani) = 4/120 = 0.0333
ratio = 0.0200 / 0.0333 = 0.60

A ratio close to 1 (here 0.6, versus 12.8 above) signals that the probe word doesn't discriminate — it's roughly neutral between the two target words. GloVe's training objective is built directly on this signal: it fits vectors v_i, v_j and scalar biases b_i, b_j to minimize

J = Σ_{i,j} f(X_ij) · (v_i · v_j + b_i + b_j - log X_ij)^2

where f(X_ij) is a weighting function that caps the influence of extremely frequent pairs (like "the delayed") so a handful of stopword co-occurrences don't dominate the loss, while still giving zero-count pairs zero weight (log 0 is , so pairs that never co-occur are simply excluded rather than penalized). Because v_i · v_j is being fit to approximate log X_ij, and log turns ratios into differences, dot-product differences between vectors end up encoding exactly the log-ratios computed above — which is why GloVe vectors support the same "vector arithmetic" property Word2Vec became famous for (the classic result: vector("king") − vector("man") + vector("woman") lands near vector("queen"), because the offset direction captures the male→female relationship consistently across word pairs).

FastText: embeddings below the word level

Both Word2Vec and GloVe assign one opaque vector per whole word, learned only if that exact string appeared often enough during training. This is a real weakness for Indian languages, which are heavily agglutinative — Hindi verb roots take on many surface forms through suffixes (khelna "to play," khelta, khelte, khela, khelenge...), and a training corpus will never contain every inflected form of every root. A word missing from the training vocabulary — out-of-vocabulary, OOV — gets no embedding at all in Word2Vec or GloVe; the IRCTC bot would be stuck if a user typed a spelling or inflection it never saw during training.

FastText (Bojanowski, Grave, Joulin, Mikolov, 2016, from the same lab as Word2Vec) fixes this by representing each word as the sum of vectors for its constituent character n-grams, not just a single lookup. Wrap the word in boundary markers and slide an n-gram window (typically n=3 to 6) across it. For khelna with n=3, wrap it as <khelna> (8 characters including markers) and take every length-3 substring:

<khelna>  →  <kh, khe, hel, eln, lna, na>   (6 n-grams, since 8-3+1=6)

The word's final embedding is the sum (or average) of its own whole-word vector plus the vectors of all these n-gram pieces. Now compare with khelte, a different inflection of the same root:

<khelte>  →  <kh, khe, hel, elt, lte, te>   (6 n-grams)

Shared with khelna's n-grams: { <kh, khe, hel }

Three of the six n-grams — exactly the ones covering the shared root khel- — are identical between the two inflected forms, verified directly by generating both lists with the same sliding-window code. Even if khelte never appeared in training, FastText can still build a reasonable embedding for it at inference time from its n-gram vectors alone (all of which likely appeared inside other words during training), landing it near khelna in vector space because they share three of six subword pieces. This is the concrete mechanism, not a hand-wave — it is literally vector summation over overlapping substrings.

def ngrams(word, n=3):
    s = "<" + word + ">"
    return [s[i:i+n] for i in range(len(s) - n + 1)]

a, b = set(ngrams("khelna")), set(ngrams("khelte"))
print(a & b)   # {'hel', '<kh', 'khe'}

The misconception to unlearn

Students who've just met these models often assume word embeddings are context-aware — that the vector for "bank" would somehow differ between "river bank" and "bank account" because the model "understands" which one is meant. It does not. Word2Vec, GloVe, and FastText each assign exactly one fixed vector per word (or per word's n-grams), computed once during training and then looked up identically no matter what sentence it later appears in. The vector for "bank" is a single point in space that is, in effect, an average over every context "bank" ever appeared in across the training corpus — river banks, money banks, and everything else blended into one representation. This is precisely why these are called static embeddings, and it is the entire reason later architectures (ELMo, BERT, and the transformer-based models covered in later chapters) were built: to compute a different vector for the same word depending on its surrounding sentence. If your intuition says "the embedding model figures out which sense of the word I mean," that intuition belongs to a contextual model, not to Word2Vec, GloVe, or FastText.

Skip-gram, visualized

The diagram below traces the exact forward pass computed above: center word "train" enters as a one-hot vector, is looked up in W to get its dense vector, is scored against every column of W′, and the scores are turned into a probability distribution via softmax — the number 28.5% under "delayed" is the same number derived by hand.

Active recall

Attempt each question before reading the answer beneath it.

1. Why does one-hot encoding fail to capture that "late" and "delayed" are related words?
2. In skip-gram, is the network predicting the center word from the context, or the context from the center? Which architecture does the opposite?
3. Given center vector v_c = [1.0, 0.0] and two output vectors v′_A = [1.0, 0.0], v′_B = [0.0, 1.0], compute the two raw scores and say (without full softmax) which word is more probable.
4. Why does GloVe use the ratio of co-occurrence probabilities rather than the raw co-occurrence count itself as its core signal?
5. A user's message contains the word "khelrha" (a common informal misspelling of "khel raha," missing a vowel). Word2Vec has no vector for it. Would FastText? Why?
6. True or false: Word2Vec gives the word "bank" a different vector when it appears near "river" than when it appears near "account." Justify.

Answers

1. One-hot vectors are constructed purely from word identity (position in the vocabulary index), with no reference to usage. Every pair of distinct one-hot vectors has cosine similarity 0 regardless of meaning, so "late" and "delayed" are exactly as (un)related as "late" and "biryani." There is no mechanism in the representation itself for context statistics to influence distance.
2. Skip-gram predicts context words from the center word (one center word in, multiple context predictions out). CBOW (Continuous Bag of Words) does the reverse: it averages the surrounding context words and predicts the single center word from that average.
3. Score for A: v_c·v′_A = (1.0)(1.0) + (0.0)(0.0) = 1.0. Score for B: (1.0)(0.0) + (0.0)(1.0) = 0.0. Since softmax is monotonic in its input scores, the higher raw score (A, at 1.0) maps to the higher probability — word A is more probable, without needing to compute the actual softmax denominator.
4. Raw co-occurrence counts are dominated by corpus frequency effects unrelated to meaning — a very frequent word like "the" will have a huge raw count with almost everything, telling you nothing discriminative. A ratio of two conditional probabilities cancels out much of this frequency effect and isolates how differently two target words relate to a probe word: a ratio far from 1 means the probe word discriminates between them (as with "train" scoring 12.8 between "delayed" and "biryani" above), while a ratio near 1 means the probe word is neutral between them (as with "sasta" scoring 0.6).
5. Yes. FastText builds a word's vector from the sum of its character n-gram vectors, not from a whole-word lookup table. Even though "khelrha" itself never appeared during training, its n-grams (things like "khe," "elr," "lrh," "rha") likely appeared inside other words that did occur in training, so FastText can compose an embedding for it on the fly. Word2Vec and GloVe have no such fallback — an unseen word string simply has no row in their lookup matrix.
6. False. Word2Vec (like GloVe and FastText) produces exactly one static vector per word, fixed after training and looked up identically regardless of surrounding sentence. The single vector for "bank" is effectively a blend of every context "bank" appeared in during training. Producing a different vector depending on the sentence is the defining feature of contextual embedding models (ELMo, BERT, transformers), not of Word2Vec, GloVe, or FastText.

Think About It

Think about this: How would you explain word embeddings: word2vec, glove, and fasttext to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where word embeddings: word2vec, glove, and fasttext is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting word embeddings: word2vec, glove, and fasttext to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind word embeddings: word2vec, glove, and fasttext, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.