Word Embeddings: Meaning in Vectors

Why Does "Chappal" Also Show You Sandals?

Open the Flipkart or Amazon app and search for "chappal." Along with results for chappals, you will also see sandals, slippers, and flip-flops. Search "mobile" instead, and results for smartphones and cell phones show up too, even though none of those words share a single letter with each other. Search "sabzi" on a grocery app and you get vegetables. Somehow, the search engine "knows" that these completely different-looking words are talking about the same kind of thing.

A computer does not read the way you do. It has no memory of wearing chappals on a hot afternoon or watching a vendor weigh sabzi on a hand scale. All it can ever work with is numbers. Every neural network, from the simplest classifier to today's largest language models, only multiplies, adds, and compares numbers. So before any AI system can search, translate, or answer a question about text, it must first solve a more basic problem: how do you turn a word into numbers without losing its meaning? Word embeddings are the answer the field of natural language processing (NLP) settled on, and they are one of the most important ideas in modern AI. By the end of this chapter, you will know exactly why "chappal" and "sandals" end up sitting close together in a mathematical space, and you will be able to calculate that closeness yourself using nothing more than multiplication, addition, and square roots.

The Naive First Attempt: One-Hot Encoding

Suppose your entire vocabulary has just five words: aam, king, queen, man, and cricket. The most direct way to turn each word into a number is to assign it an ID: aam is word 0, king is word 1, queen is word 2, man is word 3, cricket is word 4. But feeding raw ID numbers into a model is risky, since the model might infer that word 4 is somehow "more" than word 1, which makes no sense for words.

The standard fix is called one-hot encoding: represent each word as a vector as long as the entire vocabulary, filled with zeros except for a single 1 marking that word's position.

vocabulary = ["aam", "king", "queen", "man", "cricket"]

def one_hot(word):
    vector = [0] * len(vocabulary)
    vector[vocabulary.index(word)] = 1
    return vector

print(one_hot("king"))    # [0, 1, 0, 0, 0]
print(one_hot("queen"))   # [0, 0, 1, 0, 0]
print(one_hot("cricket")) # [0, 0, 0, 0, 1]

This solves the "ordering" problem, but it introduces a bigger one: every word ends up exactly as different from every other word as it is from every other word. Measure the angle between the vectors for "king" and "queen" and you get precisely the same answer as the angle between "king" and "cricket." Both pairs are completely perpendicular, sharing no overlap at all. One-hot vectors carry no information about meaning; they are just numbered ID cards. And in a realistic vocabulary of, say, 50,000 words (a modest size for a language model), every one-hot vector would be 50,000 numbers long, with 49,999 zeros and a single 1, which is enormously wasteful. This kind of mostly-zero vector is called sparse. What language models actually need is the opposite: a short, dense vector, where every number carries information, positioned so that words with related meanings land near each other.

You Shall Know a Word by the Company It Keeps

The idea that makes dense, meaningful vectors possible is the distributional hypothesis. The linguist J.R. Firth summarised it in 1957 with a line NLP researchers still quote today: "You shall know a word by the company it keeps." Words that tend to appear in similar surrounding contexts tend to carry similar meanings.

Consider three versions of the same sentence:

• "The auto driver waited outside the station for a passenger."
• "The cab driver waited outside the station for a passenger."
• "The mango driver waited outside the station for a passenger."

"Auto" and "cab" both fit naturally; across thousands of real sentences, both words appear near neighbours like driver, fare, booked, and waiting. "Mango" almost never appears in this kind of context in real text. If you scanned millions of sentences and recorded, for every word, which other words tend to appear near it, you would find that "auto" and "cab" share a large overlap of neighbouring words, while "mango" shares almost none of that overlap with either one.

A word embedding turns this pattern of shared neighbours into coordinates. Every word in the vocabulary is assigned a vector, typically somewhere between 50 and 300 numbers, trained so that words used in similar contexts across a huge collection of text (called a corpus) end up with similar vectors. Nobody hand-designs what any individual number means. The values are learned automatically by a model that reads enormous amounts of text and gradually adjusts every word's numbers until words used in similar ways land near each other.

Word2Vec: Learning Vectors by Predicting Context

The algorithm that made word embeddings mainstream is Word2Vec, published in 2013 by Google researchers Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Word2Vec trains a simple neural network on a huge amount of text using one of two setups:

• CBOW (Continuous Bag of Words): show the model the surrounding words and ask it to predict the missing word in the middle. Given "the ___ driver waited outside," the model should learn to predict a word like "auto" or "cab."
• Skip-gram: the reverse of CBOW. Show the model one word and ask it to predict the words likely to surround it. Given "auto," the model should learn that "driver," "station," and "fare" are probable neighbours.

Concretely, training pairs come from sliding a context window across real text. Take the sentence "the auto driver waited outside the station" with a window of two words on either side, and set the target word to "driver": the window at that position covers "the" and "auto" (two words before) plus "waited" and "outside" (two words after), so Skip-gram would train the network on four pairs: (driver → the), (driver → auto), (driver → waited), (driver → outside). Slide the window one word to the right so the target becomes "waited," and the covered neighbours shift to "auto," "driver," "outside," "the." Repeating this slide across every sentence in a corpus of billions of words generates the millions of training pairs that ultimately shape the final vectors.

The network never actually cares about getting good at this fill-in-the-blank game for its own sake. That is just the training exercise. What matters is a side effect: to get good at predicting context, the network is forced to build an internal numeric representation of every word that captures how it is typically used. Those internal numbers, one dense vector per word, are the embeddings that get saved and reused elsewhere. It is a little like a cricket commentator who, after describing thousands of matches, develops an intuitive sense of which batters share a similar style, even though nobody ever explicitly labelled "aggressive opener" or "anchor" for them. The grouping emerges purely from exposure to enough examples. Google's original released Word2Vec vectors were trained on roughly 100 billion words from Google News, producing a 300-dimensional vector for each of 3 million words and phrases.

A related approach called GloVe (Global Vectors for Word Representation), released in 2014 by Stanford researchers Jeffrey Pennington, Richard Socher, and Christopher Manning, reaches similarly useful vectors through a different route: instead of sliding a context window across the text, it directly counts how often every pair of words appears near each other across the whole corpus, then compresses that giant co-occurrence table into dense vectors. Word2Vec and GloVe were later joined by fastText from Facebook AI Research, which released pre-trained embeddings for more than 150 languages, including Hindi, Bengali, Tamil, and Marathi, extending these techniques well beyond English text. In modern deep learning frameworks, the word-to-vector lookup is typically built directly into the network as its first trainable layer, commonly called an embedding layer, which stores one dense vector per vocabulary word and hands it forward to the rest of the model.

Measuring Closeness: Cosine Similarity

Once every word is a vector, we need a way to measure how close two vectors are in meaning. Plain distance is misleading here, because what matters is direction, not length. Two vectors pointing the same way represent similar meanings even if one is longer, which can simply reflect how often that word appeared during training. The standard tool is cosine similarity, the cosine of the angle between two vectors: cosine_similarity(A, B) = (A · B) / (|A| × |B|).

Here A · B is the dot product (multiply each matching pair of numbers and add up the results), and |A| and |B| are the magnitudes of the vectors, found the same way you would find the length of a line using the Pythagorean theorem: square each number, add them, and take the square root. The result always falls between −1 and 1: near 1 means the vectors point in almost the same direction, near 0 means unrelated, near −1 means opposite.

Let's trace this by hand using a small, simplified set of toy embeddings. Real Word2Vec or GloVe vectors have 50 to 300 dimensions with no individual meaning attached to any single number. The dimension count is itself a design trade-off: too few dimensions squeeze genuinely different words into sharing nearly the same direction, while too many make training slower and start fitting noise in the corpus rather than real patterns of meaning. Researchers who want to actually inspect real embeddings usually compress them down to 2 dimensions first, using a technique such as PCA or t-SNE, purely so clusters of related words become visible on a scatter plot. For this chapter, we can skip that compression step and simply start with a made-up 3-dimensional space small enough to compute by hand, where the three positions loosely stand for "royalty," "femininity," and "sport-relatedness":

king    = [0.90, 0.10, 0.05]
queen   = [0.90, 0.90, 0.05]
man     = [0.05, 0.10, 0.05]
woman   = [0.05, 0.90, 0.05]
cricket = [0.05, 0.05, 0.90]

Now compute the cosine similarity between king and queen step by step.

• Step 1. Dot product: (0.90 × 0.90) + (0.10 × 0.90) + (0.05 × 0.05) = 0.81 + 0.09 + 0.0025 = 0.9025.
• Step 2. Magnitude of king: √(0.90² + 0.10² + 0.05²) = √(0.81 + 0.01 + 0.0025) = √0.8225 ≈ 0.9069.
• Step 3. Magnitude of queen: √(0.90² + 0.90² + 0.05²) = √(0.81 + 0.81 + 0.0025) = √1.6225 ≈ 1.2738.
• Step 4. Divide: 0.9025 ÷ (0.9069 × 1.2738) = 0.9025 ÷ 1.1552 ≈ 0.78.

A cosine similarity of 0.78 out of a maximum of 1.0 tells us "king" and "queen" point in a very similar direction. They share the "royalty" quality even though they differ on "femininity." Now compare "king" to "cricket": the dot product is (0.90 × 0.05) + (0.10 × 0.05) + (0.05 × 0.90) = 0.045 + 0.005 + 0.045 = 0.095, the magnitude of cricket is √(0.05² + 0.05² + 0.90²) = √0.815 ≈ 0.9028, and dividing gives 0.095 ÷ (0.9069 × 0.9028) ≈ 0.12. Barely any relationship at all. That is exactly what you would expect between a word about monarchy and a word about sport.

Vector Arithmetic: king - man + woman ≈ queen

Because embeddings place words along consistent directions of meaning, you can do arithmetic with them. The most famous demonstration from the original Word2Vec research takes the vector for "king," subtracts the vector for "man," adds the vector for "woman," and finds that the resulting point lands almost exactly where "queen" sits. Subtracting "man" from "king" isolates the "royalty" direction by cancelling out "maleness"; adding "woman" then reapplies gender in the opposite direction. Using the toy table above:

• Step 1: king - man = [0.90 − 0.05, 0.10 − 0.10, 0.05 − 0.05] = [0.85, 0.00, 0.00]
• Step 2: add woman = [0.85 + 0.05, 0.00 + 0.90, 0.00 + 0.05] = [0.90, 0.90, 0.05]
• Step 3: compare to queen = [0.90, 0.90, 0.05], an exact match.

Real embeddings are messier than this hand-picked toy example. The arithmetic rarely lands exactly on another word's vector, so in practice you search the whole vocabulary for whichever real word has the highest cosine similarity to the result. But the underlying geometry genuinely shows up in embeddings trained on large corpora: nobody told the training algorithm that gender or royalty were meaningful concepts, and those directions emerged purely from the statistics of which words appear near which other words, across millions of sentences.

Verifying It in Code

A short Python program using NumPy confirms this arithmetic, since NumPy turns dot products and magnitudes into single function calls:

import numpy as np

# Toy embeddings for illustration only.
# Real Word2Vec/GloVe vectors have 50-300 dimensions and are
# learned automatically -- no dimension is hand-labelled like this.
embeddings = {
    "king":    np.array([0.90, 0.10, 0.05]),
    "queen":   np.array([0.90, 0.90, 0.05]),
    "man":     np.array([0.05, 0.10, 0.05]),
    "woman":   np.array([0.05, 0.90, 0.05]),
    "cricket": np.array([0.05, 0.05, 0.90]),
}

def cosine_similarity(a, b):
    dot_product = np.dot(a, b)
    magnitude = np.linalg.norm(a) * np.linalg.norm(b)
    return dot_product / magnitude

# king - man + woman should land near "queen"
result_vector = embeddings["king"] - embeddings["man"] + embeddings["woman"]

best_word, best_score = None, -1
for word, vector in embeddings.items():
    score = cosine_similarity(result_vector, vector)
    if score > best_score:
        best_word, best_score = word, score

print("king - man + woman is closest to:", best_word)
print("cosine similarity:", round(best_score, 4))
# king - man + woman is closest to: queen
# cosine similarity: 1.0

Running this prints queen with a cosine similarity of 1.0, a perfect match, since the toy numbers were designed to line up exactly. In an embedding trained on billions of real words, the printed similarity would typically land around 0.7 or 0.8 rather than a perfect 1.0, and the search would scan a real vocabulary of hundreds of thousands of words instead of five. But the loop, the cosine similarity function, and the underlying idea are identical to what production NLP systems actually run.

Where This Shows Up in Real Products

Once meaning is encoded as geometry, embeddings show up everywhere in modern software:

• Search engines, including e-commerce search on Flipkart, Amazon, and Myntra, match a query to products whose descriptions use different but related words; that is exactly why "chappal" surfaces sandals and "mobile" surfaces smartphones.
• Recommendation systems on Swiggy, Zomato, and YouTube apply the same idea to users and items, placing similar restaurants, dishes, or videos near each other in the same kind of vector space.
• Machine translation tools such as Google Translate rely on embeddings to map words and phrases from Hindi, Tamil, or Bengali into a shared numeric space alongside their English equivalents.
• Spam and content filters compare a new message's embedding against known spam messages to flag suspicious text even when the exact wording has never been seen before.
• Chatbots and voice assistants use embeddings as their very first processing step, converting your typed or transcribed words into vectors before any deeper reasoning happens.
• Predictive text keyboards rank the next word you are likely to type using this same embedding-style idea, which is part of why such keyboards can offer sensible suggestions even when a sentence mixes Hindi and English the way many Indians actually type.

Static word embeddings have one well-known limitation: a plain embedding assigns exactly one fixed vector to a word, no matter how it is used in a sentence. "Bank" gets the same vector whether it means a riverbank or a place to deposit money. Word2Vec and GloVe cannot tell the two apart. Hindi has the same problem with a word already sitting in our toy vocabulary: "aam" means mango in one sentence and "common" or "ordinary" in another, as in the well-known phrase "aam aadmi" (common man). A static embedding has no way to hold both senses at once. Fixing that requires context-sensitive models, which is what later transformer-based architectures were built for. Word embeddings are the essential first step that made that later progress possible, not the final word on the subject.

Back to the Search Bar

Return to the Flipkart search box from the start of this chapter. When you typed "chappal," the system did not scan for that literal string inside every product title. Behind the scenes, "chappal" was converted into a dense vector, trained so that it sits near "sandals" and "slippers" in a high-dimensional meaning space, not because a human programmer wrote a rule connecting them, but because those words appeared in similar surrounding contexts across millions of product listings, reviews, and search queries. The system then used exactly the cosine similarity calculation you just traced by hand to rank every product by how closely its description matches your query.

That is the whole idea of word embeddings in one sentence: turn each word into a list of numbers positioned so that similar meanings end up as similar geometry, then let ordinary arithmetic (dot products, magnitudes, angles) do the reasoning about language that once seemed to require genuine human understanding.

Think About It

Think about this: How would you explain word embeddings: meaning in vectors to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind word embeddings: meaning in vectors, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.