TF-IDF and BM25: Weighting Terms

Open a cricket news app on your phone and type "Kohli century" into the search bar. In under a second, the app has to sift through thousands of articles (match reports, team-of-the-week columns, ticket-booking pages, sponsor advertisements) and decide which handful to show you first. Somewhere behind that search bar, a piece of arithmetic is scoring every article in the archive against your two-word query and sorting the results by that score. This chapter is about exactly that arithmetic: how a computer decides which words in a document actually matter, and how it turns that decision into a ranked list.

The naive idea is to count how many times the query words appear in a document, and show the document with the highest count first. That idea turns out to be badly wrong, and understanding why is the fastest way to understand why TF-IDF and BM25 exist.

Part 1: Why Raw Word Counts Are Not Enough

Search engines work with a simplified view of text called the bag-of-words model: a document is treated as an unordered collection of words, with grammar and word order thrown away. Before any scoring happens, raw text is tokenized (split into individual words) and usually lowercased, a step called case folding, so that "Kohli" and "kohli" are treated as the same token. The raw headline Kohli's century guides India! might tokenize down to ["kohli", "century", "guides", "india"] after the possessive and the punctuation are stripped away.

Once every document is a bag of tokens, the simplest possible scoring rule is: for each query word, count how many times it occurs in the document, and add up the counts. This breaks in a specific way. Imagine two articles about the same match. One is a short flash that says "India win. Kohli's century seals it." The other is a long, generic match report that mentions the word "match" nine times while discussing pitch conditions, weather delays, and the toss, and mentions "Kohli" only once in passing. Under raw counting, the long generic report can easily out-score the short, precisely relevant one, simply because it is longer and repeats words more often. Raw frequency treats every word as equally informative, and that assumption is false: some words carry almost no information about what a document is "about," while others carry a lot.

Part 2: Term Frequency, Normalized

The first fix is to stop counting raw occurrences and instead ask what fraction of the document's words is the term being searched for. This is normalized term frequency (TF):

tf(t, d) = count(t, d) / |d|

where count(t, d) is how many times term t occurs in document d, and |d| is the total number of tokens in d. Dividing by document length stops long documents from winning purely by being long: a term appearing once in a six-word headline now scores higher, per word, than the same term appearing once in a six-hundred-word article. This is a real improvement, but it only solves half the problem: it says nothing about whether the term itself is common or rare across the whole collection of documents being searched, and that turns out to matter just as much.

Part 3: Inverse Document Frequency — Rewarding Rare Words

Consider a corpus, the full collection of documents being searched, made of cricket match reports. The word "match" will show up in almost every single one; it behaves like a stopword for this particular corpus, even though it is not a stopword in English generally. A search engine that weights "match" as heavily as a specific, unusual word like "century" is wasting the query's most useful signal. Inverse document frequency (IDF) fixes this by measuring how rare a term is across the whole corpus, not within one document. The idea, rewarding rare terms and punishing common ones, was introduced by the researcher Karen Spärck Jones in a widely cited 1972 paper, and it remains one of the most influential ideas in information retrieval.

First, define document frequency, df(t), as the number of documents in the corpus that contain the term t at least once — not how many times, just whether it appears. Then, for a corpus of N documents:

idf(t) = ln(N / df(t))

A term that appears in every single document has df(t) = N, so idf(t) = ln(1) = 0 — it contributes nothing to the score, which is exactly right, since a word present everywhere cannot help distinguish one document from another. A term that appears in only one document out of a thousand gets idf(t) = ln(1000) ≈ 6.9, a large weight, because its presence is a strong signal. Rare terms are rewarded; ubiquitous terms are punished down toward irrelevance. This is the missing half of the picture: TF measures local importance (how much this document talks about the term), and IDF measures global distinctiveness (how much the term, on its own, narrows down which document you must be looking at).

Part 4: TF-IDF — A Worked Example

Multiplying the two together gives the classic TF-IDF formula:

tf-idf(t, d) = tf(t, d) × idf(t)

To score a whole document against a multi-word query, sum the tf-idf value of each query term that appears in it. Let's trace this by hand over a tiny four-document corpus standing in for a cricket news app's archive, already tokenized and lowercased:

• D1: kohli hits century india win the match (7 tokens)
• D2: india win toss elect bowl first (6 tokens)
• D3: rohit kohli fall cheaply rain hits play (7 tokens)
• D4: kohli century guides india win series over australia (8 tokens)

Here N = 4. Suppose the query is "kohli century" — two terms. First, the document frequencies: kohli appears in D1, D3, and D4, so df(kohli) = 3. century appears only in D1 and D4, so df(century) = 2. Already this tells a story: even though "kohli" is the headline word of the query, it is actually the less discriminating of the two terms in this corpus, because it shows up in three of the four documents. "century" is rarer, and should be trusted more. Notice also that "india" and "win" each turn up in three of the four documents too — exactly as often as "kohli." A search engine that weighted every word equally would let these ordinary, topic-generic words count for just as much as the words that genuinely distinguish one article from another.

The IDF values confirm the story: idf(kohli) = ln(4/3) ≈ 0.2877, while idf(century) = ln(4/2) = ln(2) ≈ 0.6931 — almost two and a half times larger. Now compute tf for each document:

• D1 (7 tokens): kohli occurs once, so tf = 1/7 ≈ 0.1429, giving tf-idf = 0.1429 × 0.2877 ≈ 0.0411. century also occurs once: tf ≈ 0.1429, tf-idf ≈ 0.1429 × 0.6931 ≈ 0.0990. Document score: 0.0411 + 0.0990 = 0.1401.
• D2 (6 tokens): neither query term appears. Score: 0.0000.
• D3 (7 tokens): kohli occurs once (tf-idf ≈ 0.0411); century does not appear at all. Score: 0.0411.
• D4 (8 tokens): both terms occur once, but the document is longer, so each tf = 1/8 = 0.125. tf-idf(kohli) ≈ 0.125 × 0.2877 ≈ 0.0360, tf-idf(century) ≈ 0.125 × 0.6931 ≈ 0.0866. Score: 0.0360 + 0.0866 = 0.1226.

Ranked by score: D1 (0.1401) beats D4 (0.1226), which beats D3 (0.0411), which beats D2 (0.0000). This is a sensible ranking: the two articles that genuinely mention both a century and Kohli come out on top, and the article with neither term is correctly ranked last. Notice, though, that D1 edges out D4 partly because D1 is one word shorter; the same two matched terms count for slightly more when there are fewer competing words diluting the denominator. Hold onto that observation — it becomes important in Part 8.

Part 5: TF-IDF in Python

The entire calculation above is only a few lines of code:

import math

corpus = {
    "D1": "kohli hits century india win the match".split(),
    "D2": "india win toss elect bowl first".split(),
    "D3": "rohit kohli fall cheaply rain hits play".split(),
    "D4": "kohli century guides india win series over australia".split(),
}

N = len(corpus)

def doc_freq(term):
    return sum(1 for tokens in corpus.values() if term in tokens)

def tf(term, doc_id):
    tokens = corpus[doc_id]
    return tokens.count(term) / len(tokens)

def idf(term):
    return math.log(N / doc_freq(term))

query = ["kohli", "century"]

scores = {}
for doc_id in corpus:
    scores[doc_id] = sum(tf(t, doc_id) * idf(t) for t in query)

for doc_id, score in sorted(scores.items(), key=lambda x: -x[1]):
    print(f"{doc_id}: {score:.4f}")

Running this prints D1: 0.1401, D4: 0.1226, D3: 0.0411, D2: 0.0000 — matching the hand calculation exactly. Production libraries such as scikit-learn's TfidfVectorizer compute the same underlying idea slightly differently in the details: idf is usually smoothed to avoid ever dividing by zero, and the final document vectors are length-normalized. The core principle, multiplying local frequency by global rarity, is unchanged.

Part 6: Where TF-IDF Breaks Down

TF-IDF was the dominant relevance formula in information retrieval for decades, but it has two well-known weaknesses that matter a great deal at web scale.

The first is unbounded term-frequency growth. Under plain TF, a term occurring 20 times in a document scores twenty times higher than the same term occurring once — even though a document that says "election" 20 times is not twenty times "more about" elections than one that says it once. After the first few repetitions, each additional occurrence should matter less. Plain TF-IDF has no built-in mechanism for this diminishing return, which is exactly the kind of gap that keyword-stuffed pages exploit.

The second is crude length normalization. Dividing by document length is a blunt instrument — it treats a small nudge from 7 tokens to 8 tokens the same way it would treat a nudge from 700 to 800, and it offers no way to dial the strength of the correction up or down for a particular corpus. In Part 4, D1 outranked D4 partly because it was one token shorter, not purely because it was more relevant — an artifact of the formula, not a genuine signal.

BM25 was built specifically to fix both problems. The name is short for Best Matching 25 — often explained simply as the twenty-fifth weighting variant its inventors worked through. It emerged from the Okapi retrieval system built at City University London, developed by Stephen Robertson, Stephen Walker, and colleagues through a series of TREC (Text REtrieval Conference) evaluations in the 1990s. It has aged remarkably well: Lucene, the search library underneath Elasticsearch and Apache Solr, replaced classic TF-IDF with BM25 as its default relevance function in 2016, and BM25 remains the standard baseline that newer neural search and retrieval-augmented AI systems are measured against.

Part 7: The BM25 Formula

BM25 keeps the same IDF idea (rare terms matter more) but replaces the tf-idf multiplication with a more careful formula for each query term t and document d:

score(t, d) = idf(t) × [f(t,d) × (k1 + 1)] / [f(t,d) + k1 × (1 − b + b × |d| / avgdl)]

Here f(t,d) is the raw count of t in d (length is not folded into this term; it is handled separately), |d| is the document's length in tokens, and avgdl is the average document length across the whole corpus. Two tunable knobs control everything: k1, usually between 1.2 and 2.0 (commonly 1.5), controls how quickly extra occurrences of a term stop adding much score; and b, between 0 and 1 (commonly 0.75), controls how strongly document length is penalized.

The saturation effect is the heart of BM25. With k1 = 1.5 and length normalization set aside for a moment, watch how the fraction f × (k1+1) / (f + k1) grows as raw frequency f increases:

• f = 1: fraction = 1.0000
• f = 2: fraction ≈ 1.4286
• f = 4: fraction ≈ 1.8182
• f = 10: fraction ≈ 2.1739
• f = 20: fraction ≈ 2.3256

Even at 20 occurrences, the fraction has not reached 2.5 — it never will, no matter how large f gets, because the expression is mathematically bounded above by k1 + 1 = 2.5. Going from 1 occurrence to 2 nearly doubles the score contribution; going from 10 to 20 barely moves it. That is exactly the diminishing-returns behaviour that plain TF-IDF lacked, and here it is controlled by a single readable parameter instead of being an accident of the formula.

The length term, 1 − b + b × |d| / avgdl, is BM25's more flexible length correction. When b = 1, this reduces to |d| / avgdl — full proportional-length normalization, in the same spirit as plain TF-IDF's division by document length. When b = 0, the term collapses to exactly 1 for every document, switching length normalization off entirely. The default, b = 0.75, sits deliberately between the two extremes.

One more subtlety explains why real systems compute IDF slightly differently from the plain ln(N/df(t)) used above. The original probabilistic form of BM25's idf term is ln((N − df(t) + 0.5) / (df(t) + 0.5)). For a term that appears in more than half the documents in the corpus, this expression can turn negative — a term the query would trust less than nothing, which does not make sense as a relevance weight. In our own mini-corpus, kohli appears in 3 of 4 documents, and plugging in the numbers gives ln(1.5/3.5) ≈ −0.8473, so the problem is not just theoretical. Real engines such as Lucene fix this by adding 1 inside the logarithm:

idf(t) = ln(1 + (N − df(t) + 0.5) / (df(t) + 0.5))

which is always positive, and is the version used in the worked example that follows.

Part 8: BM25 — The Same Query, Recomputed

Return to the four-document corpus and the query "kohli century," now scored with BM25 using k1 = 1.5 and b = 0.75. Document lengths were 7, 6, 7, and 8 tokens for D1 through D4, so avgdl = (7+6+7+8)/4 = 7.0.

Using the corrected idf: idf(kohli) = ln(1 + 1.5/3.5) = ln(1.4286) ≈ 0.3567, and idf(century) = ln(1 + 2.5/2.5) = ln(2) ≈ 0.6931.

Each document's length-normalization factor is L(d) = 1 − 0.75 + 0.75 × |d|/7:

• D1 (7 tokens, exactly the average length): L = 1.0000. Both kohli and century occur once, and because f = 1 and L = 1, the saturation fraction simplifies to exactly 1×2.5 / (1 + 1.5×1) = 2.5/2.5 = 1.0 for each term — so each term's contribution is simply its idf. Score: 0.3567 + 0.6931 = 1.0498.
• D2 (6 tokens): neither term present. Score: 0.0000.
• D3 (7 tokens, average length): the same simplification as D1 applies to kohli (contribution 0.3567); century is absent and contributes 0. Score: 0.3567.
• D4 (8 tokens, longer than average): L = 1 − 0.75 + 0.75×8/7 ≈ 1.1071. For each term, the fraction is 1×2.5 / (1 + 1.5×1.1071) = 2.5/2.6607 ≈ 0.9396. So kohli contributes 0.3567 × 0.9396 ≈ 0.3351, and century contributes 0.6931 × 0.9396 ≈ 0.6513. Score: 0.3351 + 0.6513 = 0.9864.

The ranking order is unchanged: D1 (1.0498) still edges out D4 (0.9864), ahead of D3 (0.3567) and D2 (0.0000). But look at the margins. Under plain TF-IDF, D1 scored about 14.3% higher than D4 (0.1401 vs 0.1226). Under BM25, D1 is only about 6.4% higher than D4 (1.0498 vs 0.9864). BM25's smoother length correction stopped D1's one-word length advantage from swinging the result nearly as much — the ranking is now driven far more by the fact that both articles genuinely mention both query terms, and much less by which one happened to be marginally shorter. That narrowing is not a coincidence; it is exactly what the k1/b saturation-and-normalization machinery was designed to do.

Part 9: BM25 in Python

Continuing directly from the TF-IDF code in Part 5 — reusing corpus, N, doc_freq, and query:

k1, b = 1.5, 0.75
doc_lens = {d: len(toks) for d, toks in corpus.items()}
avgdl = sum(doc_lens.values()) / N

def idf_bm25(term):
    n = doc_freq(term)
    return math.log(1 + (N - n + 0.5) / (n + 0.5))

def bm25_score(doc_id, query):
    total = 0.0
    L = 1 - b + b * (doc_lens[doc_id] / avgdl)
    for t in query:
        f = corpus[doc_id].count(t)
        if f == 0:
            continue
        num = f * (k1 + 1)
        den = f + k1 * L
        total += idf_bm25(t) * (num / den)
    return total

for doc_id in corpus:
    print(f"{doc_id}: {bm25_score(doc_id, query):.4f}")

This prints D1: 1.0498, D2: 0.0000, D3: 0.3567, D4: 0.9864 — matching the hand-worked numbers in Part 8 exactly, including the compressed margin between D1 and D4.

Part 10: Back to the Search Bar

Every real search bar used daily (a news app, an e-commerce product search, the search box inside a documentation site) runs some version of this exact arithmetic, usually BM25, on a corpus with millions of documents instead of four. The short functions in Parts 5 and 9 are not toy simplifications of what Elasticsearch or Apache Solr do internally; they are, almost verbatim, the same formula those systems run, just applied across a far larger vocabulary and index. Even as modern AI search systems layer neural embeddings and language-model re-ranking on top, BM25 typically still does the first pass — quickly narrowing millions of documents down to a shortlist that a slower, smarter model can then re-rank. More than three decades after it was designed, it remains the default relevance function that newer techniques are measured against.

One thing neither formula fixes is worth remembering: bag-of-words models cannot tell word senses apart. In this chapter's own corpus, hits appears in both D1 ("Kohli hits century") and D3 ("rain hits play") — the same token used for a batting stroke and for weather stopping play. TF-IDF and BM25 see only the identical token hits in both places, with no way to know these are two unrelated meanings. That gap is exactly what motivates the embedding-based and neural ranking methods used alongside BM25 in modern search and retrieval-augmented AI systems.

So the next time a search bar puts the right article first, the one that actually reports Kohli's century and not just any match report that happens to say "match" a dozen times, the shape of the arithmetic underneath is no longer a mystery: term frequency asking how much a document talks about a word, inverse document frequency asking how rare and trustworthy that word is across the whole collection, and, in BM25's case, two extra knobs quietly making sure a document does not win or lose just because it happened to be one word longer or shorter.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where tf-idf and bm25: weighting terms is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting tf-idf and bm25: weighting terms to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind tf-idf and bm25: weighting terms, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.