Type a factual question into a large language model with no access to outside documents and it will answer instantly, fluently, and sometimes wrong. Ask a closed-book model "Which ISRO orbiter first studied the Martian atmosphere, and in what year did it launch?" and it may confidently name the right mission with the wrong year, or blend two missions into one, because it is not looking anything up — it is generating the statistically most likely continuation of your prompt from parameters frozen at training time. This is the central failure mode that retrieval-based question answering exists to fix: instead of asking a model to recall facts from memory, you make it read them off a page you hand it at the moment of the question. Production systems that answer real user queries about live, fast-changing, or authoritative information — a train-status assistant on a booking site, a bank's policy-document chatbot, a search engine's answer box — are built this way for exactly this reason. The model that ultimately writes or extracts the answer is only as good as the passage it was given, so the harder engineering problem, and the one this chapter is about, is finding the right passage out of millions in the first place.
This chapter builds the two-stage retriever–reader architecture that underlies almost every modern QA and retrieval-augmented generation (RAG) system, works a complete numeric example by hand and in code, and then deliberately breaks that example to expose a mechanism-level misconception about what "search" actually means to a machine.
Two questions, two components
Question answering research splits the task along two independent axes, and it is worth being precise about both because they get conflated constantly.
The first axis is where the answer comes from: closed-book systems answer purely from parameters learned during training (fast, cheap, but frozen and prone to hallucination); open-book systems are given a document collection at query time and must ground their answer in it. Everything in this chapter is open-book.
The second axis is how the answer is produced once the right text has been found: extractive QA selects a contiguous span of the source text verbatim as the answer (this is what the SQuAD benchmark trains models to do); abstractive or generative QA produces a new sentence that may paraphrase, summarize, or combine several retrieved passages. A RAG pipeline is simply generative QA where the "context" fed to the generator was assembled by a retriever rather than provided by the user.
Both extractive and generative QA share an identical first stage. Before any answer can be extracted or generated, something has to decide which handful of documents, out of a potentially enormous collection, are worth reading at all. That first stage is retrieval, and it is a search problem, not a language-understanding problem — which is precisely where the common misconception in this chapter comes from.
The retriever–reader pipeline
An open-book QA system is built and run in two distinct phases.
Offline, once, before any question arrives: every document in the corpus is tokenized and turned into a fixed-size vector (a bag-of-words weighting like TF–IDF/BM25, or a dense embedding from a trained encoder), and those vectors are loaded into an index structure built for fast similarity lookup — an inverted index for sparse vectors, an approximate-nearest-neighbour graph for dense ones.
Online, for every incoming question: the retriever turns the question into a vector using the same scheme, scores it against every indexed document (in practice, only a small candidate set touched via the index), and returns the top-k highest-scoring documents. The reader then receives only those k passages — not the whole corpus — and either extracts a span from them (start/end token classification) or generates a free-text answer conditioned on them.
The reason this two-stage split exists, rather than one model reading the entire corpus per question, is computational. A transformer reader scales at best quadratically with input length; running it over an entire multi-million-document corpus for every single question is not tractable at any real latency budget. The retriever's whole job is to cheaply throw away the documents that obviously cannot contain the answer, so the expensive reader only ever looks at a handful of promising candidates. This is the same divide-and-conquer instinct behind an index in a database or a hash table in front of a linear scan: do a cheap, coarse filter first, and spend expensive, precise computation only on what survives it.
The diagram below traces this pipeline end to end using the exact corpus, query, and scores worked out in the next section.
Worked example: ranking documents with TF–IDF and cosine similarity
Take a tiny four-document corpus and one query, small enough to compute by hand and verify in code.
D1 (Chandrayaan-3): chandrayaan moon south pole landing twenty23
D2 (Aditya-L1): aditya sun lagrange point solar observatory
D3 (Mangalyaan): mangalyaan mars orbiter atmosphere mission twenty13
D4 (PSLV): pslv rocket satellites launch mission orbit
Query: "mission mars atmosphere"Each document has six distinct tokens with no repeats, so every term's raw term frequency (tf) within its own document is exactly 1. The corpus has N = 4 documents. Inverse document frequency uses idf(t) = log₂(N / df(t)), where df(t) is the number of documents containing term t. Every term in this corpus is unique to one document except "mission", which appears in both D3 and D4:
df(mission) = 2 → idf(mission) = log₂(4/2) = log₂(2) = 1.0
df(mars) = 1 → idf(mars) = log₂(4/1) = log₂(4) = 2.0
df(atmosphere) = 1 → idf(atmosphere) = log₂(4) = 2.0
(every other term also has df = 1, so idf = 2.0 for all of them)Only three terms in the query actually occur anywhere in the corpus — "mission", "mars", "atmosphere" — so the query vector has three nonzero components: q = {mission: 1×1.0, mars: 1×2.0, atmosphere: 1×2.0} = {1.0, 2.0, 2.0}, giving ‖q‖ = √(1²+2²+2²) = √9 = 3.0.
Here is the retriever, implemented directly from that definition and run against the corpus above:
import math
from collections import Counter
docs = {
"D1": ["chandrayaan","moon","south","pole","landing","twenty23"],
"D2": ["aditya","sun","lagrange","point","solar","observatory"],
"D3": ["mangalyaan","mars","orbiter","atmosphere","mission","twenty13"],
"D4": ["pslv","rocket","satellites","launch","mission","orbit"],
}
N = len(docs)
df = Counter()
for toks in docs.values():
for t in set(toks):
df[t] += 1
def idf(t):
return math.log2(N / df[t])
def vec(tokens):
tf = Counter(tokens)
return {t: tf[t] * idf(t) for t in tf}
def dot(v1, v2):
return sum(v1.get(t, 0) * v2.get(t, 0) for t in set(v1) | set(v2))
def norm(v):
return math.sqrt(sum(x * x for x in v.values()))
qvec = vec(["mission", "mars", "atmosphere"])
for name, toks in docs.items():
dv = vec(toks)
cos = dot(qvec, dv) / (norm(qvec) * norm(dv))
print(f"{name}: cos = {cos:.4f}")Running this prints exactly:
D1: cos = 0.0000
D2: cos = 0.0000
D3: cos = 0.6547
D4: cos = 0.0727Walk the arithmetic behind the two nonzero rows to see why. D3's full vector is {mangalyaan:2.0, mars:2.0, orbiter:2.0, atmosphere:2.0, mission:1.0, twenty13:2.0}, with magnitude ‖D3‖ = √(2²×5 + 1²) = √21 ≈ 4.5826. Only "mission", "mars", and "atmosphere" overlap with the query, contributing dot product (1.0×1.0) + (2.0×2.0) + (2.0×2.0) = 1+4+4 = 9, so cos(q,D3) = 9 / (3.0 × 4.5826) = 0.6547. D4's vector {pslv:2.0, rocket:2.0, satellites:2.0, launch:2.0, mission:1.0, orbit:2.0} has the same magnitude √21 ≈ 4.5826 by the same shape of numbers, but shares only "mission" with the query, giving dot product 1.0×1.0 = 1 and cos(q,D4) = 1 / (3.0 × 4.5826) = 0.0727. D1 and D2 share zero vocabulary with the query, so every term of their dot product is a product where one factor is always zero — the sum is exactly 0 regardless of how large the documents' own magnitudes are.
The retriever therefore ranks D3 far ahead of everything else (0.6547 against a runner-up of 0.0727, roughly nine times higher) and passes it — and only it, if k=1 — to the reader.
From the retrieved passage to an exact answer: the reader stage
Retrieval only finds the right haystack; it does not find the needle. Given the natural-language form of D3 — "ISRO's Mangalyaan orbiter, launched in 2013, was built to study the Martian atmosphere and became Asia's first Mars mission." — and the question "What did the Mangalyaan orbiter study?", an extractive reader must select the substring that actually answers the question, not just any sentence containing overlapping words.
A real trained reader (fine-tuned BERT/RoBERTa-style, as in the original SQuAD systems) does this by running the concatenated question+passage through a transformer encoder and attaching two classification heads on top of every passage token: one predicts the probability that this token is the start of the answer span, the other that it is the end. The predicted answer is the span [i, j], i ≤ j, that maximizes P(start=i) × P(end=j), found by a simple two-pointer search over the (usually short) passage. The two probability distributions are what the model has learned — from thousands of question/answer-span training pairs — to place high mass on tokens that plausibly begin or end an answer to this kind of question, using the full contextual representation of every word, not surface overlap.
That learned start/end scoring cannot be reproduced by hand, so here is a small deterministic stand-in that captures the shape of the idea — anchor on the passage word that also appears in the question, then extract the phrase that immediately follows it up to a clause boundary — clearly a simplification of the real mechanism above, not a claim about how trained transformer readers work internally:
def extract_answer(question, passage, boundary_words=None):
if boundary_words is None:
boundary_words = {"after","before","during","when",
"while","since","and","because"}
stop = {"the","a","an","did","what","which","is","are",
"was","were","of","its","to"}
def clean(w):
return w.strip(".,'").lower()
q_words = {clean(w) for w in question.split()} - stop
p_words = passage.split()
anchor_idx = None
for idx, w in enumerate(p_words):
cw = clean(w)
if cw in q_words and cw not in stop:
anchor_idx = idx # keep the LAST match: the verb,
# not the subject repeated from Q
if anchor_idx is None:
return None
start = anchor_idx + 1
end = start
while end < len(p_words) and clean(p_words[end]) not in boundary_words:
end += 1
return " ".join(p_words[start:end])
passage = ("ISRO's Mangalyaan orbiter, launched in 2013, was built to study "
"the Martian atmosphere and became Asia's first Mars mission.")
question = "What did the Mangalyaan orbiter study"
print(extract_answer(question, passage))Trace it: after removing stopwords, the question's content words are {mangalyaan, orbiter, study}. Scanning the passage, "Mangalyaan" and "orbiter" both match early, but the loop deliberately keeps the last match rather than the first — that turns out to be "study" at token index 9, the verb the question is actually asking about, rather than the subject the question already told us. Extraction then walks forward from index 10 ("the"), appending "Martian" and "atmosphere", and stops the instant it reaches "and" — which is in boundary_words — before it can swallow the unrelated clause "became Asia's first Mars mission" that follows. The program prints exactly the Martian atmosphere, matching the SVG diagram above.
This anchor-and-boundary trick only works because the question happens to reuse the passage's exact words. Real readers do not have that luxury and do not need it — the transformer's contextual embeddings let it recognize a paraphrased question ("What gas layer did Mangalyaan investigate?") pointing at the same span even with zero shared vocabulary, which is exactly the capability the toy retriever above is shown lacking in the next section.
The misconception: similarity score is not understanding
A very natural but wrong assumption is that when a retrieval system returns "the most relevant document," it has understood the question the way a person would and searched for documents that mean the same thing. TF–IDF and BM25 do no such thing: they count shared surface tokens, weighted by rarity. They cannot tell that two different strings refer to the same concept.
Rerun the exact retriever above with the question rephrased as "Which mission explored the Red Planet?" instead of using the word "mars". After stopword removal, the only word from this question that appears anywhere in the four-document vocabulary is "mission" — "red" and "planet" never occur in any indexed document, so they contribute nothing. The query vector collapses to a single nonzero component, q' = {mission: 1.0}, with ‖q'‖ = 1.0. Both D3 and D4 contain "mission" exactly once with the same idf = 1.0 computed earlier, so dot(q', D3) = dot(q', D4) = 1.0, and both magnitudes were already computed above as √21 ≈ 4.5826. That gives cos(q', D3) = cos(q', D4) = 1.0 / 4.5826 ≈ 0.2182 — an exact tie. The system that correctly ranked Mangalyaan first when the question said "mars" now cannot distinguish the Mars orbiter from the unrelated PSLV rocket document at all, purely because "Red Planet" is a different string from "mars".
This is not a bug in the arithmetic — the arithmetic is doing exactly what cosine similarity over TF–IDF vectors is defined to do. The fix is a different retriever, not a different formula. A dense retriever replaces the hand-built TF–IDF vector with the output of a neural text encoder (a sentence-embedding model trained so that semantically related passages land near each other in a continuous vector space), so "Red Planet" and "Mars" map to nearby points even though they share no characters. Cosine similarity is still the comparison — the geometry didn't change, only where the encoder places the points. Production systems increasingly combine both: a fast sparse method like BM25 (TF–IDF's more carefully calibrated successor, which additionally saturates the effect of very high term frequency and normalizes for document length via score = Σt idf(t) · f(t,d)(k₁+1) / (f(t,d) + k₁(1−b+b·|d|/avgdl)), typically k₁≈1.2–2.0, b=0.75) catches exact keyword and rare-term matches that embeddings sometimes miss, while a dense retriever catches paraphrase and synonymy that sparse methods miss by construction; the two ranked lists are merged (commonly by reciprocal rank fusion) into one candidate set for the reader. At corpus scale, exhaustively comparing a dense query vector against millions of document embeddings is itself too slow, so production dense retrievers index embeddings in an approximate-nearest-neighbour graph structure such as HNSW (hierarchical navigable small world), which answers a similarity query by hopping through a small-world graph of embeddings rather than checking every point, trading a small amount of recall for a large, roughly logarithmic-scale drop in search cost compared to brute-force comparison against every stored vector.
Active recall
Attempt each question before reading its answer.
1. In the worked example, "mission" appears in both D3 and D4 with idf(mission) = 1.0, not 0. Under what condition would a term's idf become exactly 0, and why would that matter for retrieval?
2. Why did D1 and D2 both score cos = 0.0000 against the query, even though their own document vectors have nonzero magnitude?
3. If this corpus had 10,000 documents instead of 4, why would scoring every document against every query with a linear scan become impractical, and what data structure fixes it?
4. Explain, in terms of the actual vectors, why TF–IDF retrieval tied D3 and D4 when the query said "Red Planet" instead of "mars", and name the retrieval approach that would break the tie correctly.
5. A team increases k (the number of passages handed to the reader) from 1 to 20, hoping more context means a better chance of finding the answer, and the reader's accuracy actually drops. Why is that plausible?
Answers.
1. idf(t) = log₂(N/df(t)) is exactly 0 when df(t) = N — the term appears in literally every document in the corpus. In that case its tf·idf weight is 0 in every document vector, so it can never contribute to any dot product and never affects ranking. This is the formal reason common stopwords ("the", "is", "a") are usually filtered before indexing: even without an explicit stopword list, TF–IDF automatically zeroes out any term so common it appears everywhere. In this corpus "mission" appears in only 2 of 4 documents, not all 4, so its idf is a middling 1.0, not 0 — reduced weight because it's not distinctive, but not eliminated.
2. Cosine similarity's numerator is the dot product, and the dot product is a sum over terms of q_t × d_t. D1 and D2 share zero vocabulary with the query {mission, mars, atmosphere}, so for every term either q_t = 0 or d_t = 0 (usually both), making every summand 0 and the total dot product exactly 0. Dividing 0 by any positive magnitude still gives 0 — the size of D1 or D2's own vector is irrelevant once the numerator is zero.
3. A brute-force scan costs O(N·L) per query, where L is average document length — at 10,000 documents this means 10,000 dot products for a single question, and it gets worse linearly as the corpus grows, which does not scale to web-sized or even enterprise-sized collections. An inverted index — a hash map from each vocabulary term to a postings list of (doc_id, tf) pairs — fixes this: at query time you only fetch the postings lists for the handful of terms actually in the query, so cost scales with how many documents contain those specific rare terms, not with the total corpus size.
4. With the query "Which mission explored the Red Planet?", the only in-vocabulary content word is "mission" — "red" and "planet" occur in no indexed document, contributing nothing to the query vector. Both D3 and D4 contain "mission" once with the same idf = 1.0 and have equal magnitude √21, so cos(q,D3) = cos(q,D4) = 1/√21 ≈ 0.2182 exactly — a tie that gives the retriever no basis to prefer the correct Mars document. Dense retrieval (embeddings from a trained neural encoder, compared by the same cosine similarity) fixes this because a well-trained encoder places "Red Planet" close to "Mars" in vector space regardless of shared characters.
5. Every extra passage handed to the reader is either signal or noise. Reader models make a relative choice among candidate spans (or, for generative readers, must actively decide what to ignore); as low-relevance passages pile up, they act as distractors that can outscore the correct-but-buried span, dilute the model's effective attention across mostly irrelevant text, and increase the odds the final answer is pulled from the wrong passage. k is therefore a tuned hyperparameter trading recall (the right passage is probably somewhere in the top-k) against precision and noise (the reader has to find it among more distractors) — bigger is not automatically better.
Think About It
Think about this: How would you explain question answering and retrieval systems to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind question answering and retrieval systems, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.