A section number that no longer exists
In July 2024, India's Bharatiya Nyaya Sanhita (BNS) replaced the 164-year-old Indian Penal Code. Overnight, "Section 420" — the code number every Bollywood film used as shorthand for fraud — stopped being the law. Criminal breach of trust, previously IPC Section 405, now lives under BNS Section 316. Any large language model trained before mid-2024 has IPC section numbers baked permanently into its weights. Ask it "which section punishes criminal breach of trust" and it will answer fluently, confidently, and wrong — not because it is malfunctioning, but because a transformer's knowledge is frozen at training time and it has no mechanism to know that the ground shifted under it. A legal-tech startup building a citation assistant for junior lawyers cannot ship that. The fix is not a bigger model or a longer prompt reminding it of the date — it is giving the model a way to look something up before it answers, and to point at the exact passage its answer came from.
That is what retrieval-augmented generation is for. The companion capstone chapter in this course walks through building a full production RAG pipeline — chunking documents, choosing a vector database, wiring up an orchestration layer. This chapter stays underneath all of that and asks three narrower, harder questions: how does a retriever actually decide that one passage is relevant and another is not; how does that decision happen in milliseconds across millions of documents; and what did "retrieval-augmented generation" mean mathematically before it became shorthand for "paste some search results into the prompt"? Answering these requires the linear algebra and transformer internals this course built up across Grade 11 and this year's mathematics track.
Dense retrieval: two encoders sharing one vector space
A classical search engine like BM25 ranks documents by weighted word overlap — it counts how often query terms appear, adjusted for term rarity and document length. It has no idea that "criminal breach of trust" and "dishonest misappropriation of entrusted property" describe the same legal concept in different vocabulary. Dense Passage Retrieval (Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering," EMNLP 2020) replaced word counting with geometry. DPR trains two separate BERT encoders — a query encoder E_Q and a passage encoder E_P — that map text into the same 768-dimensional vector space, positioned so that a question and its correct answer passage sit close together, regardless of shared vocabulary.
Training uses a contrastive objective. For a batch of N (question, correct-passage) pairs, every other passage in the batch serves as a free "in-batch negative" — text that is almost certainly irrelevant to a different question. The loss for pair i is:
L_i = -log( exp(sim(q_i, p_i+)) / Σ_j exp(sim(q_i, p_j)) )
where sim(q, p) = E_Q(q) · E_P(p) (dot product in the shared vector space)
This is the same softmax-over-similarities shape used elsewhere in this course's attention-mechanism material — the model is pushed to make the correct passage's score dominate a softmax computed over every passage in the batch. Using in-batch negatives instead of hand-labeled negative examples is what makes DPR trainable at scale: a batch of 128 questions supplies 127 negatives per question for free, no extra annotation required.
Worked example: why the wrong-vocabulary passage wins
Suppose a legal RAG system has already trained encoders and reduced the query and three candidate passages to (illustrative, small) 4-dimensional vectors for hand-tracing. The query is "which section punishes criminal breach of trust":
q = [0.80, 0.60, 0.10, 0.00] # query vector
Passage A — old IPC §405 text, exact phrase match, now superseded
a = [0.90, 0.10, 0.05, 0.00]
Passage B — current BNS §316 text, "dishonest misappropriation of
entrusted property," no shared wording with the query
b = [0.70, 0.65, 0.20, 0.10]
Passage C — unrelated BNS chapter on cyber offences
c = [0.05, 0.02, 0.90, 0.30]
Retrieval scores are the raw dot products sim(q,x) = q·x:
sim(q,A) = 0.8(0.90) + 0.6(0.10) + 0.1(0.05) + 0(0.00) = 0.785
sim(q,B) = 0.8(0.70) + 0.6(0.65) + 0.1(0.20) + 0(0.10) = 0.970
sim(q,C) = 0.8(0.05) + 0.6(0.02) + 0.1(0.90) + 0(0.30) = 0.142
Passage B wins despite sharing not one keyword with the query, because the encoder was trained to place semantically equivalent legal concepts near each other, not to reward string overlap. Passage A, the lexical trap, scores second. This is precisely the failure mode a BM25 retriever would get backwards: it would rank A first on word overlap alone and hand the model an obsolete section number with high apparent confidence.
Converting these three scores into a probability distribution over the top-k candidates — this softmax is the retriever's own probability p_η(z|x), used later in this chapter — gives:
import math
q = [0.80, 0.60, 0.10, 0.00]
a = [0.90, 0.10, 0.05, 0.00]
b = [0.70, 0.65, 0.20, 0.10]
c = [0.05, 0.02, 0.90, 0.30]
dot = lambda x, y: sum(xi*yi for xi, yi in zip(x, y))
scores = {'A': dot(q,a), 'B': dot(q,b), 'C': dot(q,c)}
exps = {k: math.exp(v) for k, v in scores.items()}
Z = sum(exps.values())
p_eta = {k: v/Z for k, v in exps.items()}
print({k: round(v, 4) for k, v in p_eta.items()})
# {'A': 0.3664, 'B': 0.4409, 'C': 0.1926}
Every variable used (q, a, b, c) is defined above the computation, and running this produces exactly {'A': 0.3664, 'B': 0.4409, 'C': 0.1926}, matching the hand-computed dot products.
Scaling the search: from brute force to navigable graphs
Computing a dot product against every document is fine for three passages. A real legal corpus — BNS, BNSS, BSA, decades of Supreme Court and High Court judgments — runs into the tens of millions of passages. Brute-force search against 10,000,000 passages at 768 dimensions means 10,000,000 dot products, each doing 768 multiply-adds, per query: roughly 7.7 billion floating-point operations before an answer can even begin generating. Even at server-grade throughput this is a latency and cost problem when thousands of users are querying concurrently.
Production vector indexes solve this with Hierarchical Navigable Small World graphs — HNSW (Malkov & Yashunin, "Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs," IEEE TPAMI, 2018). Instead of one flat set of 10 million points, HNSW builds several stacked graph layers. The top layer is sparse, containing a small random subset of points connected by long-range edges — like an expressway. Each layer below is denser, until the bottom layer contains every single point, connected only to its near neighbors — like local streets. A query starts at a fixed entry point on the top layer and greedily walks to whichever neighbor is closer to the query vector, repeating until no neighbor improves the distance; it then drops down one layer at the same point and repeats the greedy walk with finer-grained neighbors. By the time it reaches the bottom layer, it is already extremely close to the true nearest neighbors and only needs to explore a small local neighborhood to finish.
Two distinct quantities are at play here, and it is easy to conflate them. The graph itself has roughly log_M(N) layers, where M is the maximum node degree (typically M ≈ 16 in standard configurations, per Malkov & Yashunin) — for N = 10,000,000, that is log_16(10,000,000) ≈ 5.8, so about 6 layers, matching the compact multi-layer structure shown in the diagram below. The O(log N) figure usually quoted for HNSW is a separate thing: the expected total number of comparisons/hops the greedy walk performs while descending through those layers, which for N = 10,000,000 works out to on the order of a few dozen — several orders of magnitude fewer distance computations than the 10 million brute force requires, at the cost of returning an approximate nearest-neighbor set rather than a mathematically guaranteed exact one — which is the "A" in ANN search. Two parameters govern the tradeoff: efConstruction controls how thoroughly the graph is built (larger = better graph quality, slower to build, built once), and efSearch controls how large a candidate list is kept during each query's greedy walk (larger = higher recall, higher query latency, tunable per request without touching the index).
How the pieces connect: encoders, index, and marginalized generation
The formal model: RAG-Sequence and RAG-Token
The diagram's bottom half is not a metaphor — it is the literal computation defined in Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (NeurIPS 2020). The paper treats the retrieved passage z as a latent variable. The retriever contributes a probability distribution over the top-k candidates, p_η(z|x) — the same softmax computed above — and the generator contributes a conditional likelihood p_θ(y|x,z) of producing the target output given the query and one specific passage. The output probability is then a genuine marginalization: sum over the latent document, weighted by how much the retriever trusts it.
The paper defines two ways to do this marginalization. RAG-Sequence commits to one document for the entire output sequence and marginalizes at the sequence level:
p_RAG-Seq(y|x) ≈ Σ(z in top-k) p_η(z|x) · Π(i=1..T) p_θ(y_i | x, z, y_1:i-1)
RAG-Token instead lets the marginalization happen independently at every generated token, so different tokens in the same answer can effectively draw on different passages:
p_RAG-Tok(y|x) ≈ Π(i=1..T) Σ(z in top-k) p_η(z|x) · p_θ(y_i | x, z, y_1:i-1)
The operational difference matters for a two-sentence legal answer: under RAG-Sequence, the phrase "BNS Section 316" and the phrase "punishable up to seven years" must both come from whichever single document won the outer sum (usually the top-ranked one, since the product of per-token probabilities collapses quickly for a poorly matched document). Under RAG-Token, the section number could be drawn predominantly from one retrieved passage and the punishment detail predominantly from another, token by token, because the marginalization sum is recomputed fresh at every position.
Worked example: computing the marginal by hand
Reusing the retriever probabilities and assigning generator conditional probabilities — how likely each generator, conditioned on one passage, is to produce the correct target answer token for "BNS Section 316" — based on whether that passage actually contains the current information:
p_eta = {'A': 0.3664, 'B': 0.4409, 'C': 0.1926} # retriever, from worked example above
p_theta = {'A': 0.05, 'B': 0.85, 'C': 0.02} # generator's P(correct answer | doc)
marginal = sum(p_eta[k] * p_theta[k] for k in p_eta)
print(round(marginal, 3))
# 0.397
Every variable (p_eta, p_theta) is defined immediately above its use, and this produces exactly 0.397, matching the sum shown in the diagram (0.375 + 0.018 + 0.004 = 0.397).
Compare three different retrieval strategies on this same query. A lexical (BM25) retriever, fooled by keyword overlap, would return only Doc A as its top-1 result, giving the model a 0.05 probability of the correct answer — the assistant would very likely cite the superseded IPC section with total confidence. A dense retriever using pure top-1 selection (trust the single highest-scoring passage, Doc B, and ignore the rest) gives 0.85 — higher than the marginalized RAG-Sequence value of 0.397. This is a genuine, non-obvious tradeoff: marginalizing over multiple retrieved documents makes the system more robust on average across many queries, because it does not stake the entire answer on the retriever's single top guess being correct — but on any individual query where the top-1 document was already right, spreading probability mass across lower-quality documents can pull the marginal probability of the correct answer down below what pure top-1 trust would have given. Production systems tune k and, where possible, sharpen the retriever's score distribution (e.g., via a cross-encoder reranker) specifically to manage this tradeoff.
The misconception this chapter is built to correct
Most tutorials — and most students who have only seen a LangChain-style pipeline — describe "RAG" as: embed the query, fetch the top-k chunks, paste them all into one prompt, ask the LLM to answer. It is natural to assume this is literally what the Lewis et al. paper proposed, since the name matches. It is not. The original formulation is the latent-variable marginalization derived above: the retriever's confidence p_η(z|x) is a first-class probability that is mathematically combined with the generator's per-document likelihood p_θ(y|x,z), and in the paper's own experiments only the query encoder and generator were fine-tuned end-to-end; the passage encoder and its resulting index were kept fixed throughout training — unlike REALM (Guu et al. 2020), which periodically refreshes its index as its document encoder updates, an expense RAG's fixed-index design was built specifically to avoid.
That formal machinery requires access to the generator's internal probability distribution over outputs, conditioned separately on each candidate document — something you get for free when the generator is an open-weight model you control, but not when the generator is an API-only instruction-tuned model like GPT-4 or Claude, where you cannot backpropagate into the retriever or cheaply compute k separate per-document generation likelihoods. The "concatenate top-k into one context window" pattern that dominates production RAG today — covered in this course's capstone chapter — is an approximation that emerged specifically to work around that access limitation: instead of explicit per-document weighting, it relies on the transformer's attention mechanism to implicitly decide, token by token, how much each passage in the context contributes. That implicit weighting has no calibrated probability attached to it and no clean way to say "this specific sentence of the answer came from this specific document" — which is exactly why citation attribution is treated as a separate, nontrivial engineering problem in production systems, rather than falling out for free the way it does in the marginalized formulation above.
Fixing the query, not just the index: HyDE
Dense retrieval has a subtler failure mode than the lexical-overlap trap: queries and passages are written in different registers. A question is short and interrogative ("which section punishes criminal breach of trust?"); a passage is long and declarative (a statute's actual text). Even a well-trained bi-encoder can struggle when the question-style embedding and the passage-style embedding of the same underlying concept drift apart simply because the encoder saw far more passage-to-passage similarity than question-to-passage similarity during training.
Gao, Ma, Lin & Callan ("Precise Zero-Shot Dense Retrieval without Relevance Labels," 2022) proposed Hypothetical Document Embeddings (HyDE) as a fix that changes the query side rather than the index side. Instead of embedding the terse question directly, an LLM is first asked to write a plausible, passage-style hypothetical answer to it — even though that generated answer is likely to contain factual errors, since it comes from the same frozen parametric knowledge that caused the original problem. That hypothetical passage, not the original question, is what gets embedded and searched against the index:
# Illustrative pseudocode — generate_hypothetical_answer() and embed()
# are assumed helper functions (an LLM call and the passage encoder E_P
# respectively) and are not defined here; this is not runnable end-to-end.
def hyde_retrieve(query, index, k):
hypothetical_doc = generate_hypothetical_answer(query) # assumed helper
search_vector = embed(hypothetical_doc) # assumed helper
return index.search(search_vector, k)
The insight is that the generated hypothetical answer is now written in passage style, so its embedding lands much closer, geometrically, to real passages discussing the same content — even ones that use different specific facts or numbers than the hallucinated draft — than the original short question's embedding ever would. The hallucinated content in the hypothetical document is discarded entirely; only its embedding's position in vector space is used, and that position is what pulls in the real, correct passage.
Active recall
Attempt each question before reading its answer.
- In DPR's contrastive loss, what do "in-batch negatives" provide, and why does increasing batch size generally help training?
- Using the worked-example vectors, if passage B's vector were replaced by its unit-normalized version
b′ = b / ‖b‖, would the cosine-similarity ranking of A, B, C change? Would the raw dot-product ranking necessarily stay the same in general, even if it happens not to change here? - A corpus has 10,000,000 passages indexed with HNSW. Roughly how does the number of distance computations per query compare to brute force, and which HNSW parameter would you raise to trade query latency for higher recall without rebuilding the index?
- Write the RAG-Sequence and RAG-Token formulas from memory, and state in one sentence each what differs operationally when generating a two-sentence answer.
- Engineering raises
kfrom 3 to 4, adding Doc D with retriever score 1.10 (the highest of all four) butp_θ(y*|x,D) = 0, because D is a high-scoring but topically irrelevant embedding collision. Recomputep_ηfor all four documents and the new RAG-Sequence marginal. Does the marginal probability of the correct answer rise or fall, and by how much? - State the misconception this chapter corrected about "concatenating top-k chunks IS the RAG paper's method," and explain in two sentences why production systems diverge from the original formulation.
Answers
1. Every other (question, passage) pair already present in the same training batch supplies a negative example for the current question, at zero extra labeling cost — 127 negatives per question in a batch of 128, all drawn from other genuine questions' correct passages. A larger batch supplies more negatives per gradient step, which sharpens the softmax denominator in the loss and gives the encoder harder, more varied contrasts to learn from, generally improving retrieval quality up to diminishing returns and GPU memory limits.
2. Cosine similarity divides out vector magnitude by construction (cos = q·x / (‖q‖‖x‖)), so normalizing b to unit length changes nothing about the cosine ranking — it was already magnitude-invariant. Raw dot product, however, is sensitive to magnitude, and in this particular example the ranking happens not to change because B has both the best-aligned direction and the largest norm (0.9811, versus 0.9069 for A and 0.9502 for C) among the three. In general, though, a passage with a smaller angle to the query but a much larger norm than a competing passage can win on raw dot product while losing on cosine — which is precisely why many production vector indexes L2-normalize all passage embeddings at index time, so that inner-product search and cosine-similarity search become identical and ranking is governed by direction alone.
3. Brute force needs 10,000,000 dot products per query. HNSW's graph itself has roughly log_M(N) layers — with a typical branching factor M ≈ 16, that's log_16(10,000,000) ≈ 5.8, so about 6 layers, matching the diagram. The separate O(log N) figure is the expected total number of comparisons/hops the greedy walk performs while descending through those layers — on the order of a few dozen for N = 10,000,000 — each hop examining only a small, bounded candidate list rather than the full layer, giving several orders of magnitude fewer distance computations than brute force (an approximate reduction, not an exact figure, since it depends on graph parameters). To trade latency for recall at query time without rebuilding the index, raise efSearch, the size of the dynamic candidate list explored during the greedy walk; efConstruction instead controls graph quality at build time and would require re-indexing.
4. p_RAG-Seq(y|x) ≈ Σ(z∈top-k) p_η(z|x)·Π_i p_θ(y_i|x,z,y_1:i-1) and p_RAG-Tok(y|x) ≈ Π_i Σ(z∈top-k) p_η(z|x)·p_θ(y_i|x,z,y_1:i-1). Operationally: RAG-Sequence effectively commits the whole two-sentence answer to whichever single document dominates the outer sum, so both sentences tend to be sourced from the same passage; RAG-Token re-marginalizes at every generated word, so the first sentence's fact and the second sentence's fact can be drawn predominantly from two different retrieved passages within one answer.
5. New retriever softmax over four scores (0.785, 0.970, 0.142, 1.10): p_η ≈ {A: 0.244, B: 0.294, C: 0.128, D: 0.334}. Every existing document's weight drops purely from renormalization — B falls from 0.441 to 0.294, a 33% relative drop, even though nothing about B changed. New marginal: 0.244(0.05) + 0.294(0.85) + 0.128(0.02) + 0.334(0) ≈ 0.264, down from 0.397 — a drop of about 0.133, roughly a third. The marginal probability of the correct answer falls, purely because a confidently-scored but irrelevant document siphoned probability mass away from the genuinely relevant one; this is the ripple effect of softmax renormalization under an expanding k. RAG-Token would be affected differently only if the correct answer required combining facts from B and some other genuinely relevant retrieved document — in that case RAG-Token's per-token re-marginalization would let each fact draw its weight from whichever passage supports that specific token, so Doc D's high but empty score would dilute each token's sum a little (via renormalization) but would not force the entire answer to abandon a correct passage the way a RAG-Sequence collapse onto a single dominant document could.
6. The misconception is treating "paste the top-k retrieved chunks into the prompt" as literally the mechanism the original RAG paper describes. The paper instead defines a formal marginalization over a latent document variable with a calibrated retriever probability p_η(z|x) combined with a per-document generator likelihood; production systems diverge from this because API-only instruction-tuned generators do not expose the per-document conditional probabilities or backpropagation access the formal marginalization requires, so engineers substitute the transformer's attention mechanism as an implicit, uncalibrated approximation of that weighting.
Think About It
Think about this: How would you explain retrieval-augmented generation: building knowledge-enhanced ai systems to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where retrieval-augmented generation: building knowledge-enhanced ai systems is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting retrieval-augmented generation: building knowledge-enhanced ai systems to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind retrieval-augmented generation: building knowledge-enhanced ai systems, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.