Parliament passed the Bharatiya Nyaya Sanhita (BNS) in December 2023, and it came into force on 1 July 2024, replacing the colonial-era Indian Penal Code of 1860. From that date, every legal-aid chatbot, every paralegal search tool, every "ask about your rights" app trained on pre-2024 data was wrong. A pure large language model asked "what section covers theft, and what is the punishment?" would answer confidently and cite IPC Section 379 — a section number that, after 1 July 2024, refers to nothing. Worse, it might blend memorized IPC text with guessed BNS numbering and produce a fluent, specific, completely fabricated citation: "Section 305 BNS, punishable with up to five years." The number is invented. A citizen who acts on it is misinformed by a machine that sounded certain. This is not a hypothetical edge case — it is the default failure mode of every language model used past its training cutoff, on any fact that changes: a new government scheme's eligibility rules, a bank's revised KYC policy, a college's updated fee structure, this week's IRCTC refund window. The model's knowledge is parametric — baked into its weights at training time — and weights do not update when reality does.
Retrieval-Augmented Generation (RAG), introduced by Lewis et al. in "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (NeurIPS 2020), fixes this by giving the model a second, non-parametric memory: an external document store it can search at query time and read from before answering. Instead of asking the model "what is the BNS theft section," a RAG system first retrieves the actual statutory text of the relevant section from an indexed, up-to-date corpus, then asks the model to answer using only that retrieved text, with citations back to it. The model stops being the source of truth and becomes a reading-comprehension engine over a source of truth you control and can update. This chapter builds that system end to end — chunking, embedding, indexing, hybrid retrieval, reranking, grounded generation, evaluation, and the production engineering (latency, cost, failure modes) that separates a lab demo from something you could actually deploy for lakhs of users.
The two-phase architecture
Every production RAG system splits cleanly into two phases that run on completely different schedules. The offline indexing phase runs once when documents are ingested and again whenever they change: it chunks documents, embeds the chunks, and writes them into a vector index and a keyword index. The online query phase runs on every single user request: it embeds the query, searches both indexes, fuses and reranks the results, assembles a prompt, and calls the language model. Conflating these two phases — for instance, re-embedding your entire corpus inside the request handler — is the single most common reason a RAG prototype that answers in 200 milliseconds in a notebook takes 40 seconds in production.
Read the diagram in two halves. Above the dashed line, documents are chunked once, embedded once, and written into two separate stores: a vector index for semantic (meaning-based) search and a lexical index for exact keyword search — notice the chunker feeds the lexical index directly, bypassing the embedder entirely, because BM25 needs raw tokens, not vectors. Below the line, every query is embedded fresh, searched against both stores in parallel, fused into one ranking, narrowed by a more expensive reranker, and only then handed to the language model. The two dashed vertical lines mark the one place state crosses the boundary: the online phase reads indexes the offline phase built, but never writes to them mid-request.
Chunking: turning documents into retrievable units
You cannot embed and search an entire 40,000-token statute as one unit — a single vector would have to represent every offence from theft to sedition simultaneously, and cosine similarity against such a blurred average is nearly useless. So the first engineering decision is chunking: splitting each document into overlapping windows small enough that each one is topically coherent, and large enough to contain a complete thought (a full section, not half a sentence).
The overlap matters because fixed-size windows cut mid-topic. If Section 303's definition ends at token 505 and your chunk boundary falls at token 500, a naive non-overlapping split puts half the definition in one chunk and half in the next, and neither chunk alone answers the question. A stride shorter than the chunk size — chunking with overlap — guarantees that any span of text shorter than the overlap appears whole inside at least one chunk. Here is the window generator, traced by hand on a 100-token toy document with chunk_size = 40 and overlap = 10 (stride = 30):
def chunk_windows(n_tokens, chunk_size, overlap):
stride = chunk_size - overlap
starts = []
s = 0
while s < n_tokens:
starts.append(s)
if s + chunk_size >= n_tokens:
break
s += stride
return starts
for s in chunk_windows(100, 40, 10):
e = min(s + 40, 100)
print(f"chunk start={s} end={e} len={e-s}")
Trace it: s = 0 is appended; since 0 + 40 = 40 < 100, loop continues, s becomes 30. s = 30 is appended; 30 + 40 = 70 < 100, s becomes 60. s = 60 is appended; 60 + 40 = 100 ≥ 100, so the loop breaks after appending. Output:
chunk start=0 end=40 len=40
chunk start=30 end=70 len=40
chunk start=60 end=100 len=40Three chunks, each exactly 40 tokens, each overlapping its neighbour by 10 tokens, and the last chunk aligned to end exactly at the document boundary rather than trailing off as a short, half-empty fragment — that alignment is what the "break after append" guard buys you. In production, chunk_size = 500–800 tokens with 15–20% overlap is a common starting point for statute-style text; narrative or conversational text tolerates larger chunks, dense tabular or definitional text wants smaller ones.
Embeddings and the geometry of relevance
An embedding model — built from the transformer encoder stack you studied this year — maps a chunk of text to a dense vector in Rd (typically d = 384 to 1536) such that texts with similar meaning land close together in that space. "Closeness" is measured with cosine similarity: the cosine of the angle between two vectors, computed as cos(q, c) = (q · c) / (‖q‖ ‖c‖). It ignores vector magnitude and measures only direction, which is what you want — a chunk repeated twice as a longer document shouldn't score as "more relevant" just because its vector is longer.
Work through a concrete (toy, 3-dimensional — real models use hundreds of dimensions, the arithmetic scales identically) example. Query: "Section 303 — punishment?" embeds to q = [0.8, 0.1, 0.6]. Four candidate chunks:
q = [0.8, 0.1, 0.6] # query: "Section 303 — punishment?"
C1 = [0.75, 0.05, 0.65] # "Section 303 defines theft. Punishment... up to three years"
C2 = [0.10, 0.90, 0.20] # "Application for e-registration, prescribed fee under Section 7"
C3 = [0.50, 0.30, 0.40] # "Chapter XVII: property offences — theft, extortion, robbery"
C4 = [0.78, 0.08, 0.60] # "Section 103 defines murder. Punishment... death or life term"
‖q‖ = √(0.8² + 0.1² + 0.6²) = √1.01 = 1.0050. For C1: q·C1 = 0.8(0.75) + 0.1(0.05) + 0.6(0.65) = 0.600 + 0.005 + 0.390 = 0.995; ‖C1‖ = √(0.5625+0.0025+0.4225) = √0.9875 = 0.9937; cos(q,C1) = 0.995 / (1.0050 × 0.9937) = 0.9963. Repeating for the rest (verified by direct computation): cos(q,C2) = 0.3112, cos(q,C3) = 0.9428, cos(q,C4) = 0.9998.
Dense ranking by cosine similarity alone: C4 (0.9998) > C1 (0.9963) > C3 (0.9428) > C2 (0.3112). Look closely at what happened: C4 — about murder, Section 103 — outranks C1, the chunk that actually answers the question about Section 303 theft. This is not a bug in the arithmetic; it is a real, well-documented failure mode of dense retrieval. C1 and C4 are both one-line offence definitions followed by a punishment clause with a similar grammatical shape ("Section N defines X. Punishment... Y years"), so a general-purpose embedding model — trained to capture topical and structural similarity, not to distinguish specific numeric tokens — places them almost on top of each other in vector space. A dense-only system would confidently hand the model the wrong section, and the model, working faithfully from "grounded" but wrong context, would generate a fluent, wrong, and now much more convincing answer than an ungrounded hallucination would have been.
Indexing at scale: why brute force fails
Computing cosine similarity against every chunk, as the toy example does, costs O(n·d) per query — fine for four chunks, unusable for a corpus of two million. Production vector stores use approximate nearest-neighbour (ANN) structures instead. The dominant one is HNSW (Hierarchical Navigable Small World graphs, Malkov & Yashunin, IEEE TPAMI 2020), which builds a multi-layer graph over the embeddings: the top layer has very few nodes connected by long "highway" edges that let a search jump across large regions of the space quickly, and each layer below adds more nodes and shorter edges for local refinement. A query descends from the sparse top layer to the dense bottom layer, greedily moving to whichever neighbour is closest at each step, giving expected search cost close to O(log n) instead of O(n) — at the cost of being approximate: it can miss the true nearest neighbour occasionally in exchange for being orders of magnitude faster at scale.
Hybrid retrieval: fusing dense and sparse
The C1/C4 confusion above is exactly why production systems never rely on dense retrieval alone. BM25 (Robertson & Zaragoza, "The Probabilistic Relevance Framework: BM25 and Beyond," 2009) is a sparse, lexical scoring function: it ranks documents by weighted exact term overlap with the query, correcting for term rarity and document length. It has no notion of "murder" and "theft" being topically similar — it only sees that "303" and "theft" appear in C1 and not in C4. Continuing the example with simplified BM25-style raw term-overlap counts for the query terms {section, 303, theft, punishment} — C1 = 6, C3 = 2, C4 = 1, C2 = 0 — the sparse ranking is C1 > C3 > C4 > C2, correctly placing the theft section first.
Dense and sparse now disagree on the top result (C4 vs C1). Reciprocal Rank Fusion (Cormack, Clarke & Buettcher, SIGIR 2009) combines the two rankings without needing their raw scores to be on the same scale — it only uses each item's rank position in each list: RRF(c) = Σ 1/(k + rank_i(c)), summed over both retrievers, with k = 60 as the paper's standard damping constant.
dense_rank = {"C1": 2, "C2": 4, "C3": 3, "C4": 1}
sparse_rank = {"C1": 1, "C2": 4, "C3": 2, "C4": 3}
k = 60
rrf = {c: 1/(k+dense_rank[c]) + 1/(k+sparse_rank[c]) for c in dense_rank}
for c, score in sorted(rrf.items(), key=lambda x: -x[1]):
print(c, round(score, 6))
C1 0.032522 # 1/(60+2) + 1/(60+1)
C4 0.032266 # 1/(60+1) + 1/(60+3)
C3 0.032002 # 1/(60+3) + 1/(60+2)
C2 0.03125 # 1/(60+4) + 1/(60+4)Fusion restores the correct order: C1 (0.032522) narrowly beats C4 (0.032266). C1 wins because it ranks first on sparse and second on dense, while C4 ranks first on dense but only third on sparse — RRF rewards a candidate that two independent signals both rate highly, over one that only one signal loves. This is the practical justification for hybrid retrieval: dense search finds paraphrase and topical matches text search would miss entirely (a query about "theft" should still retrieve a chunk that says "stealing" without ever using the word "theft"), while sparse search anchors exact identifiers — section numbers, case citations, SKU codes, statute names — that embedding models routinely blur together.
Reranking before generation
RRF fusion is still cheap: it only reorders items using rank position, never reads the query and chunk together. The final precision step is a cross-encoder reranker (Nogueira & Cho, "Passage Re-ranking with BERT," 2019): unlike the embedding model, which encodes the query and each chunk separately into fixed vectors and compares them afterward (a "bi-encoder," fast but lossy), a cross-encoder feeds the query and a candidate chunk into the same transformer together, so every attention layer can directly compare specific tokens — "303" in the query against "303" in the chunk — and output a single relevance score. This is far more accurate but far too slow to run against millions of chunks, which is precisely why it runs only on the top 20–40 candidates that fusion already narrowed down: cheap-then-expensive, in that order, is the standard retrieval funnel.
Prompt assembly and grounded generation
The reranked top few chunks (commonly top-4) are assembled into a prompt with three parts: a system instruction constraining the model to answer only from the supplied text and to cite the chunk ID it drew each claim from; the chunks themselves, each tagged with a source identifier; and the user's question. Grounding is enforced by instruction, not guaranteed by architecture — the model can still ignore the instruction and answer from its own parametric memory, which is why evaluation (below) must separately check whether the generated answer is actually supported by the retrieved text, not merely whether retrieval found the right chunk.
Production engineering: latency and cost
A query-time latency budget, stage by stage, for a system serving from a ~2 million chunk corpus (illustrative order-of-magnitude figures, not a benchmark of any specific product): query embedding ≈ 15 ms, ANN search ≈ 8 ms, BM25 search ≈ 5 ms, RRF fusion ≈ 1 ms, cross-encoder rerank of 20 candidates ≈ 120 ms, prompt assembly ≈ 1 ms, LLM generation ≈ 1500 ms. Sum: 15+8+5+1+120+1+1500 = 1650 ms. Generation dominates by roughly 10×; if you need to cut latency, optimising the retrieval stack past this point has almost no user-visible effect — the lever that matters is generation (smaller/faster model, streaming the response token-by-token so the user sees output before the full 1500 ms elapses, or a semantic cache that skips generation entirely for a repeated question).
Cost scales with tokens, not documents. If the system assembles a prompt of 150 system tokens + 4 chunks × 220 tokens + 30 question tokens = 1060 input tokens, and generates 250 output tokens, at illustrative rates of $0.003 per 1K input tokens and $0.015 per 1K output tokens: cost = (1060/1000)(0.003) + (250/1000)(0.015) = 0.00318 + 0.00375 = $0.00693 per query. At 50,000 queries a day that is $346.50/day — and the retrieved context alone is over 38% of the token bill (880 chunk tokens of the 1060 total: (880/1000)(0.003) = $0.00264, $0.00264/$0.00693 ≈ 0.381) despite contributing zero to the words the user actually reads, which is the direct cost argument for keeping top-k small and reranking well rather than retrieving generously and hoping the model sorts it out.
Evaluating a RAG system
Retrieval quality and generation quality must be measured separately, because a system can fail at either independently. Two standard retrieval metrics: Recall@k — the fraction of test queries for which the gold-relevant chunk appears anywhere in the top-k results — and Mean Reciprocal Rank (MRR) — the average of 1/rank of the gold chunk across queries, 0 if it is missed entirely. For five test queries whose gold chunk landed at ranks [1, 3, miss, 1, 2] with k = 4: Recall@4 = 4/5 = 0.8 (four of five queries had the gold chunk somewhere in the top four; query 3 missed it entirely). MRR = (1/1 + 1/3 + 0 + 1/1 + 1/2) / 5 = (1 + 0.3333 + 0 + 1 + 0.5) / 5 = 2.8333/5 = 0.5667. Recall@4 tells you retrieval usually finds the right chunk somewhere in the window handed to the model; MRR additionally penalises finding it at rank 3 instead of rank 1, which matters because reranking, prompt position, and Lost-in-the-Middle effects (next section) all make rank within the window matter, not just presence in it.
On the generation side, frameworks such as RAGAS (Es et al., "RAGAS: Automated Evaluation of Retrieval Augmented Generation," 2023) measure faithfulness — what fraction of claims in the generated answer are actually entailed by the retrieved chunks, checked by decomposing the answer into individual claims and verifying each against the source text — separately from answer relevance, whether the answer actually addresses the question asked. A system can retrieve perfectly (high Recall@k) and still generate an unfaithful answer if the model ignores the retrieved text in favour of its own memorized (and possibly outdated) knowledge; measuring retrieval alone would never catch that failure.
Common misconception: "more retrieved context is always better"
It is intuitive to think that widening top-k — retrieving 20 chunks instead of 4 — can only help, since the model has strictly more information to draw from and can simply ignore what is irrelevant. This is false, and the mechanism why is directly measured in Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (TACL 2024): transformer language models exhibit a U-shaped attention pattern over long contexts — they attend well to information near the start and near the end of the prompt, and measurably worse to information buried in the middle, regardless of how large the model's advertised context window is. Stuffing 20 chunks into a prompt does not give the model 20 chunks' worth of usable information; it gives strong attention to roughly the first and last chunk and degraded attention to everything between, meaning the single correctly-relevant chunk is more likely to get "lost" the more chunks surround it. This is precisely why the pipeline in this chapter narrows aggressively — fusion over 20 candidates, reranking down to 4 — rather than handing the model everything retrieval found: fewer, better-ordered chunks with the most relevant one placed first genuinely outperform more chunks in raw accuracy, while also costing less and running faster. "More context" and "more retrieval" are not the same axis, and only the second one, spent carefully, improves answers.
Active recall
Attempt each question before reading its answer.
1. Why does a pure LLM (no retrieval) asked about the punishment under BNS Section 303 risk a more dangerous failure than simply saying "I don't know"?
2. In the worked cosine-similarity example, dense retrieval alone ranked C4 (murder, Section 103) above C1 (theft, Section 303) — 0.9998 vs 0.9963. Explain the mechanism behind the mistake, and show how hybrid retrieval corrected it.
3. Given gold-chunk ranks [1, 3, miss, 1, 2] across five test queries, compute Recall@4 and MRR.
4. A 50,000-token document is chunked at chunk_size=500, overlap=100, producing 125 chunks. If chunk_size is doubled to 1000 with overlap doubled to 200 (same 20% overlap ratio), trace the full ripple: (a) new chunk count and its effect on ANN search latency over an HNSW index, (b) effect on LLM input cost for a top-4 retrieval, (c) effect on retrieval precision.
5. Why is BM25 still necessary in a 2026 production RAG stack when dense embedding models are very good at capturing meaning?
6. A classmate argues: "Our LLM has a 128K token context window — let's skip retrieval and paste the entire 40,000-token BNS document into every query." Beyond Lost-in-the-Middle, give two concrete production problems with this.
Answers.
1. A plain refusal ("I don't know") is safe because the user knows to seek another source. A confident fabricated citation — a real-sounding section number with a plausible but invented punishment — is worse than no answer, because it is indistinguishable in tone from a correct one and the user has no signal to doubt it. RAG's value is not just "more accurate," it is converting silent failure risk into a checkable citation the user (or a downstream system) can verify against the source chunk.
2. C1 and C4 are both terse offence-definition-then-punishment clauses ("Section N defines X. Punishment... Y"), so a general-purpose embedding model captures their shared grammatical and topical shape more strongly than the specific numeric token that actually distinguishes them (303 vs 103, theft vs murder) — dense embeddings compress rare, specific tokens like statute numbers less faithfully than they compress broad topic and structure. BM25, scored on exact term overlap, ranked C1 first (raw score 6, containing "303" and "theft" directly) and C4 third (score 1). Reciprocal Rank Fusion combined dense rank 2/sparse rank 1 for C1 (RRF = 1/62 + 1/61 = 0.032522) against dense rank 1/sparse rank 3 for C4 (RRF = 1/61 + 1/63 = 0.032266), correctly restoring C1 to first place because it was strong on both signals rather than exceptional on only one.
3. Recall@4 = 4/5 = 0.8 (the gold chunk appeared in the top 4 for four of five queries — only the third query missed entirely). MRR = (1/1 + 1/3 + 0 + 1/1 + 1/2)/5 = 2.8333/5 ≈ 0.5667.
4. (a) With stride = chunk_size − overlap = 800, the 50,000-token document produces 63 chunks (down from 125). But HNSW search cost scales roughly with ln(n), not n: ln(125) = 4.83 versus ln(63) = 4.14 — only a 14% reduction in expected search hops, not the 50% a naive "half the chunks, half the time" intuition would predict. (b) Each retrieved chunk now carries ~1000 tokens instead of ~500, so a top-4 context grows from 4×500=2000 to 4×1000=4000 tokens — the generation input cost for the retrieved-context portion roughly doubles, even though the number of chunks retrieved (4) didn't change. (c) Larger chunks are more likely to straddle multiple statute sections or sub-topics (e.g., both theft and extortion in one 1000-token window), which averages their embedding across unrelated content and dilutes topical precision — a chunk covering two offences scores moderately for queries about either one, instead of scoring very high for exactly one, crowding out more precisely-matched smaller chunks in the ranking.
5. Dense embeddings are trained to capture semantic and topical similarity, which systematically under-distinguishes specific, low-frequency, or purely identifier-like tokens — section numbers, case citation numbers, product SKUs, employee IDs — precisely because two texts differing only in such a token are otherwise nearly identical in meaning-space (as C1 vs C4 demonstrated). BM25's exact-term matching has no such blur: it treats "303" and "103" as completely different tokens regardless of how similar their surrounding sentences are. Any domain where users search by exact identifier — legal section numbers, part numbers, error codes — needs the lexical signal dense retrieval alone cannot reliably provide.
6. First, cost and latency: every single query now pays to process 40,000 input tokens through the model regardless of whether the answer needs 200 tokens' worth of that text, multiplying both the per-query bill and the prefill (time-to-first-token) latency by roughly 40× compared to a 4-chunk, ~2000-token retrieval — at scale (the 50,000-queries/day example) this turns a few-hundred-rupee-per-day generation cost into a much larger one for no accuracy gain, since Lost-in-the-Middle means most of those 40,000 tokens are barely attended to anyway. Second, traceability and updates: with retrieval, an answer cites a specific chunk ID you can independently verify or flag as stale and re-index; with the whole document pasted in every time, there is no fine-grained citation to check, and updating even one section requires nothing structurally different from before (the whole blob is reread each time) — but you have thrown away the very mechanism, per-chunk indexing, that would let you invalidate or hot-swap just the changed section in a large corpus rather than treating the entire document as a single atomic unit at every scale.
Think About It
Think about this: How would you explain capstone: building a production rag system to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where capstone: building a production rag system is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting capstone: building a production rag system to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind capstone: building a production rag system, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.