When "no overlap" still means contamination
In 2025 and 2026, several Indian labs building multilingual foundation models pretrain on web crawls that include exam-prep sites: Testbook mirrors, Unacademy notes, and old IIT-JEE and NEET question banks that have been re-hosted, translated into Hindi or Tamil, and lightly reworded across dozens of mirror pages chasing SEO traffic. Before reporting a benchmark number, an engineering team runs the standard hygiene step: a lexical overlap filter between the pretraining corpus and the evaluation set — the kind of n-gram or MinHash matching you have already seen as the first line of defense against test-set leakage. Anything above a similarity threshold gets dropped. The filter reports zero matches. The benchmark number looks clean.
It is not clean, and the reason is worth sitting with, because it is not a bug in the filter — it is a property of what n-gram matching can and cannot see. Suppose the original JEE-Physics question sitting in the benchmark test set is "A block of mass 2 kg slides down a frictionless incline of angle 30°. Find its acceleration." Somewhere in the crawl is a Hindi-translated version of the exact same problem, or an English mirror that opens with "Find the acceleration of a 2 kg block sliding down a 30° frictionless incline." Same physics, same numbers, same answer — different token sequence, possibly a different language entirely. Every contiguous word sequence the filter hashes is different, so the Jaccard similarity between the two is near zero, and the filter — correctly, given what it measures — waves it through. The model has still seen the problem. It can still have memorized the answer. The evaluation number is still inflated. This chapter is about the class of decontamination technique built specifically to catch this: methods that test what the model has memorized by interrogating its behavior, and production pipelines that search for meaning rather than matching tokens.
Guided prompting: making the model testify against itself
Shahriar Golchin and Mihai Surdeanu, in "Time Travel in LLMs: Tracing Data Contamination in Large Language Models" (ICLR 2024), proposed a membership-inference test that sidesteps lexical matching entirely. The idea: if a model has memorized a specific benchmark instance during training, then telling the model exactly which dataset and split that instance came from should measurably help it reconstruct the missing part of that instance — because the dataset name and split act as a retrieval cue into whatever the model memorized. A model that has never seen the instance gets no such boost.
Concretely, take a benchmark instance — a question and its gold answer — and split it into Part A (a prefix, say the first 70% of tokens) and Part B (the remaining tokens, the part you're going to try to make the model reproduce). Build two prompts from Part A:
Guided prompt: Part A, followed by an explicit cue such as "This is part of an example from the <dataset name>, <split> split. Complete it exactly as it appears in the dataset."
General prompt: Part A alone, with no dataset name, no split, no hint that this is a benchmark question at all — just "complete this."
Run the same frozen model, same decoding settings, on both prompts. Score each completion against the real Part B using an overlap metric such as ROUGE-L (longest common subsequence overlap, normalized). Call the two scores score_G and score_N. If the model is contaminated on that instance, the dataset-name cue in the guided prompt should retrieve something close to the memorized original, so score_G should exceed score_N by a real margin. If the model has never seen the instance, knowing the dataset's name gives it nothing — it can't retrieve what was never stored — so score_G and score_N should be statistically indistinguishable. (The original paper additionally uses an LLM judge to classify whether a completion counts as a near-exact reproduction rather than relying on ROUGE-L alone; the ROUGE-L delta presented here is a simplified version of the comparison, kept for pedagogical clarity.) Crucially, nothing in this test looks at n-grams shared between training data and the benchmark — it looks at what the model can produce, which is exactly what survives paraphrase and translation and any other surface transformation that defeats lexical filters.
Worked example: how far a paraphrase escapes an n-gram filter
Before trusting that guided prompting is solving a real problem, verify by hand how badly lexical filtering fails on a paraphrase — the same failure mode as the JEE example above, made small enough to compute exactly. Take one sentence and one paraphrase of it:
Original: "what is the time complexity of binary search on a sorted array of n elements" (15 tokens)
Rephrased: "in a sorted array containing n elements what is the time complexity required to perform a binary search" (18 tokens)
A typical decontamination pipeline computes the Jaccard similarity of the two sentences' word 5-grams: the size of their shared 5-grams divided by the size of their combined 5-grams. The original, with 15 tokens, has 15 − 5 + 1 = 11 distinct 5-grams (sliding a 5-word window across it, one position at a time). The rephrased sentence, with 18 tokens, has 18 − 5 + 1 = 14 distinct 5-grams. Now compare the two sets position by position. Enumerating all 11 original 5-grams against all 14 rephrased 5-grams, exactly one pair matches exactly: "what is the time complexity" appears verbatim in both (it survived the reordering because that clause happened to stay intact). Every other 5-gram differs — the original's "...of binary search on a sorted..." never recurs because "of" and "on" are absent from the rephrased sentence's corresponding clauses, and the rephrased sentence's "...containing n elements what is..." never occurs in the original because "containing" doesn't appear there at all.
With intersection = 1 and union = 11 + 14 − 1 = 24, Jaccard similarity = 1/24 ≈ 0.0417. That is not a subtle near-miss below some aggressive threshold — it is a near-total absence of lexical overlap between two sentences that ask the identical question. Verify this in code rather than trusting the hand count:
def ngrams(tokens, n):
return set(tuple(tokens[i:i+n]) for i in range(len(tokens) - n + 1))
def jaccard(a, b):
return len(a & b) / len(a | b)
original = "what is the time complexity of binary search on a sorted array of n elements".split()
rephrased = "in a sorted array containing n elements what is the time complexity required to perform a binary search".split()
orig_5grams = ngrams(original, 5)
reph_5grams = ngrams(rephrased, 5)
print(len(orig_5grams), len(reph_5grams))
print(round(jaccard(orig_5grams, reph_5grams), 4))
This prints 11 14 on the first line and 0.0417 on the second, matching the hand count exactly. A pipeline that discards training documents only when their n-gram Jaccard similarity with a benchmark question exceeds some threshold like 0.8 will keep this document without a second thought — the model can train on the rephrased version of a benchmark question and the lexical filter will never register a problem, precisely because the filter is measuring token-sequence overlap, not whether the two sentences ask the same thing.
How guided prompting sits inside the pipeline
The diagram below traces the full guided-prompting test end to end: splitting an instance, building the two prompts, running them through the same frozen model, scoring both completions against the withheld ground truth, and rolling the per-instance signal up into a dataset-level verdict.
Why this can't just be brute-forced with embeddings at pretraining scale
Guided prompting is an after-the-fact audit — you run it once a model exists, on a manageable number of benchmark instances. The complementary, upstream problem is different: before training even starts, you want to remove any pretraining document whose meaning matches a benchmark question, not just its tokens. The natural fix is to replace n-gram Jaccard with cosine similarity between sentence embeddings, since a good embedding model places paraphrases and translations of the same question close together in vector space even when they share almost no tokens. But the systems cost of doing this at pretraining scale is worth tracing through, because it explains why real pipelines don't do this as a naive all-pairs comparison.
Take the benchmark side first, since it's the small, fixed side. MMLU's test split has roughly 14,000 questions. Embedding each one with a 768-dimensional encoder in float32 costs 14,000 × 768 × 4 bytes ≈ 43 MB — small enough to sit resident in GPU memory next to the encoder itself, so no approximate-nearest-neighbor index is even needed on this side; brute-force cosine similarity against all 14,000 benchmark vectors per query is only about 2 × 768 × 14,000 ≈ 21.5 million multiply-adds, trivial on any accelerator.
The training-corpus side is where the cost actually lives, and it isn't the comparison — it's the embedding pass itself. Chunk a pretraining corpus into overlapping windows (say 256 tokens, stride 128) and a 1-trillion-token corpus becomes roughly 1×10¹²/128 ≈ 7.8 billion windows that each need one forward pass through the embedding model before any comparison can happen. Assume, as an illustrative order-of-magnitude figure rather than a measured benchmark, that a single accelerator embeds windows at around 5,000/second in batched inference: 7.8×10⁹ / 5,000 ≈ 1.56×10⁶ seconds, about 433 GPU-hours on one device. That number is the real argument for how these pipelines are actually built: this pass is run once, offline, sharded embarrassingly-parallel across a few hundred GPUs for a few hours, producing a static set of embeddings and near-duplicate flags before training ever starts — not re-run per query, and not something you'd want to redo every time a new benchmark is added, which is a separate cost problem of its own.
That last point is also why some evaluation groups sidestep the filtering problem instead of solving it harder. Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica, in "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples" (2023), demonstrated directly that GPT-4-generated rephrasings of benchmark questions can be trained on without tripping standard n-gram decontamination filters, while still inflating the model's measured score on the original, unmodified benchmark — the same failure mode as the worked example above, shown to hold at model-training scale rather than on two toy sentences. Several members of that group also work on LMSYS Chatbot Arena, which takes a different tack entirely: instead of trying to prove a fixed test set was never seen, they continuously collect new evaluation data (human preference votes, freshly authored questions) after any candidate model's training cutoff, so there is no static benchmark text to leak in the first place. Filtering and dynamic evaluation are complementary, not substitutes — filtering catches near-verbatim leaks cheaply before training; dynamic benchmarks catch what filtering structurally cannot.
Common misconception
Many students assume: "if the pretraining corpus scores below the decontamination pipeline's overlap threshold against a benchmark, the benchmark is contamination-free." This is false, and the worked example above shows exactly why. An n-gram or MinHash overlap score is a measurement of shared token sequences, full stop — it is not a measurement of shared meaning. Tightening the threshold (demanding, say, 0.5 Jaccard instead of 0.8) only changes which purely lexical near-duplicates get caught; it can never catch a semantic duplicate, because "semantically identical, lexically different" is a region of the space the metric was never built to see. A paraphrase, a translation, or a reworded exam question can carry a Jaccard similarity of essentially zero while being the exact same test item the model was later evaluated on. Catching that requires an orthogonal signal that operates on meaning or on model behavior — an embedding-similarity search, or a behavioral test like guided prompting — used alongside lexical filtering, not as a replacement for it, since lexical filtering is still the cheapest way to catch the far more common case of literal, unmodified copy-paste leakage.
Active recall
Attempt each question before reading its answer.
Q1. A pipeline uses a 5-gram Jaccard threshold of 0.8 to flag contamination. Explain, using the worked example, why this threshold correctly catches verbatim copies but fails on the paraphrase pair given in this chapter.
Q2. Explain why comparing a guided completion to a general completion provides evidence of memorization even when neither completion shares a single token with the training data verbatim.
Q3. Recompute the Jaccard similarity between the same original and rephrased sentence from the worked example, but using 8-grams instead of 5-grams. Does detection get easier or harder, and why?
Q4. A lab decontaminates its English pretraining shard against an English benchmark using n-gram matching and finds zero overlap. A separate Hindi pretraining shard contains a machine-translated version of the same benchmark question, translated from a mirror site that also hosted the English original. Would the English-only filter catch this? What does this expose about decontaminating multilingual training corpora?
Q5. Explain why embedding-based semantic decontamination at pretraining scale is not run as a live, per-query brute-force search of every training document against every benchmark question, using the GPU-hour estimate from this chapter.
Q6. Why does the guided-prompting method aggregate Δ = score_G − score_N across many benchmark instances and apply a significance test, rather than declaring contamination the moment a single instance shows score_G > score_N?
A1. A 5-gram Jaccard threshold measures how many identical 5-word windows two texts share. A verbatim copy shares nearly all of its 5-grams with the original, so its Jaccard score sits near 1.0 and comfortably exceeds 0.8. The worked-example paraphrase reorders and substitutes words while preserving meaning, which destroys almost every contiguous 5-word window even though the underlying question is unchanged — the computed Jaccard was 1/24 ≈ 0.0417, nowhere close to 0.8. The threshold is doing exactly what it was designed to do; the paraphrase simply lies outside what a lexical-overlap metric can perceive.
A2. The guided prompt's only addition over the general prompt is a cue naming the dataset and split — no new content from the actual instance. If that cue measurably improves the model's ability to reproduce the withheld Part B (score_G notably higher than score_N), the improvement can only be explained by the model retrieving something it stored about that specific instance under that dataset's name, which is memorization. A model that never saw the instance has nothing for the cue to retrieve, so the cue can't help it, and score_G ≈ score_N. The comparison isolates the effect of "having seen this exact instance before" from the effect of "being generally good at completing this kind of question," without ever checking token overlap with training data.
A3. With 8-grams: the original (15 tokens) yields 15 − 8 + 1 = 8 distinct 8-grams; the rephrased sentence (18 tokens) yields 18 − 8 + 1 = 11 distinct 8-grams. Checking all 8 original 8-grams against all 11 rephrased 8-grams, none match — every original 8-gram from position 1 onward spans the token "on" (from "...binary search on a sorted..."), and "on" never appears anywhere in the rephrased sentence, so no 8-gram containing it can match; the one 8-gram that doesn't span "on" (positions 0–7, "what is the time complexity of binary search") diverges from its nearest rephrased counterpart at the sixth token ("of" versus "required"). Intersection = 0, union = 8 + 11 − 0 = 19, Jaccard = 0. Detection gets strictly harder as n grows: longer n-grams are more sensitive to any single word changing position, so raising n to reduce false positives on generic short phrases simultaneously destroys the pipeline's ability to catch paraphrased leaks — the same knob makes the filter both more lexically precise and more semantically blind at once.
A4. No, the English-only filter would not catch it, because it never sees the Hindi text at all — filtering is typically run per-language-shard against a benchmark in the corresponding language, and even a same-language filter would fail here anyway since translation destroys token overlap as completely as paraphrase does. This exposes a real risk for multilingual pretraining: decontamination has to run separately (and correctly) for every language a model trains on, matched against translated versions of the benchmark where they exist, or contamination can enter through whichever language shard nobody thought to filter carefully. A benchmark reported as "decontaminated" against the English test set says nothing about whether the model saw the same questions in Hindi, Tamil, or any other language present in the training mix.
A5. The bottleneck is the embedding forward pass over the training corpus, not the vector comparison. The benchmark side is tiny (about 43 MB of vectors for MMLU's 14,000 questions) and cheap to compare against per query. The training-corpus side, at 1 trillion tokens chunked into 256-token windows with stride 128, is roughly 7.8 billion windows, each requiring a full forward pass through the embedding model — at an illustrative 5,000 windows/second on one accelerator, that is about 433 GPU-hours on a single device. Running this as a live per-query search would mean re-embedding the same trillions of tokens repeatedly; instead it is done once, offline, sharded across many GPUs in parallel, producing a static filtered/flagged corpus before training begins.
A6. A single instance can show score_G > score_N purely by chance — decoding noise, an easy completion that any model would get partly right, or an unlucky general-prompt sample. Requiring the pattern to hold consistently across many instances, and testing it with something like a Wilcoxon signed-rank test on the paired Δ values, distinguishes a real memorization signal (guided consistently and significantly outperforms general across the dataset) from sampling noise at the level of one lucky or unlucky completion. It also matches how contamination actually behaves in practice: a model is rarely contaminated on exactly one instance in isolation, but rather on a batch of instances that co-occurred in some scraped source, so a dataset-level statistical signal is both more robust and more representative of the underlying risk than any single instance's score.
Think About It
Think about this: How would you explain data decontamination: ensuring fair model evaluation to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind data decontamination: ensuring fair model evaluation, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.