The crawl is not the corpus
AI4Bharat's Sangraha project set out to give large language models a serious pretraining diet in India's scheduled languages. Before any model architecture decision got made, the team had to solve a much less glamorous problem: the raw web text pulled from Common Crawl, OSCAR, and language-specific crawls was saturated with duplication. The same government press release mirrored across a dozen news portals. The same Wikipedia stub machine-translated and re-translated by different scrapers. The same spam template stamped across thousands of near-identical domains. None of this is unique to Indian languages: every large pretraining corpus built from the open web looks like this. What separates a corpus that produces a strong model from one that produces a mediocre one, at the same parameter count and the same compute budget, is very often not the model at all. It is what got kept, what got thrown away, and in what proportion the surviving text got shown to the model. That is training data curation for pretraining, and it is a different discipline from cleaning a labeled dataset for a supervised classifier: there are no labels to check for noise, no ground truth to validate against. The signal you are curating for is "does keeping this token, or repeating it, or upweighting its source, make the model better," and every stage of the pipeline has to answer that question at a scale of hundreds of billions to trillions of tokens.
Three levers, not one
Pretraining corpus curation reduces to three operations, applied roughly in this order: deduplication (remove or collapse near-identical text so the same content does not get counted, and trained on, many times over), quality filtering (remove text that is unlikely to teach the model anything useful, such as boilerplate, spam, or auto-generated link farms), and domain mixture reweighting (decide what fraction of the surviving tokens should come from web text versus books versus code versus reference material, independent of how those fractions occurred naturally in the crawl). A sibling chapter in this curriculum covers dataset quality for labeled, supervised data: catching mislabeled examples, documenting a dataset with a datasheet, building a cleaning pipeline for a classification task. That is a different problem with a different failure mode (a wrong label actively teaches the model the wrong function). Here there are no labels. The corpus is the target itself, self-supervised next-token prediction over raw text, and the three levers above are what stand between a trillion raw tokens and a corpus a frontier lab would actually train on.
Finding near-duplicates without comparing every pair
Exact-match deduplication (hashing whole documents and dropping repeats) catches almost nothing on real web data, because near-duplicates rarely match byte-for-byte: a mirrored article gets a different header, a different ad block, a re-encoded whitespace character. What you actually want is documents whose Jaccard similarity, the size of the intersection of their shingle sets divided by the size of the union, exceeds some threshold. A shingle is a short overlapping span, commonly a sequence of five words or five characters, and a document is represented as the set of hashes of all its shingles. The problem is that computing exact Jaccard similarity for every pair of documents in a billion-document corpus requires roughly 5 x 10^17 comparisons, which is not something any lab runs. MinHash solves this by trading an exact answer for a cheap, unbiased estimate that only requires one pass over each document.
The idea: pick k random hash functions. For a document's shingle set, compute the minimum hash value each function produces over that set, and keep that single number per hash function as the document's "signature." A classical result in this area shows that for two sets A and B, if you draw a hash function uniformly at random from a suitable family, the probability that A and B produce the same minimum value under that hash function equals exactly the true Jaccard similarity of A and B, no matter how large the sets are. So the fraction of the k hash functions on which two documents' minimums agree is an unbiased estimator of their Jaccard similarity, and it costs O(k) numbers per document rather than O(n) full-text comparisons per pair.
Worked example: estimating Jaccard similarity with MinHash
Take two small shingle-ID sets standing in for two near-duplicate documents, for instance two mirrors of the same scraped article with a different footer appended:
Doc A = {1, 2, 3, 4, 5}, Doc B = {3, 4, 5, 6, 7}.
The true Jaccard similarity is exact and checkable by hand: the intersection is {3, 4, 5}, size 3; the union is {1, 2, 3, 4, 5, 6, 7}, size 7; so Jaccard = 3/7 ≈ 0.4286.
Now estimate it with k = 4 hash functions of the form h(x) = (a·x + b) mod p, using a prime p = 11 and four (a, b) pairs. The prime p must be larger than the largest shingle ID (7 here) so that distinct shingle IDs don't collide when reduced mod p going in; separately, for any fixed a not divisible by p, x → (ax+b) mod p is a bijection on Z_p, an independent fact that holds for any prime p regardless of input range:
def h(a, b, p, x):
return (a * x + b) % p
# Four hash functions (a*x + b) mod p, p prime and larger than any
# shingle ID being hashed (IDs run 1-7 here, so p = 11 is enough).
hash_params = [(3, 1), (4, 2), (5, 3), (2, 6)]
p = 11
# Shingle-ID sets for two near-duplicate documents.
doc_A = {1, 2, 3, 4, 5}
doc_B = {3, 4, 5, 6, 7}
def minhash_signature(doc, hash_params, p):
return [min(h(a, b, p, x) for x in doc) for (a, b) in hash_params]
sig_A = minhash_signature(doc_A, hash_params, p)
sig_B = minhash_signature(doc_B, hash_params, p)
matches = sum(1 for x, y in zip(sig_A, sig_B) if x == y)
estimated_jaccard = matches / len(hash_params)
true_jaccard = len(doc_A & doc_B) / len(doc_A | doc_B)
print(sig_A, sig_B, estimated_jaccard, true_jaccard)
# sig_A = [2, 0, 1, 1]
# sig_B = [0, 0, 0, 1]
# estimated_jaccard = 2 / 4 = 0.5
# true_jaccard = 3 / 7 = 0.4285714285714286
Tracing it by hand confirms the code: for h1(x) = 3x+1 mod 11, doc A's minimum over {1..5} is 2 (at x=4) and doc B's minimum over {3..7} is 0 (at x=7), so they disagree. For h2(x) = 4x+2 mod 11, both minimums land at x=5, giving 0 and 0: a match. For h3(x) = 5x+3 mod 11, A's minimum is 1 and B's is 0: disagreement. For h4(x) = 2x+6 mod 11, both minimums land at x=3, giving 1 and 1: a match. Two agreements out of four hash functions gives an estimate of 0.5, against a true value of 0.4286. The estimator is unbiased (its expected value over many random hash functions equals the true Jaccard), but with only k=4 samples the variance is large enough that a single draw can be off by 0.07 or more purely by chance. Production deduplication pipelines such as RefinedWeb and FineWeb use k in the range of 100 to a few hundred, because the standard error of the estimator shrinks proportionally to 1/√k; earlier pipelines like CCNet relied on exact-hash paragraph deduplication instead.
From signatures to candidates: LSH banding
A k-length signature per document still leaves the problem of finding which pairs, among a billion documents, have similar signatures, and comparing all pairs of signatures is exactly the same O(n²) wall the signatures were supposed to avoid. Locality-sensitive hashing (LSH) banding fixes this: split each document's k-length signature into b bands of r rows each (k = b·r), and hash each band separately. Two documents are flagged as dedup candidates if any single band matches exactly between them. Because a band only needs to match in its own small slice of the signature, near-duplicate pairs become findable with a simple bucket lookup instead of a pairwise scan: put every document into the buckets its bands land in, and only documents that land in the same bucket ever get compared directly.
The banding scheme has a predictable trade-off. For two documents with true Jaccard similarity s, the probability that at least one of the b bands matches exactly is P(s) = 1 − (1 − s^r)^b. This function is a steep S-curve in s, and its 50% crossover point sits near s* ≈ (1/b)^(1/r). With k=100 hash functions arranged as b=20 bands of r=5 rows, s* = (1/20)^(1/5). Since 20^(1/5) = e^((ln 20)/5) = e^(0.599) ≈ 1.82, s* ≈ 1/1.82 ≈ 0.55: pairs more similar than roughly 0.55 get caught with high probability, pairs less similar are mostly (correctly) ignored, and the corpus never needs a single full pairwise comparison. Choosing b and r is itself an engineering decision: more bands (larger b, smaller r) catches lower-similarity pairs but produces more false-positive candidates that still need a final exact check; fewer, larger bands does the opposite.
Quality filtering: separating a textbook from a coupon page
Deduplication answers "is this text repeated." It says nothing about whether the text was ever worth training on in the first place. A large fraction of Common Crawl is navigation menus, cookie notices, auto-generated product listings, and SEO filler, none of which teaches a language model anything about reasoning or fact. Early pipelines filtered with hand-written heuristics: minimum document length, a cap on the fraction of non-alphabetic characters, a check against lists of blocked domains. Guilherme Penedo and collaborators at Hugging Face, building the FineWeb and FineWeb-Edu datasets (Penedo et al., 2024), pushed this further by training a lightweight classifier to predict a document's educational value, using scores generated by a large instruction-tuned model as training labels, and then running that classifier over the entire filtered crawl to keep only the highest-scoring documents. Their ablations showed that a smaller corpus selected this way, trained on for the same number of tokens, produced stronger downstream benchmark performance than a larger unfiltered or heuristically filtered corpus at the same token budget. The lesson generalizes: past a certain point, adding more raw tokens is worth less than being pickier about which tokens you already have.
Domain mixture reweighting: raw proportions are not the right diet
Even a corpus that is fully deduplicated and quality-filtered has a composition problem. Web-scraped text dominates any open crawl by sheer volume relative to books, source code, or encyclopedic reference text, simply because there is more of the web than there are digitized books. Training in proportion to how the data naturally occurred is not the same as training on the mixture that minimizes loss, because web text is stylistically repetitive and narrow in register relative to how much of it there is, while a domain like books or code carries more information per token for the kinds of reasoning and long-range structure a language model needs to learn.
Sang Michael Xie and collaborators formalized this as an optimization problem in DoReMi (Xie et al., 2023): train a small reference model on the naive, size-proportional domain mixture; then train a second small proxy model whose per-domain sampling weights are updated by a minimax (group distributionally robust) objective, pushing weight toward whichever domain currently has the largest excess loss relative to the reference model; the proxy model's converged domain weights are then used, unchanged, as the sampling distribution for training the full-size model. The mechanism never needs to train the large model more than once. It only needs a small proxy run to discover which domains are underweighted relative to what the loss landscape actually rewards, and the paper's finding was that training on the reweighted mixture reached the reference model's validation perplexity in substantially fewer training steps than training on the original, size-proportional mixture. The diagram below illustrates the shape of this reweighting with example numbers (not the paper's published weights, which vary by corpus): a raw web-token-dominated mixture gets rebalanced toward books, code, and reference text, at no cost in total token count, because the mixture step resamples what already survived filtering rather than adding anything new.
How many times can the model see the same token
Deduplication, filtering, and reweighting all shrink the usable corpus relative to the raw crawl. Compute budgets for frontier training runs, following the scaling relationships from Hoffmann et al. (2022), often call for more tokens than the filtered corpus contains, which raises a question those levers do not answer: is it safe to just show the model the same filtered corpus more than once? Niklas Muennighoff and collaborators addressed this directly in "Scaling Data-Constrained Language Models" (Muennighoff et al., 2023), fitting a scaling law over training runs that deliberately repeated a fixed dataset for varying numbers of epochs. Their headline empirical result: repeating a fixed pretraining corpus for up to around four epochs costs little relative to training on an equivalently sized corpus of entirely unique tokens, but beyond that the marginal value of each additional pass decays quickly, and heavily repeated data contributes rapidly diminishing returns to the loss. The practical consequence for a curation pipeline is that the filtering and reweighting stages do not need to hit the full compute-optimal token count in unique tokens; they need to land within roughly a four-epoch multiple of it, after which it is worth spending curation effort finding more unique sources rather than training longer on the same filtered set.
Common misconception: "deduplication just saves storage"
A student who has only thought about dedup as a data-engineering housekeeping step tends to assume that once the corpus fits on fewer disks, the job is done, and that keeping duplicates around would cost more compute and money but would not change what the model actually learns, because the same information would still be present as it was before. This is wrong on two separate grounds, both demonstrated empirically by Lee et al. (2022) in "Deduplicating Training Data Makes Language Models Better." First, duplicated passages receive multiple independent gradient updates that reinforce the exact same continuation, which measurably raises the rate at which a trained model will later regurgitate that passage verbatim when prompted with its prefix, a memorization and, in deployment, a privacy and copyright risk that scales with how many times a given string was repeated in training. Second, held-out perplexity on non-duplicated evaluation text was measurably better for models trained on deduplicated data than for models trained on the raw, duplicate-heavy corpus at matched total token counts, meaning duplicates were not neutral filler; they actively skewed what the model treated as high-probability continuations, since whatever happened to be duplicated most (often exactly the boilerplate and spam quality filtering also targets) got effectively overweighted relative to its true information content. Dedup is not a storage optimization that happens to also save compute. It changes the function the model learns.
Active recall
Attempt each question before reading its answer.
1. In the worked MinHash example, suppose only the two hash functions that happened to match, h2 and h4, had been used instead of all four (k=2). What would the estimated Jaccard similarity be, and what does this reveal about small values of k?
2. With k=100 hash functions arranged as b=20 bands of r=5 rows, roughly what true Jaccard similarity corresponds to a 50% chance that a pair becomes an LSH dedup candidate?
3. The pipeline in the diagram assumed the quality classifier keeps about 45% of the 1.39T deduplicated tokens, yielding 620B filtered tokens. Suppose recalibrating the classifier drops the keep rate to 30% instead. Trace the full ripple: (a) how many tokens now reach the domain-reweighting stage, (b) do the raw-versus-reweighted domain percentages in the bar chart need to change, (c) does the upstream MinHash/LSH dedup stage need to be rerun, and (d) if the training run still targets 620B total training tokens, how many epochs over the new filtered corpus does that require, and is that within the safe range from the data-constrained scaling result?
4. Why does a real pipeline run cheap MinHash deduplication before the expensive neural quality classifier, rather than the other way around, and would running them in the reverse order change more than just the cost?
5. A junior engineer argues: "we deduplicated and quality-filtered the corpus, so every remaining document is clean, so we should just increase the Web domain's share back toward its natural proportion and drop Books and Wiki, since Web is clean now." What is wrong with this reasoning?
6. In one sentence, why does removing duplicates change a model's behavior even when the total token count fed to training is kept fixed by adding other text to compensate?
Answers.
1. Both matched, so the estimate would be 2/2 = 1.0, implying the documents are near-identical, when the true Jaccard is only 3/7 ≈ 0.4286. This is not a bug in the method; it is the expected behavior of an unbiased estimator with very few samples. The estimator's variance falls off proportionally to 1/k, so k=2 can land far from the true value purely by which two hash functions were chosen, which is exactly why production systems use k in the hundreds rather than a handful.
2. s* ≈ (1/b)^(1/r) = (1/20)^(1/5). Since 20^(1/5) = e^((ln20)/5) = e^(0.599) ≈ 1.82, s* ≈ 1/1.82 ≈ 0.55. Pairs with true similarity above roughly 0.55 are caught with high probability under this banding scheme; pairs below it mostly are not.
3. (a) 1.39T tokens × 30% ≈ 0.417T ≈ 417B tokens reach the reweighting stage, down from 620B. (b) No: domain proportions are percentages of whatever survives filtering, not absolute counts, so the raw-crawl and reweighted percentages in the chart are unaffected unless the classifier removes different domains at different rates (a classifier that happens to reject Web text more aggressively than Books would shift the raw-side percentages too, but a uniform keep-rate change does not). (c) No: MinHash/LSH dedup runs upstream of quality filtering and only depends on the deduplicated corpus, which is untouched by a change to the filter threshold; nothing about it needs to be rerun. (d) To reach 620B total training tokens from a 417B-token unique corpus requires 620/417 ≈ 1.49 passes, i.e. roughly one and a half epochs, well inside the "up to about four epochs" safe range from Muennighoff et al. (2023), so this recalibration would not be expected to meaningfully hurt model quality on its own.
4. MinHash/LSH dedup is cheap (hashing and bucket lookups); the quality classifier is a neural forward pass per document and costs far more per item scored. Running dedup first shrinks the corpus before the expensive step runs, so the classifier never wastes inference cost scoring many near-identical copies of the same text. The order can also change the final corpus, not just its cost: if quality filtering runs first, it may discard the specific copy within a near-duplicate cluster that would otherwise have been the copy dedup kept, changing which single surviving instance of that content ends up in the final corpus.
5. Dedup and quality filtering remove redundant and low-value text within each domain, but they do not fix the compositional imbalance across domains. Even a fully "clean" Web corpus remains stylistically narrower and less information-dense per token than well-curated books, code, or reference text, for the kinds of long-range structure and reasoning a language model needs to learn. Domain reweighting exists precisely because the loss-minimizing training mixture is not the mixture that occurs naturally at web scale, clean or not; "clean" and "correctly weighted" are orthogonal properties of the same corpus.
6. Because duplicated passages receive multiple independent gradient updates reinforcing the exact same continuation, which both increases verbatim memorization of that specific text and skews the effective training distribution toward whatever content happened to be duplicated most, regardless of how many unrelated tokens are added elsewhere to keep the total count the same.
Think About It
Think about this: How would you explain training data curation: the art of feeding models well to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where training data curation: the art of feeding models well is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting training data curation: the art of feeding models well to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind training data curation: the art of feeding models well, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.