Pre-training at Scale: Data, Compute, and Scaling Laws

In 2024, the Indian startup Sarvam AI raised funding to build a foundation model for Indian languages and had to answer a question with real rupees attached: given a fixed compute budget — say, a cluster rental worth a few crore rupees, translating into a fixed number of GPU-hours — how large should the model be, and how much text should it be trained on? Make the model too big and train it on too little text, and you waste compute building a network that never sees enough examples to use its own capacity. Make it too small and drown it in data, and you waste compute on a network too narrow to extract more signal from the text it has already seen. This is not a vague trade-off to be eyeballed. It is a precise optimization problem, and the single most consequential paper in the history of large language model training — Hoffmann et al., "Training Compute-Optimal Large Language Models" (DeepMind, 2022), known as the Chinchilla paper — solved it quantitatively. This chapter builds the machinery to solve it yourself: the compute accounting that turns "how big" and "how much data" into FLOPs, the scaling laws that predict loss from those quantities, and the systems engineering — data pipelines and distributed training — that turns the arithmetic into an actual trained model.

What pre-training actually optimizes

Pre-training a large language model means minimizing cross-entropy loss on next-token prediction over a massive, mostly unlabeled text corpus: at each position, the model outputs a probability distribution over the vocabulary, and the loss penalizes low probability assigned to the token that actually came next. There is no separate "training signal" beyond the text itself — every sentence on the internet is simultaneously a million tiny supervised examples. This is why pre-training scales so differently from the supervised learning you studied in Grade 11: labeled datasets top out at what humans can annotate, but self-supervised next-token prediction can consume essentially all machine-readable text that exists, which is why the questions in this chapter are about hundreds of billions to trillions of tokens rather than thousands of labeled rows.

Three quantities govern the outcome of a pre-training run: N, the number of trainable parameters in the model; D, the number of training tokens the model is shown; and C, the total compute spent, measured in floating-point operations (FLOPs). These three are not independent — compute is spent processing tokens through parameters — and the relationship between them is the load-bearing formula of this entire chapter.

The 6ND approximation

For a standard transformer, a forward pass over one token costs approximately 2N FLOPs: each of the N parameters participates in roughly one multiply-add per token (a multiply and an add, i.e. 2 FLOPs, per parameter), because nearly every parameter sits inside a matrix multiplication that touches every token in the sequence. This ignores the attention mechanism's cost, which scales with sequence length squared rather than with N — a fair approximation as long as N is large relative to the sequence length, true for every model this chapter discusses. Backpropagation costs roughly twice the forward pass, because gradients must be computed with respect to both the activations and the weights, giving about 4N FLOPs for the backward pass. Total per-token cost: 2N + 4N = 6N FLOPs. Training on D tokens therefore costs:

C ≈ 6 · N · D

This approximation — first used systematically by Kaplan, McCandlish, et al. in "Scaling Laws for Neural Language Models" (OpenAI, 2020) — is the exchange rate between the two things you actually control (model size and dataset size) and the one thing you actually pay for (compute). Every calculation in this chapter runs through this equation.

Kaplan (2020) and the correction by Chinchilla (2022)

Kaplan et al. (2020) trained hundreds of models of varying size and measured how loss falls as a power law in N, in D, and in C independently, when the other two are not the bottleneck: loss ∝ N^(−α), loss ∝ D^(−β), for small positive exponents. This was a genuine breakthrough — it showed that language model quality is predictable in advance, not a matter of trial and error — and it directly justified the industry's 2020–2022 strategy of building ever-larger models: GPT-3 (Brown et al., 2020) at 175 billion parameters, Gopher and Megatron-Turing NLG following the same instinct. But Kaplan et al.'s experiments held training duration close to one epoch over a large but roughly fixed-size corpus and varied N as the main lever, which biased their fitted optimal allocation toward spending new compute mostly on model size rather than on more data.

Hoffmann et al. (2022) re-ran the question properly using what they called IsoFLOP profiles: fix a compute budget C, train many models of different sizes N (each trained on however many tokens D = C/(6N) that budget allows at that size), and plot the resulting loss against N. The result is a U-shaped curve for every fixed C — too small a model wastes the budget on undertrained excess data with too little capacity to use it well; too large a model wastes the budget on excess parameters starved of data. Somewhere in between sits a minimum: the compute-optimal N for that C. Repeating this at several values of C and connecting the minima traces out a frontier — the compute-optimal scaling law. That frontier is what the diagram below reconstructs.

The empirical result from fitting this frontier across many orders of magnitude of compute: N_opt and D_opt should both scale as roughly C^0.5 — equally — rather than pushing nearly all new compute into N as the field had been doing. Fitted to real data, this works out to a rough rule of thumb: the compute-optimal token-to-parameter ratio is approximately 20 tokens per parameter. Chinchilla itself, the model built to validate this law, used 70 billion parameters trained on 1.4 trillion tokens — a ratio of exactly 20 — and outperformed Gopher, a 280-billion-parameter model from the same lab trained on only 300 billion tokens (a ratio of about 1), despite using the same total training compute. Gopher was too big for the data it saw; Chinchilla, smaller but fed proportionally far more text, won.

Worked example: sizing a compute-optimal model

Suppose you are given a training budget of C = 6×10²³ FLOPs — roughly the scale DeepMind worked at for the Chinchilla comparison. Using the Chinchilla ratio D ≈ 20N and the compute identity C = 6ND, substitute:

C = 6 · N · D
D = 20N
⇒ C = 6 · N · (20N) = 120 N²
⇒ N = sqrt(C / 120)

Step by step: C/120 = 6×10²³ / 120 = 5×10²¹. Taking the square root: sqrt(5×10²¹) = sqrt(5) × sqrt(10²¹) = 2.236 × 3.162×10¹⁰ = 7.07×10¹⁰. So N_opt ≈ 70.7 billion parameters. Then D_opt = 20 × N_opt = 20 × 7.07×10¹⁰ = 1.414×10¹² ≈ 1.41 trillion tokens. These derived numbers land almost exactly on the actual published Chinchilla configuration — 70B parameters, 1.4T tokens — which is a useful sanity check that the 6ND / 20:1 approximations are doing real work, not just producing plausible-looking round numbers.

Now contrast this with the pre-Chinchilla instinct. Suppose, hypothetically, you had this same 6×10²³ FLOP budget but — following the older habit of fixing dataset size and pouring extra compute into parameters, the way GPT-3's 175B-on-300B-tokens recipe did — you fixed D at 3×10¹¹ (300 billion) tokens and let N absorb the rest of the budget:

N = C / (6D) = 6×10²³ / (6 × 3×10¹¹) = 6×10²³ / 1.8×10¹² ≈ 3.33×10¹¹

That is a 333-billion-parameter model — nearly five times larger than the compute-optimal 70.7B — trained on the same 300 billion tokens, giving a ratio of only 0.9 tokens per parameter, twenty-two times below the compute-optimal ratio of 20. Chinchilla's central finding was exactly this: essentially every large model trained before 2022 (GPT-3, Gopher, MT-NLG) sat in this regime — oversized for the data it was shown, spending compute on parameters that were undertrained rather than on more examples those parameters could have learned from.

Common misconception: "bigger model, always better"

The intuitive belief students carry from Grade 11's deep learning unit — that more parameters straightforwardly means more capability — is only true if the extra parameters are matched with enough data to train them. At fixed compute, this is false: a bigger model trained on proportionally less data can perform worse than a smaller model trained compute-optimally, because oversized parameters that never see enough tokens remain undertrained dead weight. The correct mental model is not "N determines quality" but "N and D jointly determine quality, and at any fixed compute budget there is a specific, computable N that is too big just as surely as there is one that is too small." Scale is a two-dimensional resource allocation problem, not a single dial.

Data at scale: from raw crawl to training tokens

Where does D actually come from? The dominant source for general-purpose pre-training corpora is Common Crawl, a continuously updated, petabyte-scale scrape of the public web, refined into training-ready datasets like C4, RedPajama, and (in 2024) FineWeb. Raw crawl data cannot be fed to a model directly — it is dominated by boilerplate, duplicated pages, machine-generated spam, and non-linguistic content — so every serious pre-training pipeline applies a filtering cascade before a single token is tokenized:

• Language and format filters discard pages in the wrong language or malformed HTML/markup remnants.
• Deduplication, typically via MinHash locality-sensitive hashing at the document or paragraph level, removes near-identical text — search-engine-optimized page farms and mirror sites can otherwise inflate a token's effective frequency by orders of magnitude, wasting compute re-teaching the model something it already learned.
• Quality classifiers, trained to distinguish crawl text from a curated reference set (Wikipedia, licensed books, edited web text), downweight or drop low-quality pages — this is the technique GPT-3's dataset construction used to filter Common Crawl.

Tokenization then converts the surviving text into the integer sequences models actually train on, almost always via byte-pair encoding (BPE): starting from individual bytes, the most frequent adjacent pair is repeatedly merged into a new vocabulary entry until a target vocabulary size (commonly 32,000–100,000+) is reached. Because BPE vocabularies are built by frequency on the training mix, and that mix is historically English-dominated, non-Latin scripts fragment far more per unit of text — a well-documented inefficiency for Indic languages, which is precisely why AI4Bharat's Sangraha corpus — a large-scale pre-training dataset spanning around two dozen scheduled Indian languages, released in 2024 — matters beyond being "more data": it exists because a model that has never seen enough Devanagari, Tamil, or Bengali text cannot be tokenization-efficient in those scripts no matter how good its English-derived BPE merges are.

import math

def estimate_tokens(char_count, chars_per_token):
    return math.ceil(char_count / chars_per_token)

# Illustrative chars-per-token ratios (approximations, not a live tokenizer):
# English text compresses well under BPE trained on English-heavy corpora;
# Devanagari text, built from multi-byte UTF-8 sequences the tokenizer's
# merge rules were not optimized for, compresses far less.
char_count = 4000
english_tokens = estimate_tokens(char_count, 4.0)
hindi_tokens = estimate_tokens(char_count, 1.8)

print(english_tokens, hindi_tokens)
# 1000 2223

Tracing this: 4000 / 4.0 = 1000.0, ceiling 1000. 4000 / 1.8 = 2222.22, ceiling 2223. Same underlying content, rendered in a script the tokenizer handles poorly, costs 2223/1000 ≈ 2.22× as many tokens — and since C = 6ND, that is 2.22× the training compute to cover equivalent content, at fixed N. Tokenization efficiency is not a cosmetic detail; it is a direct multiplier on the compute bill, which is exactly why multilingual and Indic-focused labs treat tokenizer design as a scaling-law problem, not an afterthought.

Compute at scale: fitting a model in GPU memory

Knowing N_opt tells you how big a model to build; it does not tell you it will fit on your hardware. Mixed-precision training with the Adam optimizer — the standard recipe — keeps, per parameter: 2 bytes for the fp16 weight, 2 bytes for the fp16 gradient, and in fp32: a 4-byte master weight copy, a 4-byte first-moment (momentum) buffer, and a 4-byte second-moment (variance) buffer. Total: 2+2+4+4+4 = 16 bytes per parameter just for model and optimizer state — the figure documented in Rajbhandari et al., "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models" (Microsoft, 2020) — before counting the activation memory needed for backpropagation.

For our compute-optimal N_opt ≈ 7.07×10¹⁰ parameters: 7.07×10¹⁰ × 16 bytes = 1.131×10¹² bytes ≈ 1131 GB. A single NVIDIA A100 GPU carries 80 GB of memory, so 1131/80 ≈ 14.1 — at least 15 GPUs just to hold optimizer state, before a single activation is stored. This is why no large model trains on one accelerator: data parallelism replicates the full model on every GPU and averages gradients across replicas; tensor parallelism (Shoeybi et al., "Megatron-LM," 2019) splits individual weight matrices across GPUs so no single device holds a whole layer; pipeline parallelism (Huang et al., "GPipe," 2019) assigns different layers to different GPUs and streams micro-batches through them in sequence; and ZeRO itself partitions the 16-bytes-per-parameter optimizer state across the data-parallel replicas instead of duplicating it on every one, turning that 1131 GB from "1131 GB on every GPU" into "1131 GB divided across the cluster."

Training time follows from throughput. An A100 peaks at 312 TFLOP/s (3.12×10¹⁴ FLOP/s) in dense bf16 tensor-core math, but real training runs never hit peak — communication between GPUs, memory stalls, and non-matmul operations all cost cycles. Model FLOPs utilization (MFU) of 40–50% is considered good; Google's PaLM paper (Chowdhery et al., 2022) reported 46.2% MFU on a 540B model as a notable systems achievement. Take a round 40% MFU on a 1000-GPU A100 cluster:

per-GPU achieved = 312e12 * 0.40 = 124.8e12 FLOP/s
cluster achieved  = 124.8e12 * 1000 = 1.248e17 FLOP/s
time = C / cluster achieved = 6e23 / 1.248e17 ≈ 4.808e6 s

4.808×10⁶ seconds ÷ 3600 ≈ 1335.5 hours ÷ 24 ≈ 55.6 days — a bit under two months, on a thousand-GPU cluster, purely to spend the compute this section's worked example called for. This is the number that turns a scaling-law calculation into a budget line: FLOPs translate into GPU-hours, GPU-hours translate into rental cost, and the Chinchilla ratio tells you how to not waste any of it.

Active recall

Attempt each question before reading its answer.

1. Given a compute budget C = 1.2×10²⁴ FLOPs and the Chinchilla ratio D ≈ 20N, find N_opt and D_opt.
2. Using the same 6×10²³ FLOP budget as the worked example, but fixing D at 300 billion tokens (the older, Kaplan-era instinct), what N results, and what token-to-parameter ratio does that imply?
3. The worked example found N_opt ≈ 7.07×10¹⁰ needs about 1131 GB of memory just for weights and optimizer state, requiring at least 15 A100 GPUs (80 GB each) purely to fit that state. If you instead used 40 GB A100s, how many would you need at minimum for the same purpose?
4. Suppose the compute budget from the worked example quadruples to C = 2.4×10²⁴ FLOPs (still on the same 1000-GPU, 40%-MFU cluster). Find the new N_opt, D_opt, the new minimum GPU count for optimizer/weight state (16 bytes/param, 80 GB GPUs), and the new training wall-clock time.
5. Explain in one or two sentences why Chinchilla's finding is N_opt ∝ C^0.5 and D_opt ∝ C^0.5 — equal exponents — rather than, say, N growing faster than D as compute increases.
6. Using the tokenization snippet's ratios (4.0 chars/token English, 1.8 chars/token Hindi), if you want Hindi-language pre-training coverage semantically equivalent to 500 billion English tokens, how many actual Hindi tokens are needed, and — holding the compute-optimal N ≈ 7.07×10¹⁰ from the worked example fixed — how much more FLOPs does that Hindi portion cost than the English-equivalent portion?

Answers.

1. C = 120N² ⇒ N = sqrt(C/120) = sqrt(1.2×10²⁴/120) = sqrt(1×10²²) = 1×10¹¹ = 100 billion parameters. D = 20N = 2×10¹² = 2 trillion tokens. (Check: 1.2×10²⁴ is 2× the worked example's 6×10²³, and sqrt(2) ≈ 1.414, so N should be ≈1.414 × 7.07×10¹⁰ ≈ 1.0×10¹¹ — matches. Always recompute from the formula rather than eyeballing a multiple.)

2. N = C/(6D) = 6×10²³/(6×3×10¹¹) = 6×10²³/1.8×10¹² ≈ 3.33×10¹¹ (333 billion parameters). Ratio D/N = 3×10¹¹/3.33×10¹¹ ≈ 0.9 tokens per parameter — about 22× below the compute-optimal ratio of 20, meaning this model would be severely oversized and undertrained for its compute budget, the exact failure mode Chinchilla identified in the pre-2022 generation of large models.

3. Total memory need is unchanged at ≈1131 GB (it depends only on N, not on GPU choice). 1131/40 ≈ 28.3, so at least 29 GPUs of 40 GB each — roughly double the 15 needed with 80 GB GPUs, as expected since halving per-device capacity roughly doubles the device count required to hold the same total state.

4. N_opt = sqrt(2.4×10²⁴/120) = sqrt(2×10²²) ≈ 1.414×10¹¹ (141.4 billion parameters). D_opt = 20 × 1.414×10¹¹ ≈ 2.828×10¹² (2.83 trillion tokens). Memory: 1.414×10¹¹ × 16 bytes = 2.263×10¹² bytes ≈ 2263 GB — exactly double the original 1131 GB, since N exactly doubles; on 80 GB GPUs, 2263/80 ≈ 28.3 → at least 29 GPUs minimum just for state (not quite double the original 15, since the ceiling operation rounds 28.3 up to 29 rather than to an exact 30). Training time: C/cluster throughput = 2.4×10²⁴/1.248×10¹⁷ ≈ 1.923×10⁷ s ÷ 86400 ≈ 222.6 days — four times the original 55.6 days, because wall-clock time scales linearly with C at fixed cluster throughput, while N and D each scale only as C^0.5 (doubling under a 4× compute increase). Every one of these four downstream quantities — N, D, GPU count, and time — shifts when C changes, and each shifts by its own rule, not uniformly.

5. The IsoFLOP curves are symmetric in what limits them: a model that is too small is capacity-limited (it cannot extract more from unlimited data), and a model that is too small on data is data-limited (excess parameters sit undertrained) — these two failure modes are mirror images of each other, so the compute-optimal point balances them by growing both resources at the same rate as C grows; if one grew faster than the other, the imbalance would eventually recreate exactly the oversized-or-undertrained failure the frontier is defined to avoid.

6. Ratio of tokens needed: (1/1.8)/(1/4.0) = 4.0/1.8 ≈ 2.222. Hindi tokens required = 500×10⁹ × 2.222 ≈ 1.111×10¹² (1.11 trillion tokens). FLOPs at fixed N = 7.07×10¹⁰: C_english = 6 × 7.07×10¹⁰ × 5×10¹¹ ≈ 2.121×10²³ FLOPs; C_hindi = 6 × 7.07×10¹⁰ × 1.111×10¹² ≈ 4.713×10²³ FLOPs. Ratio 4.713/2.121 ≈ 2.22 — matching the token ratio exactly, since C is linear in D at fixed N. The Hindi-equivalent portion costs about 2.22× the compute of the English-equivalent portion for the same underlying content, purely from tokenizer inefficiency.

Think About It

Think about this: How would you explain pre-training at scale: data, compute, and scaling laws to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where pre-training at scale: data, compute, and scaling laws is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting pre-training at scale: data, compute, and scaling laws to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind pre-training at scale: data, compute, and scaling laws, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.