Scaling Laws: Understanding Model and Data Relationships

A national compute grant — the kind the IndiaAI Mission or an AIRAWAT-class supercomputing allocation hands out to a university lab or a startup like Sarvam AI — does not arrive as "build a good model." It arrives as a fixed number of GPU-hours on a fixed cluster. Multiply GPU-hours by the chip's throughput and by the fraction of that throughput your training loop actually achieves, and you get a hard ceiling: a compute budget C, measured in floating-point operations, that will not grow no matter how the training run goes. Before a single line of a training script is written, someone has to decide how to spend it: pour it into a bigger network with more layers and wider attention heads, or pour it into more passes over a larger, more carefully deduplicated crawl of Hindi, Tamil, and English text. Both choices burn identical GPU-hours. They do not produce equally good models. This chapter is about the mathematics that tells you, in advance, which choice wins — and by how much.

Turning GPU-hours into a constraint: C ≈ 6ND

Let N be the parameter count of a transformer and D the number of training tokens it will see. To connect these to compute, count floating-point operations per token. In a dense transformer, almost every parameter participates in exactly one multiply-add per token during the forward pass — each weight matrix does a matrix-vector product against the token's activation, and a multiply-add is two FLOPs. That gives roughly 2N FLOPs per token for the forward pass. Backpropagation needs a gradient with respect to both the activations and the weights for every operation the forward pass performed, which costs about twice the forward pass's FLOPs: 4N. Add the two and one token costs about 6N FLOPs to train on. Over D tokens:

C ≈ 6 · N · D

This is a leading-order count — it treats the network as dominated by matrix multiplications and ignores lower-order terms like embedding lookups and the attention mechanism's own O(sequence-length) cost, which matter only when context windows get very long relative to the model's width. Kaplan et al. (2020) introduced this approximation and Hoffmann et al. (2022) reused it; it is accurate enough that every result in this chapter rests on it, and every "compute budget" from here on is really this product.

The constraint C = 6ND is the whole trade-off in one line: for a fixed C, choosing N automatically chooses D = C/(6N). Push parameters up and tokens must come down. The question "how should I split my compute?" is really the question "which point on this seesaw minimizes the loss?"

The loss law: L(N, D) = E + A/N^α + B/D^β

Hoffmann et al. (2022) — the DeepMind team behind the model Chinchilla — trained over 400 transformers ranging from 70 million to more than 16 billion parameters on between 5 and 500 billion tokens, and fit a single parametric law to the resulting pretraining losses:

L(N, D) = E + A / N^α + B / D^β

Each term has a distinct physical meaning, and it is worth separating them before touching a single number:

• E — the irreducible entropy of natural text under the model's tokenizer. Language carries genuine uncertainty (multiple grammatical continuations, multiple valid word choices); no model, however large, however much data it sees, drives the loss below this floor. It is the asymptote as N, D → ∞.
• A/N^α — capacity error. A model with too few parameters cannot represent the true distribution of language no matter how much data it is shown; this term shrinks as N grows and vanishes only as N → ∞.
• B/D^β — estimation error. Even an infinitely expressive model, shown only a finite slice of text once or twice, has not seen enough examples to pin down its parameters exactly; this term shrinks as D grows.

The two error terms are additive and separate because they are different failure modes. A 200-billion-parameter model trained on 2 billion tokens is undertrained (D-starved) even though it has enormous capacity; a 200-million-parameter model trained on 2 trillion tokens is underparameterized (N-starved) even though it has seen almost everything there is to see. One loss surface, two independent knobs.

The exponents and coefficients themselves come from curve-fitting real training runs. This chapter uses the corrected fit published by Besiroglu, Erdil, Barnett, and You (2024) in their replication study "Chinchilla Scaling: A Replication Attempt" — they found that Hoffmann et al.'s own originally reported fit for this specific parametric approach was internally inconsistent with the paper's other two estimation methods, re-extracted the underlying data, and produced revised constants:

E = 1.8172   A = 482.01   B = 2085.43   α = 0.3478   β = 0.3658

This is a useful thing to sit with before doing any arithmetic: even a landmark, heavily cited paper's numeric fit needed an independent replication to pin down, and got revised — that is normal science, not a scandal, and the qualitative conclusion below survived the correction untouched.

The IsoFLOP method: deriving the compute-optimal frontier

Substitute the compute constraint D = C/(6N) directly into the loss law. For one fixed C, loss becomes a function of N alone:

L(N | C) = E + A·N^(−α) + B·(6N/C)^β

As N → 0 (which forces D → ∞ to hold C fixed), the first error term explodes — a vanishingly small model, however much text it sees, cannot express the distribution. As N → ∞ (forcing D → 0), the second error term explodes — a huge model with almost no data to fit it is estimation-starved. In between, the curve dips through a minimum. Plotted against log N at fixed C, this traces out a shape that looks like a parabola: an "IsoFLOP curve," because every point on it costs the same compute. Sweep N across a grid, hold C fixed, and the loss values trace exactly this bowl. This is Hoffmann et al.'s Approach 2, and it is the method every subsequent scaling-law paper — including the 2024 replication — has had to grapple with, because fitting a smooth curve to noisy, discrete training runs is where the real methodological difficulty lives.

Two ways to find the bottom of the bowl:

Analytically. Differentiate L(N | C) with respect to N, set it to zero, and solve. Carrying through the algebra (differentiate each term, equate, collect powers of N) gives a closed form:

N*(C) = [αA / (βB · 6^β)]^(1/(α+β)) · C^(β/(α+β))
D*(C) = C / (6·N*(C))

With the fitted constants above, β/(α+β) = 0.3658/0.7136 ≈ 0.5126 and its complement α/(α+β) ≈ 0.4874. Both exponents are close to one half — meaning the optimal policy grows N and D at nearly the same rate as C increases. That near-symmetry, not any single number, is the actual Chinchilla result.

Numerically, from real measurements. In practice you cannot differentiate a set of measured losses — you can only train a handful of models at different N for a fixed C, measure their losses, and fit a smooth curve through the discrete points. A quadratic in log N is the standard local approximation to a bowl-shaped curve. This is exactly Approach 2 as it is actually executed, and it is worth tracing in full because it is the piece a purely symbolic treatment skips.

Fully worked example: fitting one IsoFLOP curve

Fix a modest, lab-scale budget: C = 6 × 10^20 FLOPs. Sweep eight candidate model sizes, compute the token count each one forces via D = C/(6N), and evaluate the loss law at each point:

N            D              tokens/param    L(N,D)
2.00e+08     5.000e+11        2500.00       2.55189
4.00e+08     2.500e+11         625.00       2.44960
8.00e+08     1.250e+11         156.25       2.38512
1.60e+09     6.250e+10          39.06       2.35495
3.20e+09     3.125e+10           9.77       2.35764
6.40e+09     1.562e+10           2.44       2.39378
1.28e+10     7.812e+09           0.61       2.46602
2.56e+10     3.906e+09           0.15       2.57927

Loss falls, bottoms out somewhere between the 1.6-billion and 3.2-billion-parameter rows, then rises again — the bowl, in eight discrete samples. Fit a parabola to these eight points in log₁₀(N):

import numpy as np

E, A, B, alpha, beta = 1.8172, 482.01, 2085.43, 0.3478, 0.3658

def L(N, D):
    return E + A / N**alpha + B / D**beta

C = 6e20  # fixed compute budget, FLOPs
grid_N = [2e8, 4e8, 8e8, 1.6e9, 3.2e9, 6.4e9, 1.28e10, 2.56e10]

rows = []
for N in grid_N:
    D = C / (6 * N)          # compute constraint: C = 6ND
    rows.append((N, L(N, D)))

x = np.log10([n for n, _ in rows])
y = np.array([loss for _, loss in rows])
a, b, c = np.polyfit(x, y, 2)      # fit a parabola in log10(N)
N_vertex = 10 ** (-b / (2 * a))
L_vertex = np.polyval([a, b, c], np.log10(N_vertex))

print(f"IsoFLOP-fit optimum: N* = {N_vertex:.3e}, L* = {L_vertex:.4f}")
# IsoFLOP-fit optimum: N* = 2.107e+09, L* = 2.3504

The vertex of the fitted parabola lands at N* ≈ 2.107 × 10^9 parameters. Compare that against the exact analytic optimum computed directly from the closed form above at this same C: N* = 2.138 × 10^9, D* = 4.677 × 10^10, L* = 2.3521. The eight-point curve fit lands within 1.5% of the true continuous minimum — this is precisely why the method works with only a handful of expensive training runs per budget: you never need to train at the exact optimal size, you need enough points to see the bowl and let the parabola interpolate the bottom for you. At the true optimum, D*/N* ≈ 46.77B/2.138B ≈ 21.9 tokens per parameter.

The ratio, across many budgets

Repeat the same analytic optimization at a spread of compute budgets, and the compute-optimal frontier — the curve connecting every IsoFLOP bowl's vertex — emerges:

C (FLOPs)   N* (params)   D* (tokens)   D*/N*   L*
1.0e+19     0.26 B          6.4 B       24.25   2.927
1.0e+20     0.85 B         19.5 B       22.88   2.553
1.0e+21     2.78 B         60.0 B       21.59   2.306
1.0e+22     9.05 B        184.3 B       20.37   2.141
1.0e+23    29.45 B        566.0 B       19.22   2.032
5.76e+23    72.25 B      1,328.7 B      18.39   1.974
1.0e+24    95.86 B      1,738.6 B       18.14   1.960

At Gopher-scale compute (C ≈ 5.76 × 10^23, matching Gopher's own 280B-parameter, 300B-token training run), the frontier's optimum sits at roughly 72B parameters and 1.33 trillion tokens — this is, to within rounding, exactly the Chinchilla result: a 70-billion-parameter model trained on about four times Gopher's token count, which Hoffmann et al. built — matching Gopher's own training compute — and showed uniformly outperforms Gopher, GPT-3 (175B), Jurassic-1, and Megatron-Turing NLG on downstream evaluations, reaching 67.5% on MMLU — a jump of more than 7 points over Gopher.

Contrast this with the earlier prescription from Kaplan et al. (2020), whose scaling-law analysis of an earlier generation of models concluded that compute should be split unevenly — parameters scaling as C^0.73, tokens as only C^0.27. A tenfold increase in compute, on Kaplan's rule, should grow the model 5.4× while growing the data only 1.9×. That prescription is what justified GPT-3's own recipe: 175 billion parameters trained on 300 billion tokens — a ratio of 1.7 tokens per parameter, wildly below the frontier's roughly-20-per-parameter target at any comparable compute scale. Plugging N = 175 × 10^9 back into the closed form above, the compute-optimal token count for a model that size is about 3.08 trillion tokens — GPT-3 used under a tenth of that. Under the newer accounting, GPT-3 was not too small; it was catastrophically data-starved for its own size, spending compute on parameters the data available to it could not properly fit.

Misconception: "20 tokens per parameter" is not a universal constant

The single most common way students misuse the Chinchilla result is to treat "20 tokens per parameter" as a fixed design rule — pick any N, multiply by 20, that is your token target, done. The frontier table above already contradicts this: the ratio D*/N* drifts from about 24 at C = 10^19 down to about 18 at C = 10^24. It is not constant; it moves slowly with compute. This is not an artifact of rounding — it follows directly from the algebra. Because N* and D* scale as C raised to two different exponents (β/(α+β) and α/(α+β)), their ratio itself scales as a power of C:

D*/N* ∝ C^((α−β)/(α+β))

Here (α−β)/(α+β) = (0.3478 − 0.3658)/0.7136 ≈ −0.025 — a small negative exponent, which is exactly why the ratio creeps downward as compute grows rather than sitting still. "Twenty" is a reasonable midpoint across the compute range Hoffmann et al. actually trained models in (roughly 10^19 to a few times 10^23 FLOPs); it is a consequence of this particular α, β pair, not a law of nature. Change the data mixture, the tokenizer, or the architecture family, and α and β refit to new values — and the ratio moves with them, as the next section's numbers show directly.

Beyond the training-loss optimum: when it is rational to overtrain

Everything above optimizes one quantity: pretraining loss per unit of training compute. It says nothing about what happens after training ends. A deployed model pays a fresh compute cost — roughly 2N FLOPs per generated token, no backward pass required — every single time a user sends it a query, for as long as it stays in production. For a model served billions of times, that lifetime inference cost can dwarf the one-time training cost, and the Chinchilla-optimal N stops being the cost-optimal choice.

Sardana, Portes, Doubov, and Frankle (2024), in "Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws," trained 47 models to fold expected inference volume directly into the optimization objective, and found that once serving demand is large enough, the optimal policy shifts toward a smaller N trained on proportionally more data than the pure training-loss frontier calls for — because shrinking the model cuts the per-query inference bill for every future request, and that saving compounds across a model's entire deployed lifetime in a way training-time savings never do.

Meta's Llama 3 gives a concrete, publicly documented illustration. Both the 8-billion and 70-billion-parameter models were pretrained on the same corpus of more than 15 trillion tokens. For the 8B model, that is 15,000B / 8B = 1,875 tokens per parameter. Plugging N = 8 × 10^9 into the closed-form frontier above gives a training-loss-optimal token count of about 164 billion tokens (a ratio of roughly 20.5, right on the Chinchilla midpoint) — meaning Llama 3 8B was trained on roughly 91 times more data than the point that minimizes training loss per training FLOP. That is not a mistake; it is the inference-aware optimum in action. Meta anticipated running this model at enormous scale, so it paid extra GPU-hours once, during training, to buy a permanently smaller — and permanently cheaper to serve — model. Chinchilla-optimal and deployment-optimal are two different questions, and a serious scaling-law analysis has to know which one it is actually answering.

Active recall

Attempt each question before reading its answer. Use E = 1.8172, A = 482.01, B = 2085.43, α = 0.3478, β = 0.3658 and C = 6ND throughout.

1. Why does L(N, D) need two separate power-law error terms rather than one term written directly in terms of total compute C?
2. At a fixed budget C = 10^21 FLOPs, which produces lower loss: a 1-billion-parameter model or a 5-billion-parameter model? Show the token counts each implies.
3. Suppose better data curation effectively raises β from 0.3658 to 0.40, holding α, A, B, E fixed. Does the compute-optimal split shift more toward parameters or more toward tokens? Does the tokens-per-parameter ratio drift faster or slower with compute than before?
4. If your compute budget doubles, by what factor should N alone grow, and by what factor should D alone grow, under the compute-optimal policy derived above?
5. A company plans to serve a model to 50 million users, each generating hundreds of queries a month, for several years, on a fixed training budget. Should they train at the Chinchilla training-loss optimum for that budget, or deliberately choose a smaller model trained on more tokens? Name the paper that formalizes this choice.
6. True or false: any model trained at roughly 20 tokens per parameter is, by definition, compute-optimal.

Answers

1. Because the two failure modes are logically independent. A model can be starved of capacity while swimming in data (huge D, tiny N) or starved of data while overflowing with capacity (huge N, tiny D); these are different regimes with different causes, so the loss law needs one term that depends only on N (vanishing as N → ∞ regardless of D) and one that depends only on D (vanishing as D → ∞ regardless of N). Collapsing both into a single function of C = 6ND would erase the very distinction the whole optimization is about — for a fixed C, N and D are not interchangeable, which is exactly why there is an optimal split to find in the first place.
2. At N = 1 × 10^9: D = C/(6N) = 1.667 × 10^11 tokens, L = 2.3381. At N = 5 × 10^9: D = 3.333 × 10^10 tokens, L = 2.3163. The 5-billion-parameter model wins — it sits closer to this budget's true compute-optimal point, N* ≈ 2.78 × 10^9, L* = 2.3055, which lies between the two candidates but nearer the larger one on the log scale. Neither candidate reaches the true optimum; both are informative brackets around it, exactly as a grid search would produce in practice.
3. Ripple through the whole chain, not just the obvious term. Raising β changes the N-exponent a = β/(α+β) from 0.3658/0.7136 ≈ 0.5126 to 0.40/0.7478 ≈ 0.5349 — larger, so a bigger share of any additional compute now goes to parameters, and correspondingly the D-exponent b = α/(α+β) falls from 0.4874 to 0.4651. The split shifts toward N. The ratio-drift exponent (α−β)/(α+β) also changes, from −0.025 to (0.3478−0.40)/0.7478 ≈ −0.070 — more negative, so the tokens-per-parameter ratio now falls faster as compute grows than it did before. Both consequences point the same way: better-curated data (higher β, meaning each additional token buys more loss reduction) makes it optimal to lean harder into parameters and lighter into raw token count as budgets scale up — the opposite of the naive intuition that better data should mean "just use more of it."
4. Multiply N by 2^0.5126 ≈ 1.427 and D by 2^0.4874 ≈ 1.402. Neither is doubled; both grow by roughly 40%. Check: 1.427 × 1.402 ≈ 2.00, confirming the two multipliers must multiply to exactly the compute multiplier, since the two exponents sum to 1 by construction.
5. Choose the smaller model trained on more tokens. At 50 million users generating hundreds of queries monthly for years, cumulative inference FLOPs will dwarf the one-time training FLOPs, so the objective to minimize is lifetime cost, not training loss alone — exactly the regime Sardana, Portes, Doubov, and Frankle's "Beyond Chinchilla-Optimal" (2024) analyzes, and exactly the logic behind Llama 3 8B's 91x-beyond-Chinchilla token count.
6. False. Twenty is the ratio at the compute-optimal point only for this particular fitted (α, β) pair, and only near the compute scales the fit was built from — the frontier table shows it sliding from about 24 down to about 18 across four orders of magnitude of compute, and question 3 shows it shifts further if the underlying exponents change. It is also silent on inference cost: a model can sit exactly on the 20-tokens-per-parameter line and still be the wrong choice once deployment volume is accounted for, per question 5.

Think About It

Think about this: How would you explain scaling laws: understanding model and data relationships to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind scaling laws: understanding model and data relationships, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.