Scaling Laws: The Mathematical Blueprint Behind GPT-4

The Bet OpenAI Made Before Spending Millions

Training a GPT-4-class model consumes months of cluster time across thousands of GPUs and costs tens of millions of dollars in compute alone. If a single architectural choice or hyperparameter turns out to be wrong at that scale, there is no cheap way to find out until the run has already burned through the budget. In the GPT-4 Technical Report (OpenAI, 2023), the authors describe how they avoided this trap: before committing to the full training run, they trained a family of much smaller models, some using 1,000 to 10,000 times less compute than the final system, and used the pattern in how those smaller models' loss fell with compute to forecast the loss of the model they had not yet built. They used a similar extrapolation to predict specific benchmark performance, including pass rates on the HumanEval coding benchmark, before the full model finished training, and the predictions landed close to the measured results. This chapter reconstructs the mathematics behind that forecast: how a handful of small, cheap training runs can be turned into a quantitative bet on a system that costs a thousand times more to build.

From First Principles: Why Loss Falls as a Power Law

Training loss is usually reported as cross-entropy in nats per token: the average number of nats needed to encode the next token given the model's predicted distribution. As you pour more compute into training, this loss falls, but it never reaches zero. Natural language carries genuine unpredictability: synonyms, ambiguity, facts a model cannot know without more context, multiple valid continuations of a sentence. There is a floor below which no amount of compute pushes the loss, an irreducible entropy that depends on the language, tokenizer, and evaluation distribution, not on model quality.

This is what gives the loss-versus-compute curve its shape: a term that shrinks as compute grows, added to a constant floor it approaches but never crosses. Written as a function of compute C:

L(C) = a · C-α + L∞

Here a and α govern how fast the reducible part of the loss shrinks, and L∞ is the irreducible floor. Kaplan et al. (2020, "Scaling Laws for Neural Language Models") first documented this shape at OpenAI, and Hoffmann et al. (2022, the Chinchilla paper) showed how to split a fixed compute budget between parameters and data optimally within it. Both results are about what to do with a compute budget you already have. This chapter covers a different, more applied question: once you accept the power-law shape, how do you fit its three unknowns from data you can actually afford to collect, and how far can you trust the curve once you extrapolate it a thousandfold past your largest measurement?

Fitting Three Unknowns From Three Runs: The Geometric-Progression Trick

A pure power law, y = a · x-α, is easy to fit: take logarithms of both sides and it becomes a straight line, log y = log a - α log x, solvable by ordinary linear regression. The loss curve above is not that simple, because of the added constant L∞. log(a · C-α + L∞) does not collapse into anything linear in log C, so naive log-log regression silently fits the wrong curve whenever L∞ is not zero, which for language modeling it never is.

One way around this, useful for building intuition before reaching for a numerical optimizer, uses training runs spaced in a geometric progression of compute: u, u·r, u·r² for some ratio r. Taking successive differences of the measured losses cancels the constant floor:

D₁ = L(u) - L(ur) = a·u-α(1 - r-α)
D₂ = L(ur) - L(ur²) = a·u-αr-α(1 - r-α)

Dividing the two, everything except rα cancels: D₁ / D₂ = rα. Because r is a design choice, this single ratio hands you α in closed form, and a and L∞ follow by back-substitution. It is the same idea used to estimate a rate constant from three unevenly-but-geometrically spaced measurements in an exponential-decay problem, applied here to a loss curve instead of a decaying quantity.

Worked Example: Forecasting a Run 4096× Larger

The exact coefficients OpenAI fit for GPT-4 are not public. What follows is a from-scratch reconstruction of the method using illustrative numbers chosen so every step can be checked by hand; treat the specific loss values as a teaching example, not as GPT-4's real numbers.

Three small-scale runs are trained at compute budgets in a geometric progression with ratio r = 8, measured relative to the cheapest run (u = 1):

Run A: u = 1, L = 5.5000
Run B: u = 8, L = 2.0000
Run C: u = 64, L = 1.5625

Step 1, find α. D₁ = 5.5000 - 2.0000 = 3.5000. D₂ = 2.0000 - 1.5625 = 0.4375. D₁/D₂ = 3.5000 / 0.4375 = 8.0000. Since D₁/D₂ = rα = 8α and the result is exactly 8, α = 1.

Step 2, find a. D₁ = a·u-α(1 - r-α) = a·1·(1 - 8-1) = a·0.8750. So a = 3.5000 / 0.8750 = 4.0000.

Step 3, find L∞. L∞ = L(u=1) - a·1-α = 5.5000 - 4.0000 = 1.5000.

Step 4, check against the held-out third point. Run C was not used to solve for the parameters, so it is a genuine check. L(64) = 4.0000 · 64-1 + 1.5000 = 0.0625 + 1.5000 = 1.5625, exactly matching the measured value.

Step 5, extrapolate to the target run. Suppose the full-scale system uses u = 4096 times the compute of Run A, comfortably inside the 1,000×-10,000× range the GPT-4 report describes. L(4096) = 4.0000 · 4096-1 + 1.5000 = 0.0009766 + 1.5000 = 1.5009766.

Two things stand out in that last line. First, the predicted loss, about 1.5010, sits barely above the fitted floor of 1.5000: a 4096× jump in compute buys a change in loss of under one thousandth of a nat, a direct, quantitative statement of diminishing returns near the entropy floor. Second, the entire forecast for the expensive run rests on a floor value, L∞, estimated from data that never came close to that floor. Whether that floor estimate is trustworthy is the real risk in this method, and the ripple-effect question at the end of this chapter puts a number on how fragile it can be.

Cross-Checking the Hand Fit With a Numerical Optimizer

In practice nobody solves the three-run system by hand; the same idea is fit with nonlinear least squares over however many training runs a budget allows, using a library like SciPy. The code below defines the power-law loss model, hands it the same three data points from the worked example, and lets the optimizer find the best-fit parameters and the forecast at u = 4096.

import numpy as np
from scipy.optimize import curve_fit

def power_law_loss(u, a, alpha, L_inf):
    return a * np.power(u, -alpha) + L_inf

# u = compute relative to the smallest run (u=1); L = measured loss (nats/token)
u = np.array([1.0, 8.0, 64.0])
L = np.array([5.5, 2.0, 1.5625])

popt, _ = curve_fit(power_law_loss, u, L, p0=[1.0, 0.5, 1.0])
a_fit, alpha_fit, Linf_fit = popt
print(f"a={a_fit:.4f}  alpha={alpha_fit:.4f}  L_inf={Linf_fit:.4f}")

u_target = 4096.0
L_pred = power_law_loss(u_target, a_fit, alpha_fit, Linf_fit)
print(f"predicted loss at u={u_target:.0f}: {L_pred:.4f}")

Because the three data points were generated from the exact model with no measurement noise, and the model has exactly three free parameters, this is a fully determined system with a unique solution and zero residual error at the optimum. The optimizer converges to that solution: a=4.0000 alpha=1.0000 L_inf=1.5000, and the second print statement outputs predicted loss at u=4096: 1.5010, matching the hand derivation above. With real, noisy measurements across dozens of runs, the same call would instead return a least-squares best fit with nonzero residual, which is why production scaling-law work never relies on just three points.

From Loss to Capability: Correcting a Common Misconception

A student who has just seen loss fall smoothly and predictably with compute often assumes the same must be true of every benchmark score: that accuracy on any downstream task should climb along its own tidy power law as compute increases. This is the misconception worth naming directly, because it is exactly backwards for many benchmarks, and the exception is instructive rather than a flaw in the theory.

Cross-entropy loss is a continuous, smooth quantity by construction: it is an average over every token in a large evaluation set, so small improvements average out into a smooth curve. Benchmark accuracy, by contrast, is often a thresholded quantity. A pass/fail metric like exact-match accuracy on a math problem, or a fixed-threshold pass rate on a coding benchmark, only registers as correct once the model's confidence in the right answer crosses some cutoff. A model can be getting steadily better at a task, in the sense that it assigns more and more probability mass to the correct token sequence, while the pass/fail metric stays flat at zero right up until the crossing point, then jumps. Read off a handful of model sizes, and that jump looks like a phase transition, an ability that "emerged" out of nowhere at some critical scale. Schaeffer, Miranda, and Koyejo (NeurIPS 2023, "Are Emergent Abilities of Large Language Models a Mirage?") showed that many reported emergent jumps disappear, replaced by smooth curves, once the metric is changed from a thresholded pass/fail score to a continuous one, such as token-level log-likelihood, measured on the same models.

This is the same tension the GPT-4 report's authors had to solve to predict a benchmark score in advance: fitting a power law directly to HumanEval pass rate, a bounded, thresholded quantity, does not work well, but fitting a power law to a transformed quantity that behaves more like a continuous loss, such as the negative log of the failure rate, can recover a smooth, extrapolable trend even though the raw pass rate looks like it is emerging suddenly. The lesson is not that predictable scaling fails for capabilities; it is that predicting a capability first requires choosing a metric that is actually smooth in the underlying quantity the model is optimizing, and raw pass/fail accuracy usually is not that metric.

Why This Matters Once Real Money Is on the Line

The economic case for fitting these curves before the full run is direct. If the cheapest diagnostic run in a scaling study costs a few hundred dollars of cluster time, and the full run sits 1,000 to 10,000 times higher on the same compute axis the report describes, the full run costs somewhere between roughly a few hundred thousand and a few million dollars. A team that can forecast, with reasonable confidence, whether that spend will land the loss (and by extension, rough capability) where they need it before committing the budget has turned an all-or-nothing bet into something closer to a controlled experiment; a team that skips the forecast finds out only after the money is spent whether the architecture and data mix were sound.

The same predictable-scaling philosophy extends past the loss curve into hyperparameter choice. Yang et al. (2022, "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer") describe a parametrization, maximal update parametrization (μP), under which the optimal learning rate and initialization scale found by a sweep on a small model transfer unchanged to a much larger model in the same family, rather than needing to be re-tuned at full scale. Combined with a loss-versus-compute forecast, this means both "what result should we expect" and "what settings should we use to get it" can be answered largely on cheap models, before the expensive run starts.

None of this is free of risk. A power-law fit made at small scale silently assumes the architecture, data distribution, and training recipe stay fixed as compute grows; change any of those between the small diagnostic runs and the full run (a new tokenizer, a different data mixture, a change in model family) and the extrapolation is fitting the wrong curve without any way to know it from the small-scale data alone. The GPT-4 report itself is explicit that not every capability could be forecast this way, precisely because some benchmarks depend on properties, like specific post-training choices, that a pretraining-loss scaling law does not capture.

The Mechanism, Drawn Out

The diagram below lays out the process used in the worked example: three cheap, measured runs on the left, a power-law curve fit through them, and an extrapolated prediction at 4096× the smallest run's compute, deep inside a region where no model was actually trained.

Active Recall

Attempt each question before reading its answer.

1. Why does ordinary log-log linear regression fail to fit L(C) = a·C-α + L∞ whenever L∞ ≠ 0?
2. Using the fitted model from the worked example (a=4, α=1, L∞=1.5), compute the predicted loss at u = 512.
3. The GPT-4 report predicted loss accurately but had to transform the metric to predict HumanEval pass rate. Why does raw pass rate resist the same direct power-law fit that works for loss?
4. Ripple effect: everything else in the worked example stays the same, except Run B's measured loss is 2.5 instead of 2.0 (Run A stays at 5.5, Run C stays at 1.5625, both at their original u). Recompute α, a, L∞, and the predicted loss at u=4096. What does the size of the change tell you about the method?
5. A team has budget for only two small-scale training runs, not three. Can they still fit all three parameters of the power law? If not, what would they need to assume or borrow from outside their own data to proceed?
6. If the cheapest diagnostic run in a scaling study costs $500 and the full run sits 1,000× to 10,000× higher on the same compute axis, what dollar range should the team budget for the full run?

Answers.

1. Because log(a·C-α + L∞) is the log of a sum, not the log of a pure power, and the log of a sum does not decompose into a linear expression in log C. Log-log regression only produces a straight line when the quantity being fit is a pure power of C, i.e. when L∞=0. Fitting the sum requires either nonlinear regression or a trick, like the difference method above, that removes the constant before taking logs.
2. L(512) = 4 · 512-1 + 1.5 = 0.0078125 + 1.5 = 1.5078125, so about 1.5078.
3. Pass rate is a thresholded, bounded quantity, it saturates between 0 and 1 and only counts a fully correct completion, so a model can be improving steadily in the confidence it assigns the right answer while the pass/fail count stays at zero until a threshold is crossed, then jumps. Loss averages over every token in a large evaluation set and is not thresholded, so it stays smooth. Transforming pass rate into something like the negative log of the failure rate turns a thresholded, bounded quantity back into something closer to a continuous one that a power law can fit.
4. Differences: D₁ = 5.5 - 2.5 = 3.0, D₂ = 2.5 - 1.5625 = 0.9375, D₁/D₂ = 3.2 = 8α, so α = ln(3.2)/ln(8) ≈ 0.5594. Then 8-α ≈ 0.3125, a = 3.0 / (1 - 0.3125) = 3.0/0.6875 ≈ 4.3636, and L∞ = 5.5 - a ≈ 1.1364. Extrapolating: 4096-α = (8-α)4 ≈ 0.31254 ≈ 0.009537, so L(4096) ≈ 4.3636 · 0.009537 + 1.1364 ≈ 1.1780. A single middle measurement moving by 0.5 nats (25% of its own value) swung the fitted floor L∞ from 1.5000 down to about 1.1364, and the extrapolated loss at 4096× compute from 1.5010 down to about 1.1780, a much larger relative change than the original perturbation. This is the practical warning behind the elegant three-point algebra: an exact fit through exactly three points has zero slack to absorb measurement noise, and the parameter it is least forgiving about is L∞, the one furthest, in extrapolation distance, from any point actually measured near it. Production scaling-law work fits many more runs with least squares specifically to damp this sensitivity.
5. No. Three unknowns need at least three independent equations to pin down a unique solution; two points leave one free dimension, so infinitely many (a, α, L∞) triples fit both points exactly. To proceed with two runs, the team has to fix one parameter from outside information, most commonly L∞, estimated independently from the tokenizer's known entropy or from published values for a similar data mixture, and then solve for the remaining two unknowns from the two measured points.
6. $500 · 1,000 = $500,000 at the low end and $500 · 10,000 = $5,000,000 at the high end, so the team should budget somewhere in the $500,000 to $5,000,000 range.

Think About It

Think about this: How would you explain scaling laws: the mathematical blueprint behind gpt-4 to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind scaling laws: the mathematical blueprint behind gpt-4, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.