Scaling Laws in Deep Learning

In 2023, Meta released LLaMA, a family of language models built on a deliberately unusual choice. Their 7-billion-parameter model was trained on roughly one trillion tokens — a ratio of about 143 tokens per parameter. If you have already studied the Chinchilla scaling laws, that number should look wrong. Hoffmann et al. (2022) established that for a fixed training-compute budget, loss is minimized when parameter count and training tokens grow in roughly equal proportion, landing near 20 tokens per parameter. LLaMA-7B was trained on more than seven times that ratio. Either Meta's researchers ignored the most influential scaling-laws paper of the decade, or "compute-optimal" was answering a narrower question than it appeared to. It was the latter. Chinchilla optimizes training compute. It says nothing about what happens after training ends — and for any model that gets deployed, what happens after training is where almost all of the compute actually gets spent. This chapter builds the missing half of the picture: how scaling laws change once you account for a model's entire operational lifetime, not just the run that created it.

Recap: the loss surface you already know

You have seen the Chinchilla loss surface before, so this is a reminder, not a re-derivation. For a dense transformer, expected pretraining loss as a function of parameter count N and training tokens D is well approximated by a two-term power law:

L(N, D) = E + A / N^alpha + B / D^beta

E is the irreducible entropy of natural text that no model can predict away; the two power-law terms capture the loss you still pay for having finite parameters and finite data respectively (Hoffmann et al., 2022). Training compute is well approximated by C_train ≈ 6ND FLOPs (Kaplan et al., 2020): roughly 2N FLOPs per token for the forward pass, and about twice that again for backpropagation, giving 6N FLOPs per token, times D tokens. Minimizing L(N,D) subject to a fixed C_train is a constrained optimization with a closed-form solution, and it is the origin of the famous result: at the compute scales labs were training at in 2022, the loss-minimizing split was close to 20 tokens for every parameter. That is the calculation the sibling chapters in this course walk through in full. What we do here is ask what changes when "minimize compute" no longer means "minimize the compute of the training run" — because for a model anyone will actually use, it doesn't.

The missing term: compute you pay after training ends

Picture an Indian startup building a support-chat model for a logistics platform at Flipkart or Swiggy scale — millions of customer queries a day, in Hindi, Tamil, Bengali, and English, every day for the two or three years the model stays in production before it gets replaced. The training run happens once. The inference doesn't. By the model's first anniversary, it may have generated far more tokens answering customers than it ever read during pretraining. Any accounting of "compute" that stops at the end of the training run is only counting the smaller of the two bills.

Call the number of tokens the model will generate over its deployed lifetime D_inf. Autoregressive generation needs only a forward pass — no backward pass, no gradient storage — so serving one token costs approximately 2N FLOPs, a third of what that same token would have cost during training. The model's total lifecycle compute is:

C_total = C_train + C_infer
        = 6*N*D_train  +  2*N*D_inf

Look at how N enters each term. In the training term it multiplies D_train, a quantity you choose once and then stop paying for. In the inference term it multiplies D_inf, a quantity that keeps accumulating for as long as the product exists and keeps getting multiplied by whatever N you committed to at launch. Every parameter you add to the model is a parameter you pay for on every single token anyone ever sends it. That asymmetry — one-time cost proportional to N·D_train, recurring cost proportional to N·D_inf — is the entire mechanism this chapter is about. It is also exactly the term Chinchilla's optimization leaves out, because Chinchilla fixes D_inf = 0 without saying so.

Worked example: choosing N when the model will actually be used

Fix a target quality bar and ask a new question: among every (N, D) pair that reaches that quality, which one minimizes total lifecycle compute, given how much inference the model will actually see?

Take a loss surface with the same functional form Hoffmann et al. fit, using illustrative constants chosen only to keep the arithmetic clean — not the paper's own fitted values: E = 1.70, A = 350.0, B = 236.93, alpha = 0.34, beta = 0.28 (the two exponents are the paper's real reported values; the loss-scale constants are stand-ins tuned so the pure-training optimum below lands near the well-known 20-tokens-per-parameter ratio). Fix the quality bar at L* = 1.94.

For a fixed N, the loss equation has exactly one value of D that hits L* — any less and you haven't reached the bar, any more and you've spent tokens you didn't need to reach it. Solving L(N, D) = L* for D:

D(N) = [ B / (L* - E - A/N^alpha) ] ^ (1/beta)

Every point on this curve reaches the identical quality bar with a different (N, D) split. Walking along it and evaluating C_total(N) = 6·N·D(N) + 2·N·D_inf at each point turns "what's the best model size" into a one-variable minimization — easy to scan numerically for any assumed D_inf:

import numpy as np

E, A, B = 1.70, 350.0, 236.93       # illustrative constants, same
alpha, beta = 0.34, 0.28            # functional form as Hoffmann et al. (2022)
L_target = 1.94                     # fixed quality bar (nats)

def tokens_needed(N):
    """Tokens D required to reach L_target at parameter count N."""
    remaining = L_target - E - A / N ** alpha
    return (B / remaining) ** (1 / beta)

def lifecycle_compute(N, D_inf):
    D_train = tokens_needed(N)
    C_train = 6 * N * D_train        # forward + backward, paid once
    C_infer = 2 * N * D_inf          # forward only, paid per served token
    return C_train + C_infer

N_grid = np.linspace(3e9, 45e9, 42001)   # 3B to 45B parameters

for D_inf in [0, 500e9, 2e12, 5e12, 10e12]:
    costs = np.array([lifecycle_compute(N, D_inf) for N in N_grid])
    N_star = N_grid[int(np.argmin(costs))]
    D_star = tokens_needed(N_star)
    print(f"D_inf={D_inf:>10.1e}  N*={N_star/1e9:5.1f}B  "
          f"D_train*={D_star/1e9:7.1f}B  ratio={D_star/N_star:6.1f}")

Running this scan produces:

D_inf=   0.0e+00  N*= 20.9B  D_train*=  422.8B  ratio=  20.2
D_inf=   5.0e+11  N*= 14.6B  D_train*=  632.0B  ratio=  43.2
D_inf=   2.0e+12  N*= 10.3B  D_train*= 1044.7B  ratio= 101.2
D_inf=   5.0e+12  N*=  8.1B  D_train*= 1634.9B  ratio= 202.6
D_inf=   1.0e+13  N*=  6.8B  D_train*= 2402.4B  ratio= 354.8

With no inference workload, the scan recovers the familiar Chinchilla-style answer: about 21 billion parameters, 423 billion tokens, ratio near 20:1 — reassuring, since that is exactly the regime the sibling chapters derive by a different route. But the moment D_inf becomes a serious number — and 2 to 10 trillion served tokens is a modest estimate for a product with millions of daily users over a couple of years — the optimal parameter count collapses, and the token-per-parameter ratio the training-only theory predicted stops applying by more than an order of magnitude.

The compute savings are not a rounding error. Compare each inference-aware optimum against naively deploying the 20.9-billion-parameter model that pure training-compute optimization would recommend, and paying its 2·N·D_inf inference bill anyway:

At the largest scale in this table, choosing the training-compute-optimal model size wastes over half of the total GPU-hours the product will ever consume, for a model that answers the identical quality bar either way. That is the concrete cost of treating "compute-optimal" as if it meant "optimal," full stop, rather than "optimal for the training run alone."

Seeing the shift

The diagram below plots total lifecycle compute against model size for three of these inference workloads. Each curve traces C_total(N) along the same fixed-quality path D(N); the marked minimum on each curve is the row from the table above. As the anticipated inference volume grows from the flat blue curve (train-only) through purple to red, the minimum visibly slides left — toward smaller models — and grows increasingly steep, because more of the total cost is now the recurring 2·N·D_inf term rather than the one-time training term.

The misconception: "bigger is always better"

The natural misreading of scaling laws — the one nearly every student forms on first contact with them — is: "scaling laws prove that more parameters and more compute always produce a better model, so the right strategy is always to train the largest model your budget allows." That statement is not wrong about loss going down with scale. It is wrong about what "your budget" means. Chinchilla-style scaling laws minimize the compute of one specific event: the training run. They say nothing about a model that will be queried by users next year, or the year after, because D_inf does not appear anywhere in the training-compute constraint. The correction is not that bigger models are worse — for a fixed compute budget with no deployment, bigger-and-shorter-trained-versus-smaller-and-longer-trained is exactly the tradeoff Chinchilla resolves correctly. The correction is that "compute" has two very different meanings depending on whether the model ever leaves the training cluster, and a serious production decision has to add the second one back in. Once you do, the loss-minimizing strategy for a widely used product is consistently smaller and more data-hungry than pure training-compute optimization would suggest — which is precisely, numerically, what the table above shows.

This match isn't a coincidence

The LLaMA-7B example that opened this chapter is not an isolated anecdote. Touvron et al. (2023), introducing LLaMA, explain the choice directly: given a target performance level, several different (N, D) combinations can reach it, but the model that is cheapest to run at inference time — the smaller one, trained on more tokens — is the one worth optimizing for, because inference cost recurs for the entire deployed lifetime of the model while training cost is paid once. LLaMA-7B's ratio of roughly 143 tokens per parameter is not a mistake relative to Chinchilla; it is the answer to a different, more complete optimization problem, one that treats D_inf as too large to set to zero. Sardana and Frankle (2024) formalized exactly this extension in "Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws," deriving inference-aware compute-optimal frontiers of the same shape as the one plotted above, and showing the same qualitative result across a range of realistic deployment volumes: once you expect to serve meaningfully more tokens than you trained on, the training-compute-optimal point stops being the total-compute-optimal point, often by a wide margin.

Where the approximation strains

Treat 2N FLOPs per inference token as a first-order model, not the final word. It assumes inference is compute-bound, but production decoding is frequently memory-bandwidth-bound instead: at batch size one, a GPU spends most of its time streaming weights and the growing key-value cache from memory rather than doing arithmetic, so wall-clock latency and dollar cost per token can diverge substantially from a pure FLOPs count. Quantizing a deployed model to int8 or int4 changes its effective per-token cost without changing N at all, and batching multiple users' requests together changes the effective FLOPs-to-memory-traffic ratio again. None of this breaks the argument in this chapter — smaller models are still cheaper to serve under every one of these effects, usually more dramatically than the FLOPs-only estimate suggests, since memory bandwidth scales with parameter count too. But it does mean that a production team choosing a model size from this framework should treat the FLOPs-based lifecycle curve as a first-pass estimate to be refined with an actual serving benchmark on their target hardware, not as a number precise enough to defend to two significant figures.

Active recall

Attempt each question before reading its answer.

1. Why does a larger expected inference volume push the total-compute-optimal parameter count down, even though the quality target stays fixed and unchanged?
2. The startup revises its two-year inference-volume estimate from 10 trillion tokens to 20 trillion tokens (it is expanding to more regional languages). Using the same loss surface as the worked example, what happens to N*, D_train*, the tokens-per-parameter ratio, and total lifecycle compute?
3. Using this chapter's framework, explain in two or three sentences why training LLaMA-7B on roughly 143 tokens per parameter — about seven times the classic 20:1 ratio — was a reasonable engineering decision rather than a departure from scaling laws.
4. Derive, from first principles, why training costs roughly 6N FLOPs per token while inference costs roughly 2N FLOPs per token. What is the ratio, and why does that ratio matter less than the fact that D_inf keeps growing after D_train stops?
5. A classmate says: "Scaling laws prove bigger models are strictly better, so you should always train the largest model your compute budget allows." Give a two-sentence rebuttal.
6. Suppose the fixed quality bar is relaxed from L* = 1.94 to a weaker L* = 2.05. Holding everything else in the worked example unchanged, how does this change ripple through both the train-only optimum (D_inf = 0) and the heavy-inference optimum (D_inf = 10T)? Does the qualitative pattern from Q1 still hold?

Answers

1. Training compute is 6·N·D_train — a one-time charge, paid once regardless of how long the model stays in production. Inference compute is 2·N·D_inf — a recurring charge, paid on every one of D_inf tokens for as long as the model is deployed. Both terms scale linearly in N, but only the inference term keeps accumulating indefinitely. As D_inf grows, that recurring linear-in-N term dominates total lifecycle compute, so reducing N — even at the cost of training on proportionally more tokens to hold quality fixed — reduces the much larger ongoing bill. The quality target never moves; only the cheapest route to it does.

2. Scanning the same loss surface at D_inf = 20 × 10^12 tokens gives N* ≈ 5.77B, D_train* ≈ 3639B tokens, ratio ≈ 630:1, total compute ≈ 3.569 × 10^23 FLOPs. Relative to the 10T scenario (N*=6.8B, D_train*=2402B, ratio 355:1, C_total=2.331×10^23), doubling the inference estimate drops the optimal parameter count by about 15%, raises the required training tokens by about 51%, raises the ratio by about 78% (1.78x), and raises total lifecycle compute by about 53%. Notice that all four quantities move, not just the "obvious" one (N*) — the ratio shift is actually the largest relative change, because it compounds a falling numerator's partner (N shrinking) with a rising denominator's partner (D growing) in the same direction.

3. LLaMA was built to be deployed and queried at scale, including by third parties running it themselves, so its expected D_inf was always going to be enormous relative to its training set. Per this chapter's framework, once D_inf is large, the total-compute-minimizing model at a fixed quality bar is smaller and trained on proportionally more tokens than the pure training-compute-optimal point — exactly the direction and, roughly, the order of magnitude LLaMA-7B's 143:1 ratio moved in relative to the 20:1 Chinchilla baseline.

4. A forward pass does roughly one multiply-add per parameter per token, and each multiply-add is about 2 FLOPs, giving ≈2N FLOPs/token. Backpropagation computes gradients with respect to both the weights and the activations, which costs roughly twice the forward pass again, so training totals ≈2N + 4N = 6N FLOPs/token — three times the per-token cost of inference. That 3:1 ratio matters far less than it looks like it should, because it is a fixed multiplier applied once to D_train, while the 1:1 (in FLOPs-per-token terms) inference cost is applied to D_inf, a quantity with no fixed ceiling — a popular product can generate many times its entire pretraining token count within months of launch.

5. Scaling laws only say "bigger is better" if you're minimizing loss for a training run with no deployment attached — they say nothing about what a model costs to run afterward. Once a model is going to be queried by real users, its true compute bill is dominated by inference, which scales with parameter count on every served token, so the compute-minimizing model at a fixed quality bar is usually smaller and trained longer than the largest model the training budget alone would justify.

6. Relaxing the bar to L* = 2.05 shrinks both endpoints substantially: the train-only optimum falls from N*≈20.9B, D*≈423B (ratio 20:1) to N*≈6.9B, D*≈110B (ratio 16:1), and the heavy-inference optimum falls from N*≈6.8B, D*≈2402B (ratio 355:1) to N*≈1.7B, D*≈1431B (ratio 861:1) — note that this last optimum falls below the worked example's 3-billion-parameter grid floor, so reproducing it needs a wider, domain-safe search (e.g. N_grid = np.linspace(0.5e9, 45e9, ...) together with a guard against the loss formula's invalid domain, since remaining turns negative below N≈0.67B at this L*) rather than the worked example's code exactly as printed. Every absolute value drops — a lower quality bar is cheaper to hit at every inference volume — but the qualitative pattern from Q1 survives untouched: N* still shrinks and the ratio still rises sharply as D_inf grows, at both quality levels. The mechanism is about the relative allocation between N and D for a given deployment load, not about the absolute scale of the quality target, so loosening or tightening L* moves the whole picture up or down without changing its shape.

Think About It

Think about this: How would you explain scaling laws in deep learning to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where scaling laws in deep learning is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting scaling laws in deep learning to at least 3 other topics you have studied.