A hospital network with branches across Pune, Nagpur, and Nashik wants three AI capabilities: transcribe a doctor's spoken dictation into a clinical note, summarize a patient's multi-year history into a one-page brief before a consult, and answer patient questions in a WhatsApp chatbot in Hindi and English. Under the AI paradigm that dominated from roughly 2012 to 2018, this is three separate projects. Transcription needs a speech-to-text model trained on thousands of hours of labeled doctor-audio-to-transcript pairs. Summarization needs a sequence-to-sequence model trained on labeled history-to-summary pairs, probably written by hired medical scribes. The chatbot needs a retrieval or classification model trained on labeled question-answer pairs, again hand-curated. Three datasets, three architectures tuned separately, three training runs, three teams maintaining three models — and if the hospital later wants a fourth capability, say translating discharge summaries into Marathi, that is a fourth project from zero.
This is the paradigm foundation models replaced. A single model — pretrained once on a broad, largely unlabeled corpus of text (and increasingly audio, images, and code) — can be adapted to transcription, summarization, and chat with orders of magnitude less task-specific data and engineering, because the heavy lifting of "understanding language" was already done during pretraining. Bommasani et al. (2021), in the Stanford Center for Research on Foundation Models report that coined the term, define a foundation model as "any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks." The word "foundation" is deliberate: it is not the finished building, it is what everything downstream gets built on. This chapter works through why that shift happened, what makes it mathematically possible, where it breaks down, and how you actually adapt one base model into three different hospital products.
From narrow supervised learning to a shared base
Pre-2018 deep learning was overwhelmingly narrow and supervised: pick a task, collect a labeled dataset sized for that task, train a model whose architecture and weights exist only for that task, deploy it, and repeat from scratch for the next task. The bottleneck was never model expressiveness — deep networks had been expressive enough for years — it was labeled data. Every new capability cost a new annotation project.
Foundation models break the tie between "new capability" and "new labeled dataset" by inserting a stage that needs no task-specific labels at all: self-supervised pretraining on raw data, followed by lightweight adaptation to many tasks. Bommasani et al. point to two properties this produces. Emergence: capabilities that were never explicitly targeted by the training objective show up anyway once the model and its training data are large enough — a model trained only to predict the next word acquires something usable for translation, arithmetic, and code completion as side effects. Homogenization: because building a foundation model from scratch is so expensive, a small number of them end up underneath a very large number of downstream applications. The hospital's transcription, summarization, and chat products might all sit on the same base model, adapted three different ways. Emergence is what makes the paradigm useful; homogenization is what makes it a single point of systemic risk, a point this chapter returns to at the end.
Self-supervised pretraining: the trick that removes the labeling bottleneck
The Transformer architecture (Vaswani et al., 2017, "Attention Is All You Need") made it computationally practical to train on far more data than recurrent networks could absorb, because self-attention over a sequence parallelizes across positions instead of processing token-by-token. But the architecture alone doesn't remove the labeling bottleneck — the training objective does. Two objectives dominate:
Masked language modeling (BERT — Devlin et al., 2018): randomly hide some tokens in a sentence and train the model to predict them from the surrounding context in both directions. Autoregressive (causal) language modeling (the GPT family — Radford et al., 2018; Radford et al., 2019; Brown et al., 2020): train the model to predict the next token given only the tokens that came before it. Both are "self-supervised" in the precise sense that the label was never written by a human — it is already sitting inside the raw text. Take the sentence "The Reserve Bank of India raised the repo rate to 6.5 percent." An autoregressive model sees "The Reserve Bank of India raised the repo rate to" and its training label is simply the next token, "6.5" — extracted automatically from any RBI news article on the internet, at zero annotation cost. Do this across a trillion tokens of web text, books, and code, and you get a supervised-learning-shaped training signal at a scale that human annotation could never fund. This is the actual mechanism behind "broad data" in the definition — it is not that labels stopped mattering, it is that the label source moved from human annotators to the text itself.
Scale as a designed variable: scaling laws
Once labels are free, the constraint shifts from "how much labeled data can we afford" to "how much compute can we afford," which raises a genuinely new engineering question: given a fixed compute budget, how should it be split between a bigger model (more parameters, N) and more training data (more tokens, D)? Kaplan et al. (2020) first showed that pretraining loss falls as a smooth power law in N, D, and total compute C, which made scale a plannable engineering variable rather than a matter of trial and error. Hoffmann et al. (2022) — the Chinchilla paper — refined this with a specific, checkable answer: for a transformer, training compute is well approximated as C ≈ 6ND FLOPs (forward pass ≈ 2ND, backward pass roughly double that, ≈ 4ND, total ≈ 6ND), and the compute-optimal split trains on roughly D ≈ 20N tokens. Their headline finding was that most large models built before Chinchilla — including the original 175-billion-parameter GPT-3 (Brown et al., 2020), trained on about 300 billion tokens — were undertrained relative to this ratio: 300 billion tokens over 175 billion parameters is a ratio of only about 1.7 tokens per parameter, far short of 20. Chinchilla itself, at 70 billion parameters trained on 1.4 trillion tokens, sits almost exactly at the 20:1 ratio (1.4×10¹² / 7×10¹⁰ = 20) and outperformed the much larger GPT-3 on downstream benchmarks using less total compute — direct evidence that pouring a fixed budget into parameters alone, past a point, is a worse trade than splitting it with data.
Work the compute-optimal allocation yourself. Suppose a lab has a training budget of C = 1.2 × 10²¹ FLOPs and wants to hit the Chinchilla-optimal ratio D = 20N. Substituting into C = 6ND gives C = 6N(20N) = 120N², so N = √(C / 120):
import math
C = 1.2e21 # training compute budget, in FLOPs
ratio = 20 # Chinchilla's compute-optimal tokens-per-parameter ratio (D = ratio * N)
# C = 6*N*D = 6*N*(ratio*N) = 6*ratio*N**2
N = math.sqrt(C / (6 * ratio))
D = ratio * N
print(f"N = {N:.3e} parameters")
print(f"D = {D:.3e} tokens")
print(f"check C = {6 * N * D:.3e} FLOPs")Trace it by hand before trusting the output: 6 × 20 = 120; C / 120 = 1.2×10²¹ / 120 = 1×10¹⁹; N = √(1×10¹⁹) ≈ 3.162×10⁹. Then D = 20 × 3.162×10⁹ ≈ 6.325×10¹⁰. The check recomputes C = 6 × 3.162×10⁹ × 6.325×10¹⁰ ≈ 1.200×10²¹, matching the budget. So the code prints N = 3.162e+09 parameters, D = 6.325e+10 tokens, check C = 1.200e+21 FLOPs — a roughly 3.2-billion-parameter model trained on about 63 billion tokens is the compute-optimal use of that budget, not the biggest model the budget could technically fit.
Emergent capabilities — and the misconception around them
Wei et al. (2022), surveying benchmark performance across model scales, documented capabilities — multi-step arithmetic, certain chain-of-thought reasoning tasks — that stay near chance accuracy across a wide range of smaller models and then rise sharply once parameter count crosses some threshold, which they termed emergent abilities. The common misconception a student forms here is that this sharp rise reflects a qualitative, almost mechanical event inside the network — as if a "reasoning module" switches on once the model gets big enough, the way a phase transition turns water to steam at exactly 100°C.
Schaeffer, Miranda, and Koyejo (2023), in "Are Emergent Abilities of Large Language Models a Mirage?", identify the actual source of many of these apparent jumps: the choice of evaluation metric, not a hidden architectural switch. Many emergent-ability benchmarks score with exact match — for multi-digit arithmetic, the model must get every single digit right or the whole answer counts as wrong. If a model's per-digit accuracy is improving smoothly and continuously with scale (say from 50% to 90% correct per digit), the probability of getting all digits right in, say, a 5-digit answer is that per-digit accuracy raised to the 5th power — a nonlinear, sharply accelerating function of a smoothly improving quantity. Under exact-match scoring this produces exactly the sudden-cliff shape Wei et al. observed; when Schaeffer et al. re-score the same model checkpoints with a smoother, continuous metric — such as per-token log-likelihood or edit distance — the improvement with scale is gradual and predictable, with no cliff. The correction is not that scale never buys new behavior — it clearly does — it is that a sharp jump on a discontinuous, all-or-nothing metric is not by itself evidence of a discontinuous change inside the model. Before crediting an "emergent" capability to some qualitative shift, check whether a continuous version of the same metric shows the same shape.
One pretraining run, many adaptations
The second half of the paradigm shift is adaptation: turning one pretrained base into the hospital's three products without three from-scratch training runs. Four methods matter, in increasing order of how much they touch the base model's weights.
Few-shot / zero-shot in-context learning (Brown et al., 2020) needs no weight update at all — you place a handful of example question-answer pairs, or none, directly in the prompt, and the model's next-token predictions condition on them. This is the cheapest adaptation and is often enough for the hospital's FAQ chatbot if the questions are reasonably standard. Full fine-tuning updates every one of the model's weights on a smaller, task-specific labeled set — still far smaller than a from-scratch dataset, since the model already knows language, but it is the most compute- and storage-expensive adaptation: every fine-tuned copy is a full duplicate of the base model's weights. Instruction tuning (Wei et al., 2021, FLAN) and RLHF (Ouyang et al., 2022, InstructGPT — a reward model trained on human preference comparisons between candidate outputs, then used to optimize the base model's policy) are both variants of full fine-tuning aimed specifically at making the base model follow instructions and match human judgment of "good" responses, rather than at a narrow downstream task.
Low-Rank Adaptation, LoRA (Hu et al., 2021), is the method that makes maintaining three separate hospital products on one base model actually affordable. Instead of updating a full weight matrix W (shape d_out × d_in) during fine-tuning, LoRA freezes W entirely and learns a much smaller update ΔW = BA, where B has shape d_out × r and A has shape r × d_in, for a rank r that is tiny compared to d_out or d_in (r = 8 is a common practical choice; Hu et al.'s ablations find ranks as low as 1–2 retain most of full fine-tuning's quality on several tasks). Only B and A are trained and stored per task; W stays shared. Trace the savings on one GPT-3-scale attention projection matrix, where GPT-3's hidden size is d_model = 12,288 (Brown et al., 2020):
d_in = 12288 # GPT-3-175B hidden size (Brown et al., 2020)
d_out = 12288
r = 8 # LoRA rank
full_finetune_params = d_in * d_out
lora_params = r * (d_in + d_out)
print(f"Full fine-tune: {full_finetune_params:,} parameters")
print(f"LoRA (r={r}): {lora_params:,} parameters")
print(f"Reduction: {full_finetune_params / lora_params:.1f}x fewer trainable parameters")By hand: full_finetune_params = 12,288 × 12,288 = 150,994,944. lora_params = 8 × (12,288 + 12,288) = 8 × 24,576 = 196,608. The ratio 150,994,944 / 196,608 = 768.0 exactly. So the code prints Full fine-tune: 150,994,944 parameters, LoRA (r=8): 196,608 parameters, Reduction: 768.0x fewer trainable parameters. The hospital can keep one 175-billion-parameter base loaded in memory and swap in a ~200,000-parameter adapter file per product — transcription-adapter, summary-adapter, chat-adapter — instead of hosting three full duplicate copies of the base model.
This is also where homogenization becomes concrete risk, not just abstraction. If the shared base model has a systematic bias — say it underperforms on Marathi-accented English, or has a factual blind spot about Indian drug names — that flaw is inherited by every adapter built on top of it: transcription, summary, and chat all fail the same way, simultaneously, because they share one foundation. Under the old narrow-model paradigm, a flaw in the transcription model had no reason to also appear in an unrelated chatbot trained from a completely different dataset and architecture. Consolidating capability onto one base also consolidates its failure modes onto every product downstream of it — the reason foundation-model deployment increasingly comes with its own governance and evaluation obligations, distinct from the modeling work itself.
How pretraining and adaptation fit together
Active recall
Attempt each question before reading its answer.
Q1. In your own words, distinguish "emergence" from "homogenization" as Bommasani et al. (2021) use the terms — why is one the benefit of the foundation-model paradigm and the other its systemic risk?
Q2. The hospital's compute budget grows fourfold, from C = 1.2 × 10²¹ FLOPs to C = 4.8 × 10²¹ FLOPs, and they still want the Chinchilla-optimal ratio D = 20N. Find the new N and D, and state precisely how they scaled relative to the original 3.162 × 10⁹ parameters and 6.325 × 10¹⁰ tokens.
Q3. Using the same GPT-3 attention matrix (d_in = d_out = 12,288) as the LoRA worked example, recompute the trainable-parameter count and the reduction factor versus full fine-tuning if the rank is raised from r = 8 to r = 64.
Q4. A classmate says: "GPT-4 can suddenly do 3-digit multiplication that smaller models can't — that proves a dedicated reasoning circuit switches on above some parameter threshold." Using Schaeffer, Miranda, and Koyejo (2023), identify the flawed premise.
Q5. Why is self-supervised pretraining described as removing the labeling bottleneck of the pre-2018 paradigm? Name the concrete source of the training label in autoregressive pretraining.
Q6. Name one systemic risk that homogenization introduces which would not exist under the old narrow-model paradigm, using the hospital's three products as the example.
A1. Emergence is a property of the model in isolation: capabilities not explicitly targeted by the pretraining objective (translation, arithmetic, code) appear anyway once scale crosses some point, and this is what makes one base model useful for many downstream tasks. Homogenization is a property of the ecosystem built on top of the model: because so few foundation models exist relative to the number of applications built on them, any flaw, bias, or vulnerability in one base model is inherited by every downstream system built on it. Emergence is why the paradigm works; homogenization is why a single point of failure now sits underneath products that used to be architecturally independent.
A2. C/120 = 4.8×10²¹/120 = 4×10¹⁹, so N = √(4×10¹⁹) = 2 × √(1×10¹⁹) = 2 × 3.1623×10⁹ ≈ 6.325×10⁹ parameters, and D = 20N ≈ 1.265×10¹¹ tokens. Check: 6 × 6.325×10⁹ × 1.265×10¹¹ ≈ 4.80×10²¹, matching the budget. Both N and D exactly doubled — not quadrupled — because C = 120N² makes N proportional to √C: a 4× compute increase yields a √4 = 2× increase in model size (and, since D = 20N, a 2× increase in data too), not a proportional 4× increase in either. This is the ripple a naive answer misses: more compute buys a smaller model boost than intuition suggests, because compute is spent on both N and D simultaneously, and N only grows as the square root of the budget.
A3. Full fine-tune parameter count is unchanged, since the matrix shape didn't change: 12,288 × 12,288 = 150,994,944. LoRA parameter count becomes r(d_in + d_out) = 64 × 24,576 = 1,572,864. The reduction factor is 150,994,944 / 1,572,864 = 96.0 exactly — down from 768.0x at r = 8. Because full-fine-tune parameters are fixed and LoRA parameters scale linearly in r, the reduction factor scales as 1/r: multiplying r by 8 (from 8 to 64) divides the reduction factor by exactly 8 (768/8 = 96), which the direct recomputation confirms.
A4. The flawed premise is treating a sharp jump on an exact-match benchmark as direct evidence of a discontinuous internal change. Multi-digit multiplication scored by exact match requires every digit correct; if per-digit accuracy is improving smoothly with scale, the probability of a fully correct multi-digit answer (per-digit accuracy raised to the power of the digit count) accelerates sharply near high accuracy purely as a mathematical consequence of the scoring rule — no new circuit is required to produce that shape. Schaeffer, Miranda, and Koyejo (2023) show that re-scoring the same checkpoints with a continuous metric (e.g., token-level log-likelihood) typically turns the cliff into a smooth curve, meaning the sharpness was substantially a property of the metric, not demonstrated evidence of a qualitative architectural switch.
A5. Under narrow supervised learning, every new capability required a human to write labels, capping training data at whatever annotation budgets could afford. Self-supervised pretraining eliminates that cap because the label is already embedded in unlabeled raw text: for autoregressive pretraining, the label for any given position is simply the next token that already occurs in the naturally occurring text, extracted automatically rather than annotated by a person — which is why pretraining corpora can reach trillions of tokens instead of the thousands-to-millions typical of hand-labeled datasets.
A6. If the shared base model underneath transcription, summarization, and chat has a systematic weakness — for instance, degraded accuracy on Marathi-accented medical speech, or an outdated understanding of a drug interaction — that single weakness surfaces simultaneously across all three products, because all three inherit it from the same foundation. Under the old narrow-model paradigm, the transcription model's speech-recognition dataset and the chatbot's text-classification dataset shared no common component, so a flaw in one had no mechanism to also appear in the other; homogenization removes that isolation.
Think About It
Think about this: How would you explain foundation models: the paradigm shift in ai to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where foundation models: the paradigm shift in ai is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting foundation models: the paradigm shift in ai to at least 3 other topics you have studied.