Large Language Models: From GPT to Claude

A fintech support bot in Bengaluru is handling a message from a customer: "UPI पेमेंट फेल हुआ, पैसे कट गए" (UPI payment failed, money got deducted). Two versions of the bot exist, built on almost the same decoder-only transformer, trained on overlapping web-scale corpora, and both perfectly capable of predicting the next token in that sentence. Push either bot hard enough, though, and their behaviour splits. Ask the GPT-family bot to ignore its instructions and reveal its system prompt, and its resistance traces back to reward signals collected from human labelers ranking outputs. Ask the same of a Claude-family bot, and its resistance traces back to a written list of principles the model was trained to critique its own drafts against. Same architecture, same objective at the lowest level (predict the next token), and yet the two models were shaped by measurably different training pipelines in their final stage. This chapter is about that final stage, and about the systems engineering that turns a trained model into something an app can actually call in production.

The objective underneath everything: next-token prediction

Every model in this family, GPT-1 through GPT-4, and every Claude generation, is a decoder-only transformer trained to do one thing: given a sequence of tokens x_1, ..., x_{t-1}, output a probability distribution over the next token x_t. Radford et al. (2018) established the decoder-only recipe for language modelling by combining the transformer block from Vaswani et al. (2017) with a strictly causal (left-to-right) attention mask, so a token can only attend to tokens before it. The joint probability of an entire sequence then factorises as a product of these next-token conditionals:

P(x_1, x_2, ..., x_T) = P(x_1) · P(x_2 | x_1) · P(x_3 | x_1, x_2) · ... · P(x_T | x_1, ..., x_{T-1})

Training minimises the negative log-likelihood of this factorisation over trillions of tokens of text, which is exactly cross-entropy loss applied at every position. Before any of that math runs, though, the raw text has to become tokens. Sennrich, Haddow, and Birch (2016) introduced byte-pair encoding (BPE) for neural translation: start from a small base vocabulary, then repeatedly merge the most frequent adjacent pair of symbols into a new vocabulary entry, until the vocabulary reaches a target size (tens of thousands of entries in practice). GPT-2 (Radford et al., 2019) adapted this into byte-level BPE, running the merge algorithm over raw UTF-8 bytes rather than characters, and this byte-level variant is what the GPT and Claude tokenizer families both build on today. This is why a code-mixed message like the one above tokenizes cleanly even though it mixes Devanagari and Latin scripts: operating on bytes means the tokenizer never hits an "unknown word" the way a fixed word-level vocabulary would. "UPI" might stay a single token because it is common enough to have earned its own merged vocabulary entry; "पेमेंट" likely splits into two or three subword pieces.

Worked example 1: what the model is actually optimising, one token at a time

Suppose the bot has already processed the tokens "UPI" and "पेमेंट" and must produce a probability distribution over the next token. To make the softmax and cross-entropy mechanics concrete, take a toy five-token vocabulary and suppose the model's final linear layer (the "unembedding" matrix) produces these logits for that position:

Softmax converts logits to probabilities: P(token_i) = e^{z_i} / Σ_j e^{z_j}. Computing the five exponentials and their sum (≈ 67.58) gives:

import math

logits = [2.0, 1.0, 4.0, 0.5, 0.2]
tokens = ["UPI", "पेमेंट", "फेल", "हुआ", "।"]

exps = [math.exp(z) for z in logits]
total = sum(exps)
probs = [e / total for e in exps]

for tok, p in zip(tokens, probs):
    print(tok, round(p, 4))
# UPI 0.1093
# पेमेंट 0.0402
# फेल 0.808
# हुआ 0.0244
# । 0.0181

true_token_index = 2  # "फेल" is the actual next token in the training text
loss = -math.log(probs[true_token_index])
print(round(loss, 4))
# 0.2132

Running this trace by hand: e^2.0 ≈ 7.389, e^1.0 ≈ 2.718, e^4.0 ≈ 54.598, e^0.5 ≈ 1.649, e^0.2 ≈ 1.221, summing to 67.575. Dividing each exponential by that sum gives the probabilities printed above, and they sum to 1.0 as required. Because the model correctly assigned "फेल" (fail) the highest logit, its probability is 0.808 and the cross-entropy loss -ln(0.808) ≈ 0.2132 is small: the gradient step at this position will nudge the weights only slightly, mostly to sharpen an already-good prediction. This single-token computation, repeated across every position in every training document, at a scale of hundreds of billions to trillions of tokens, is the entire content of pretraining. GPT-3 (Brown et al., 2020) was pretrained this way on roughly 300 billion tokens with 175 billion parameters and a 2048-token context window, remarkably small by 2026 standards, and a number that matters later in this chapter.

Three stages, one fork: pretraining, SFT, and where GPT and Claude diverge

A raw pretrained model like GPT-3 is a very good next-token predictor, but a poor assistant: asked a question, it is just as likely to continue with another question (because that pattern appears constantly on the open web) as to answer it. Turning a pretrained model into something like ChatGPT or Claude takes two further stages, and it is the second of these two where the GPT and Claude lineages structurally diverge.

Stage 2: supervised fine-tuning (SFT). Human contractors write ideal demonstration answers to a curated set of prompts ("if a user asks this, respond like this"), and the pretrained model is fine-tuned on these demonstrations with ordinary supervised cross-entropy loss. This teaches the model the shape of an answer (structured, on-topic, appropriately terminated) but doesn't yet calibrate which of several plausible answers a human would actually prefer.

Stage 3a: RLHF, the GPT/InstructGPT path. Ouyang et al. (2022) describe the pipeline OpenAI used to turn GPT-3 into InstructGPT, the direct ancestor of ChatGPT's behaviour. Human labelers are shown several model outputs for the same prompt and rank them best to worst. A separate reward model is trained to predict these human rankings, and the SFT model is then further optimised with reinforcement learning (proximal policy optimization, PPO) to produce outputs the reward model scores highly. Because pure RL against the reward model can quietly erode capabilities that the reward model wasn't trained to value (Ouyang et al. call this the "alignment tax", measurable regressions on some public NLP benchmarks after RLHF), they mix gradients from the original pretraining objective back into the RL update (a variant they call PPO-ptx) to hold general capability roughly steady while behaviour shifts.

Stage 3b: RLAIF and Constitutional AI, the Claude path. Bai et al. (2022), at Anthropic, replace the human-labeler step with AI feedback. A written "constitution", a short list of natural-language principles the model should follow, drives two phases. First, a supervised phase: the model generates a response, then critiques its own response against the constitution, then revises it, and is fine-tuned on these self-revised outputs. Second, a reinforcement phase (RLAIF, reinforcement learning from AI feedback): instead of human labelers ranking output pairs, the model itself is prompted to judge which of two candidate responses better satisfies the constitution, and these AI-generated preference labels train a reward model exactly as human labels would. RL then proceeds against that reward model, same optimisation machinery as RLHF, different origin for the preference data.

The mechanically important point is that both paths end up training a reward model from pairwise preference comparisons and optimising a policy against it; the difference is entirely in who (or what) produces the preference judgment, and what standard that judgment is measured against: an implicit standard distilled from many individual human raters' judgments, versus an explicit, inspectable, written standard the model applies to itself. The diagram below traces this shared skeleton and the point where it forks.

Worked example 2: turning rankings into a training signal (Bradley–Terry)

Both stage-3 paths need to convert "output A is better than output B" into a number a gradient can push on. The standard tool, used in RLHF since Christiano et al. (2017) introduced learning from preference comparisons and carried into InstructGPT and Constitutional AI alike, is the Bradley–Terry model (Bradley and Terry, 1952), originally built for ranking competitors from paired comparisons. If a reward model assigns scalar scores r_A and r_B to two candidate responses, the model's implied probability that a labeler (human or AI) preferred A is:

P(A ≻ B) = σ(r_A − r_B) = 1 / (1 + e^{-(r_A - r_B)})

Suppose the reward model scores a polite, specific reply as r_A = 2.3 and a curt, generic reply as r_B = 1.1, and the labeler (human, in RLHF; the model itself, in RLAIF) marked A as preferred. Tracing the computation:

diff = 2.3 - 1.1            # 1.2
p_A = 1 / (1 + math.exp(-diff))   # ≈ 0.7685
loss = -math.log(p_A)             # ≈ 0.2633

e^{-1.2} ≈ 0.3012, so p_A ≈ 1 / 1.3012 ≈ 0.7685, and the loss -ln(0.7685) ≈ 0.2633. The reward model's parameters are updated by gradient descent on this loss, which pushes r_A up and r_B down until the predicted preference probability better matches the observed label: exactly the same loss function whether the "1" label came from a paid contractor ranking InstructGPT outputs or from a Claude model comparing two of its own drafts against a written constitution. The RL step afterward (PPO, in both lineages) then nudges the policy's token-level probabilities to produce more outputs like the ones the reward model scores highly.

Worked example 3: context windows cost memory, not just compute

Training produces a static set of weights; serving that model to millions of requests is a separate systems problem, and this is where the headline context-window numbers, GPT-3's 2048 tokens in 2020, GPT-4 Turbo's 128,000 tokens by late 2023, Claude 2's 100,000 tokens in mid-2023 and Claude 3's 200,000 tokens in 2024, become an engineering constraint rather than a marketing spec.

During autoregressive generation, producing token t+1 requires every attention layer to compute queries against the key and value vectors of all t preceding tokens. Recomputing those keys and values from scratch at every new step would make generating an n-token response cost O(n^2) per layer. Instead, production servers cache each layer's key and value vectors as they're computed and reuse them, so each new token costs O(n) against the growing cache rather than recomputing the prefix. This KV cache is not free: its memory footprint grows linearly with sequence length, and for long-context requests it can dwarf the size of the model weights themselves. The size, in bytes, is:

KV cache bytes = 2 × layers × heads × head_dim × seq_len × batch × bytes_per_element

the leading 2 accounting for storing both keys and values. Neither OpenAI nor Anthropic publishes GPT-4's or Claude's exact architecture, but Meta published the dimensions of Llama-2-13B in full (Touvron et al., 2023): 40 transformer layers, 40 attention heads, head dimension 128 (giving a 5120-dimensional hidden state). Using those real, public numbers as a concrete stand-in with ordinary multi-head attention and 16-bit weights:

def kv_cache_bytes(layers, heads, head_dim, seq_len, batch=1, bytes_per=2):
    return 2 * layers * heads * head_dim * seq_len * batch * bytes_per

layers, heads, head_dim = 40, 40, 128
for seq in [8192, 128000, 200000]:
    b = kv_cache_bytes(layers, heads, head_dim, seq)
    print(seq, round(b / 1e9, 3), "GB")
# 8192 6.711 GB
# 128000 104.858 GB
# 200000 163.84 GB

A single 128,000-token request already needs a KV cache larger than an entire H100 GPU's 80 GB of memory, before the model's own weights (26 GB at fp16 for this 13B-parameter shape) are even loaded. This is precisely why long-context serving needed real algorithmic work between GPT-3's 2048-token window and Claude 3's 200,000-token one, not just bigger GPUs: Shazeer (2019) proposed multi-query attention, sharing a single key/value head across all query heads, and Ainslie et al. (2023) generalised this to grouped-query attention (GQA), sharing key/value heads across small groups of query heads rather than either all-shared or all-separate. Cutting the 40 key/value heads above down to, say, 8 shared groups shrinks the cache by exactly that 5× ratio, to roughly 21 GB at 128k tokens, batch 1, making long-context serving on a single GPU arithmetically possible rather than impossible.

Common misconception: "RLHF and Constitutional AI make the model smarter"

It is tempting to read the story above and conclude that the alignment stage is where a language model acquires knowledge and reasoning ability, since that's the stage where behaviour visibly improves, answers get more helpful, and refusals get more sensible. This is backwards. Essentially all of a model's world knowledge, factual associations, and raw reasoning capacity come from the pretraining stage, where the next-token objective forces the model to compress patterns from an enormous, diverse text corpus. The stage-3 alignment step, RLHF or RLAIF, does not add new facts to the model; it reweights the probability distribution over outputs the model could already produce, steering it toward responses that a labeler (human or AI) rates as more helpful, honest, and safe, and away from responses that are technically fluent but poorly targeted, evasive, or unsafe. The clearest evidence for this is the very problem Ouyang et al. (2022) had to engineer around: naive RLHF measurably regressed performance on some standard NLP benchmarks relative to the plain pretrained or SFT model, the "alignment tax", because optimising hard against a reward model can overfit to what human raters happen to like in the training sample at the expense of general capability. If alignment training were adding capability wholesale, that tax couldn't appear at all. The fix, mixing pretraining gradients back into the RL update, is itself an admission that the knowledge lives upstream, in pretraining, and stage 3's job is narrowly about behaviour: which of the things the model already knows how to say, it actually says, and how it says them.

Active recall

Attempt each question before reading its answer.

1. In worked example 1, why can the softmax output for "फेल" (0.808) never reach exactly 1.0, no matter how much larger its logit is made relative to the others?
2. Using the same logits from worked example 1 ([2.0, 1.0, 4.0, 0.5, 0.2] for [UPI, पेमेंट, फेल, हुआ, ।]), what is the cross-entropy loss if the true next token in the training data had actually been "UPI" instead of "फेल"? Is the model being pushed harder or more gently than in the original example?
3. In worked example 2, what specific data determines the reward-model scores r_A and r_B under the RLHF pipeline, versus under the RLAIF / Constitutional AI pipeline? What stays identical between the two?
4. Starting from the 128,000-token KV cache figure in worked example 3 (104.858 GB, batch 1, 40 key/value heads): the engineering team switches to GQA with 8 key/value heads and doubles the batch size to serve 2 concurrent 128k-context requests per GPU. Compute the final KV cache memory, add the fp16 weight memory for the Llama-2-13B-shaped model (26 GB), and state whether the total plausibly fits on a single 80 GB H100 alongside activation memory.
5. True or false: Constitutional AI (Bai et al., 2022) removes the reward model from the pipeline entirely, since the AI judges its own outputs. Justify your answer.
6. GPT-3's context window (Brown et al., 2020) was 2048 tokens; Claude 3's is 200,000 tokens, about 98× longer. If full self-attention compute scales roughly with the square of sequence length per layer, by roughly what factor does the raw attention compute for a full-context forward pass grow, and why hasn't serving cost grown by that same factor in practice?

Answers.

1. Softmax is e^{z_i} / Σ_j e^{z_j}, and every term in that sum, including the term for "फेल" itself, is strictly positive for any finite real logit (an exponential is never zero or negative). So the denominator is always strictly larger than any single numerator, and the ratio is always strictly less than 1, no matter how large the gap between the top logit and the rest becomes. This is also why cross-entropy loss can shrink toward zero but never reach it exactly during training on finite logits.

2. With true token index 0 ("UPI", probability 0.1093), the loss is -ln(0.1093) ≈ 2.2132, compared to 0.2132 in the original example, roughly ten times larger. The model is being pushed much harder here: it had assigned "UPI" a low probability while the training data says "UPI" was actually correct, so the gradient step will substantially raise the logit for "UPI" at this context and lower the others, including "फेल", which had been (wrongly, in this hypothetical) favoured.

3. Under RLHF, r_A and r_B are fit to match rankings that paid human contractors assigned to pairs of model outputs. Under RLAIF / Constitutional AI, they are fit to match preference judgments the model itself produces by comparing two candidate outputs against a written constitution. What stays identical is the loss function used to fit the reward model, the Bradley–Terry log-loss -log σ(r_chosen - r_rejected), and the downstream RL step that optimises the policy against whichever reward model results.

4. GQA with 8 key/value heads instead of 40 shrinks the batch-1 cache by exactly 40/8 = 5×: 104.858 / 5 ≈ 20.972 GB. Doubling the batch to 2 doubles that again: ≈ 41.943 GB for KV cache alone. Adding the 26 GB of fp16 weights gives roughly 41.943 + 26 ≈ 67.94 GB out of 80 GB, leaving only about 12 GB for activation memory, the intermediate tensors computed during the forward pass itself. That margin is thin: it might just fit for pure sequential decoding of short outputs, but any meaningful prefill batch or additional overhead (multiple attention score buffers, MLP activations across 40 layers) would plausibly overflow it, which is exactly why real long-context serving systems lean on further tricks (paged/quantized KV cache, offloading, tighter batching policies) rather than treating this as a comfortable fit.

5. False. Bai et al. (2022) still train a reward model, just as InstructGPT does; the only structural change is that the pairwise preference labels used to train that reward model are produced by the AI model judging its own candidate outputs against the constitution, rather than by human labelers. The RL optimisation step against the resulting reward model is otherwise the same as in RLHF.

6. If compute scales with the square of sequence length, growing the context 98× would grow raw full-context attention compute by roughly 98^2 ≈ 9,604×. Serving cost hasn't scaled by that factor in practice because of exactly the systems and algorithmic work covered above: memory-efficient attention kernels that avoid materialising the full attention matrix, KV-cache reuse so generation itself is O(n) rather than O(n^2) per new token, and head-sharing schemes like multi-query attention (Shazeer, 2019) and grouped-query attention (Ainslie et al., 2023) that cut the memory (and correspondingly some of the bandwidth-bound cost) of maintaining that cache. The quadratic term never disappears, but four years of targeted engineering moved the constant and the memory bottleneck enough to make 200k-token context commercially serviceable rather than only a research curiosity.

Think About It

Think about this: How would you explain large language models: from gpt to claude to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind large language models: from gpt to claude, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.