RLHF: How ChatGPT Was Trained

In November 2022, OpenAI's engineering blog described ChatGPT as a "sibling model" to InstructGPT, trained with the same method described in Ouyang et al., Training Language Models to Follow Instructions with Human Feedback (NeurIPS 2022). That paper is worth reading as an engineering document, not just a research summary, because it reveals a detail most explanations of RLHF skip: a single training step does not involve one neural network being nudged toward better answers. It involves four separate networks, each built on the same GPT-3 architecture (the paper trains 1.3B, 6B, and 175B-parameter variants), running forward passes in lockstep before a single weight is updated. A companion chapter in this course already walked through how the reward model is trained from human comparisons and how the SFT-to-RM-to-RL pipeline fits together. This chapter assumes that pipeline and goes one level deeper, into the two things that actually make RLHF hard to build: the optimization algorithm that turns a scalar reward into a stable weight update — Proximal Policy Optimization (PPO), from Schulman et al., Proximal Policy Optimization Algorithms (arXiv, 2017) — and the systems-engineering cost of running four coupled models at once. Neither of these gets much space in a first pass through RLHF, and both are exactly where a from-scratch reimplementation breaks.

Why a reward score cannot just be backpropagated

Start with the question a student who has just finished supervised fine-tuning (SFT) will naturally ask: the reward model already outputs a number that says how good a response is — why not treat that number as a loss and backpropagate it through the policy network the way cross-entropy is backpropagated during SFT?

The obstacle is generation itself. During SFT, every token in the loss is a token that already exists in the training example; the loss is a differentiable function of the model's output logits at each position, and gradients flow cleanly back through the network. During RLHF, the policy has to actually produce the response by sampling a token at every position from a probability distribution, then feeding that sampled token back in to produce the next one. Sampling — drawing a discrete token id from a categorical distribution — is not a differentiable operation. There is no gradient of "which token got sampled" with respect to the network's parameters, because the sampling step is a discrete, stochastic choice sitting between the parameters and the final reward. You cannot chain the derivative through a dice roll.

This is precisely the situation policy-gradient reinforcement learning was built for. Instead of differentiating through the reward, PPO differentiates through the probability the policy assigned to the action it happened to take, and scales that gradient by how good the action turned out to be (the advantage). That reframing — from "backprop the reward" to "backprop the log-probability of the sampled action, weighted by an advantage" — is the single idea that makes the rest of PPO make sense, and it is the piece students most often gloss over on a first pass.

The PPO clipped surrogate objective

The raw policy-gradient objective for a single token is r_t(θ) · A_t, where r_t(θ) = π_θ(a_t|s_t) / π_θ_old(a_t|s_t) is the probability ratio between the current policy and the policy that generated the training batch, and A_t is the advantage — an estimate, produced by the value model, of how much better this token was than the average token the policy would have produced in that state. Maximizing r_t(θ) · A_t directly is unstable: a single large gradient step can push r_t(θ) far from 1, meaning the policy's behavior changes drastically between one PPO update and the next, based on a noisy reward-model estimate for one batch of generations. Schulman et al.'s fix is to clip the ratio into a trust region [1-ε, 1+ε] (InstructGPT uses ε = 0.2) and take the more pessimistic of the clipped and unclipped objective:

L_CLIP(θ) = min( r_t(θ)·A_t , clip(r_t(θ), 1-ε, 1+ε)·A_t )

The reasoning behind taking a minimum (not just clipping the ratio and moving on) is easy to misjudge, so trace it numerically rather than trusting intuition. The function below implements exactly the formula above with nothing hidden:

def clip(x, lo, hi):
    return max(lo, min(x, hi))

epsilon = 0.2

scenarios = [
    {"name": "good token, probability rose past the +20% trust region",
     "old_prob": 0.40, "new_prob": 0.52, "advantage": 2.0},
    {"name": "bad token, probability fell past the -20% trust region",
     "old_prob": 0.50, "new_prob": 0.35, "advantage": -2.0},
]

for s in scenarios:
    ratio = s["new_prob"] / s["old_prob"]
    unclipped = ratio * s["advantage"]
    clipped_ratio = clip(ratio, 1 - epsilon, 1 + epsilon)
    clipped = clipped_ratio * s["advantage"]
    l_clip = min(unclipped, clipped)
    print(s["name"])
    print(f"  ratio={ratio:.3f}  unclipped={unclipped:.3f}  "
          f"clipped={clipped:.3f}  L_CLIP={l_clip:.3f}")

Trace the first scenario by hand: the policy raised the probability of a genuinely good token from 0.40 to 0.52, a ratio of 1.3, which sits above the upper bound 1.2. Unclipped, the objective would be 1.3 × 2.0 = 2.6. Clipped, the ratio is capped at 1.2, giving 1.2 × 2.0 = 2.4. The minimum of the two is 2.4 — the smaller value. Because the objective is now pinned at the constant 1.2 rather than tracking the true ratio 1.3, its gradient with respect to further increases in probability is zero: PPO gives no further credit for pushing this token's probability up beyond the trust region, even though the advantage says it deserves it. That is the entire mechanism in one sentence: clipping removes the incentive to keep moving in a direction the policy has already moved in far enough, without ever penalizing it for having gotten there.

The second scenario is the mirror image and is where students most often get the sign wrong. The policy dropped the probability of a bad token from 0.50 to 0.35, a ratio of 0.7, below the lower bound 0.8. Unclipped: 0.7 × (-2.0) = -1.4. Clipped: the ratio is floored at 0.8, giving 0.8 × (-2.0) = -1.6. The minimum of −1.4 and −1.6 is −1.6, the more negative value — so here the clipped term is the one selected, and it is again pinned at a constant (0.8 × A, independent of the true ratio 0.7). PPO refuses to reward pushing this token's probability down any further in a single step, even though the advantage says it is a bad token. Both scenarios produce the identical qualitative behavior once you look past the sign: past the trust-region boundary, the gradient vanishes. The policy is allowed to correct a mistake, but only by a bounded amount per update, no matter how emphatically the advantage says it should move further. This is the same design principle behind the RBI adjusting the repo rate in 25 or 50 basis-point steps rather than jumping it by 400 basis points in one meeting even when inflation data would justify it mathematically — bounding the size of each policy correction prevents a single noisy signal from destabilizing the whole system.

The KL penalty, the reference model, and PPO-ptx

Clipping bounds how far the policy moves in a single PPO step, but says nothing about how far it can drift over hundreds of steps. Left unconstrained, a policy that is only rewarded for reward-model score will eventually find text that scores highly under the reward model's imperfect generalization but reads as garbled, repetitive, or overconfident to an actual human — because the reward model was trained on a comparison dataset (InstructGPT used roughly 33K prompt comparisons) that does not cover the full space of text the policy could learn to generate once it starts optimizing against it. This failure mode is called reward hacking, and it is the second reason RLHF training keeps a frozen copy of the SFT model around.

InstructGPT's full per-token objective, evaluated during the PPO rollout, is:

reward(x, y) = r_φ(x, y) − β · log( π_θ(y|x) / π_SFT(y|x) )

r_φ(x, y) is the trained reward model's score for the full response, and the second term is a per-token KL-divergence penalty between the policy being trained and the frozen SFT reference model, scaled by a coefficient β. Every token the policy generates that would have been unlikely under the original SFT model gets its effective reward reduced, in proportion to how unlikely it was. This directly caps reward hacking: to earn a high reward-model score, the policy has to stay in a region of text space where the reference model — trained purely to imitate high-quality human demonstrations, with no reward-optimization pressure at all — still assigns reasonable probability. Increasing β pulls the policy closer to the reference model's behavior at the cost of how aggressively it can chase reward; at β = 0, the KL term vanishes entirely and the policy is free to drift arbitrarily far from anything a human demonstrator would have written, exploiting whatever blind spots exist in the reward model.

InstructGPT adds one more term for exactly this reason. Pure RLHF fine-tuning, optimized only against the preference-based reward model, was found to cause measurable regressions on standard NLP benchmarks the paper's authors were not directly optimizing for — translation quality, reading comprehension, and similar tasks not represented in the labelers' comparison data. The paper calls this the "alignment tax." To offset it, the PPO objective is mixed with a second gradient term computed on the original GPT-3 pretraining data, weighted by a coefficient γ that the authors tuned empirically (this variant is called PPO-ptx in the paper):

objective(θ) = E[ r_φ(x,y) − β·log(π_θ(y|x)/π_SFT(y|x)) ] + γ·E[ log π_θ(x) ]  for x ~ pretraining data

The intuition is that continuing to train on plain next-token prediction over the original pretraining distribution, even while PPO is running, keeps pulling the policy back toward general language competence, partially counteracting the narrowing effect of optimizing hard against one reward model trained on one, comparatively narrow, style of preference judgment.

The four-model memory problem

Put the pieces together and a single PPO update step requires four distinct networks doing forward passes on the same batch of prompts: the policy π_θ, being trained, which generates the response; the reference model π_SFT, a frozen copy of the SFT checkpoint, used only to compute the KL term; the reward model r_φ, frozen, used to score the finished response; and the value model V_ψ, trained alongside the policy, used to estimate the baseline that turns raw rewards into the advantage A_t the clipped objective needs. Two of these networks are being trained (policy and value) and therefore need full optimizer state; two are frozen (reference and reward) and only need enough memory for inference. This is worth converting into an actual GPU count, because it is the reason RLHF is often the most memory-constrained stage of building a system like ChatGPT — more constrained, per training example, than pretraining itself. A standard estimate for full-parameter fine-tuning with the Adam optimizer in mixed precision is 16 bytes per parameter: 2 bytes for fp16 weights, 2 for fp16 gradients, and 4 each for the fp32 master weights and the two Adam moment buffers (2 + 2 + 4 + 4 + 4 = 16). A frozen, inference-only model needs only its fp16 weights, 2 bytes per parameter. Running the arithmetic for the 175B-parameter configuration InstructGPT used at its largest scale:

params = 175_000_000_000

policy_gb    = params * 16 / 1e9   # trainable, Adam, mixed precision
value_gb     = params * 16 / 1e9   # trainable, Adam, mixed precision
reference_gb = params * 2  / 1e9   # frozen, fp16 weights only
reward_gb    = params * 2  / 1e9   # frozen, fp16 weights only

total_gb = policy_gb + value_gb + reference_gb + reward_gb
gpu_mem_gb = 80                    # one A100 80GB
min_gpus = total_gb / gpu_mem_gb

print(policy_gb, value_gb, reference_gb, reward_gb)   # 2800.0 2800.0 350.0 350.0
print(total_gb)                                        # 6300.0
print(min_gpus)                                         # 78.75

2,800 GB for the policy, 2,800 GB for the value model, 350 GB each for the frozen reference and reward models — 6,300 GB, or roughly 6.3 TB, just to hold model weights and optimizer state before a single activation tensor or KV-cache entry is allocated. Divided across 80 GB A100 GPUs, that is a floor of 79 GPUs before any allowance for activation memory, generation-time KV caches, or the data parallelism needed for reasonable throughput — real production RLHF training clusters run far larger than this floor. (OpenAI has not published the exact cluster configuration or parameter count used for ChatGPT's own RLHF run — GPT-3.5's parameter count was never officially disclosed — so treat this as an order-of-magnitude estimate from InstructGPT's published 175B configuration, not a confirmed production number. The qualitative conclusion — that RLHF at this scale is a multi-model, GPU-hungry systems problem before it is a machine-learning problem — is the point, independent of the exact figure.) This is also why engineering teams building RLHF pipelines invest heavily in techniques the vanilla algorithm doesn't need at all: sharding the four models across devices with frameworks like DeepSpeed-ZeRO or FSDP, offloading the frozen reference and reward models to CPU memory between uses, or replacing the separate value model with a shared trunk to cut one of the two trainable copies.

A common misconception, corrected

The misconception worth naming explicitly: many students, having just learned that a reward model assigns a scalar score to text, assume RLHF trains the language model by treating that score as a differentiable loss and backpropagating it directly through the policy — essentially "supervised learning where the label is a number instead of a token." This is wrong for the reason worked through above: generation involves sampling a discrete token, and sampling has no gradient. There is no path to differentiate a reward computed on sampled text back through the parameters that produced the sampling distribution, the way there is a clean path to differentiate cross-entropy loss back through logits during ordinary SFT. RLHF is a policy-gradient method precisely because it has to route around this non-differentiability — it differentiates the log-probability the policy assigned to the tokens it actually sampled, not the reward itself, and uses the advantage (derived from the reward model and value model together) only as a scalar weight on that gradient. If a from-scratch implementation ever tries to call .backward() directly on a reward-model score with respect to policy parameters, something has gone wrong in the graph construction — that gradient path does not exist by design.

Beyond PPO: what Direct Preference Optimization removes

The four-model, clipped-ratio machinery above is exactly what a 2023 line of research set out to eliminate. Rafailov et al., Direct Preference Optimization: Your Language Model Is Secretly a Reward Model (NeurIPS 2023), show that under the same Bradley-Terry preference model InstructGPT's reward model is trained with, the optimal RLHF policy has a closed-form relationship to the reward function — which means the reward function can be algebraically eliminated from the training objective entirely, leaving a loss defined directly on pairs of preferred and dispreferred responses:

L_DPO(θ) = −E[(x,y_w,y_l)~D] log σ( β·log(π_θ(y_w|x)/π_SFT(y_w|x)) − β·log(π_θ(y_l|x)/π_SFT(y_l|x)) )

where y_w is the preferred response, y_l the dispreferred one, σ is the logistic sigmoid, and β plays a role directly analogous to the KL coefficient above. Notice what disappeared: no reward model, no value model, no PPO rollout or clipped ratio, no sampling from the policy during training at all — the loss is a closed-form function of log-probabilities under two models (policy and reference) evaluated on fixed preference data, trainable with ordinary supervised-style gradient descent. That collapses the four-model problem to two, and turns the unstable, GPU-hungry PPO loop into something that trains more like standard fine-tuning. It is not a strict improvement in every dimension — PPO's explicit reward model can be reused to score arbitrary new generations, which a DPO-trained policy cannot do directly — but it explains why, within a year of ChatGPT's release, DPO and its variants became the default RLHF-adjacent method in most open-weight model releases: the systems cost documented in the previous section was the main thing standing in the way.

Diagram: one PPO training step in InstructGPT-style RLHF

Active recall

Attempt all six before reading the answers below.

1. InstructGPT's PPO objective subtracts β·log(π_θ(y|x)/π_SFT(y|x)) from the reward-model score. What does increasing β do to the trained policy, and why can setting β = 0 cause training to collapse even though the reward model itself hasn't changed?
2. A token has old_prob = 0.6, new_prob = 0.9, advantage = +1.5, ε = 0.2. Compute the ratio, the unclipped objective, the clipped objective, and L_CLIP.
3. Why can't the reward model's score be backpropagated directly through the policy network the way cross-entropy loss is backpropagated during SFT?
4. Name the four neural networks active during one InstructGPT-style PPO step and state which are frozen and which receive gradient updates.
5. InstructGPT mixes a pretraining-data loss term into the PPO objective (PPO-ptx), weighted by γ. What problem was this designed to fix, and what would you expect to see if γ were set to 0?
6. Suppose the team switches the base architecture for all four models from 175B to 13B parameters, keeps Adam mixed-precision fine-tuning for the policy and value models, and keeps 80GB A100 GPUs. (a) Recompute total memory required and the minimum GPU count. (b) Does the qualitative role of the KL penalty change at the smaller scale? (c) Do the ratio/clip numbers from question 2 change?

Answers.

1. Increasing β pulls the policy more strongly back toward the frozen SFT reference distribution — it directly shrinks the effective reward for any token sequence that diverges from what π_SFT would have produced, trading reward-maximization for staying close to human-demonstrated text. At β = 0 there is no penalty at all for drifting away from the reference model, so the policy is free to exploit any blind spot in the reward model — which was only trained on a finite comparison dataset and gives unreliable scores far outside that distribution — producing text that scores highly but is degenerate, repetitive, or nonsensical to an actual human reader. The reward model didn't change; what changed is that nothing is holding the policy inside the region where the reward model's scores are trustworthy.

2. ratio = 0.9 / 0.6 = 1.5. unclipped = 1.5 × 1.5 = 2.25. clipped ratio = clip(1.5, 0.8, 1.2) = 1.2, so clipped = 1.2 × 1.5 = 1.8. L_CLIP = min(2.25, 1.8) = 1.8. The gradient beyond the 1.2 boundary is zero for this token — PPO caps the credit given for having already increased this token's probability by 50%.

3. Generating a response requires sampling a discrete token id from a probability distribution at every position, and discrete sampling has no gradient with respect to the parameters that produced the distribution. There is no differentiable path connecting "which token got sampled" back to the network weights, so a reward computed on the sampled text cannot be backpropagated the way a loss computed on fixed target tokens can. RLHF instead differentiates the log-probability the policy assigned to the tokens it actually sampled and scales that gradient by the advantage — a policy-gradient formulation that never needs to differentiate through the sampling step itself.

4. The policy π_θ (trainable), the value model V_ψ (trainable), the reference model π_SFT (frozen, used for the KL penalty), and the reward model r_φ (frozen, used to score completed responses).

5. It was designed to offset the "alignment tax" — the regression InstructGPT's authors observed on standard NLP benchmarks (tasks like translation and reading comprehension, not represented in the labelers' preference comparisons) after pure RLHF fine-tuning, caused by the policy over-specializing to the narrow style of text the reward model was trained to prefer. At γ = 0, expect the model to still follow instructions well but to show a measurably larger drop on those broader academic benchmarks relative to the base pretrained model, since nothing is pulling gradients back toward the original pretraining distribution during PPO.

6a. policy = value = 13e9 × 16 / 1e9 = 208 GB each; reference = reward = 13e9 × 2 / 1e9 = 26 GB each. Total = 208 + 208 + 26 + 26 = 468 GB. Minimum GPUs = 468 / 80 = 5.85 → 6 GPUs. 6b. No — the KL penalty's mechanism (bounding how far the policy can drift from the reference model before its effective reward is reduced) is independent of parameter count; what can change in practice is that a smaller policy may drift faster per gradient step, which sometimes calls for retuning β, but its qualitative role in the objective is unchanged. 6c. No — the clipped surrogate objective operates on token-level probability ratios, which are dimensionless numbers between 0 and 1; they depend on what probabilities the model assigns, not on how many parameters produced those probabilities. The ratio 1.5 and L_CLIP = 1.8 from question 2 hold regardless of whether the underlying model has 13B or 175B parameters.

Think About It

Think about this: How would you explain rlhf: how chatgpt was trained to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.