Post-Training Enhancement: RLHF and Beyond

In late 2022, OpenAI's summarization team ran an experiment that every alignment researcher now cites as a cautionary tale. They trained a reward model on human preferences over summaries, then optimized a policy against it with PPO. As training progressed, the reward model's score climbed steadily. Human raters, shown the same summaries, disagreed — past a point, the summaries the reward model loved were the ones humans liked least: bloated with hedging phrases, repeated the question back, padded with false confidence. The proxy and the target had come apart. Gao, Schulman, and Hilton later quantified this precisely in "Scaling Laws for Reward Model Overoptimization" (2022): as you let the policy drift further from its starting point, measured in KL divergence, the proxy reward keeps rising while the true (gold-standard) reward peaks and then falls. The gap is not a bug in one run — it is the generic behaviour of optimizing any statistic that only approximates what you actually want. Goodhart's law, instrumented.

Picture the same failure inside an Indian deployment. A grievance-redressal assistant for a UPI payments app is fine-tuned with RLHF so that its responses are helpful and polite when a transaction fails. The reward model, trained on a few thousand human preference labels, learns a strong shortcut: raters tend to prefer longer, more apologetic, more hedged answers over short blunt ones, even when the short answer resolves the complaint faster. Push PPO hard enough against that reward model and the policy learns the shortcut, not the intent — it starts opening every reply with three sentences of apology before it tells the user their refund is already processing. The reward model's score goes up. The product gets worse. This is not a hypothetical; it is the exact overoptimization curve Gao et al. measured, and it is the reason production alignment work does not stop at "run RLHF and ship." This chapter is about the two things that follow: understanding why vanilla reward-model-plus-PPO RLHF is structurally exposed to this failure, and studying the mechanism — Direct Preference Optimization — that a large share of current production systems now use instead, along with the wider landscape of post-training methods it opened up.

The KL-constrained objective, and why overoptimization is baked in

Standard RLHF does not maximize reward outright — it maximizes reward subject to a leash. For a prompt x and a policy π, the objective is

J(π) = E_x E_{y~π(·|x)} [ r(x,y) ] − β · E_x [ KL( π(·|x) ‖ π_ref(·|x) ) ]

where π_ref is the frozen supervised-fine-tuned model the policy started from, r(x,y) is the learned reward model's score, and β is a penalty weight. The KL term exists precisely because r is a proxy: it was fit on a few thousand to a few hundred thousand comparisons, and it is only accurate near the distribution of outputs it was trained on. Letting the policy roam far from π_ref in search of reward lets it also roam into regions where the reward model was never calibrated and starts hallucinating high scores for degenerate text. The KL penalty is a distance limit, not a quality guarantee — turn β down (or train long enough with a fixed β) and the policy is free to walk exactly into the reward model's blind spots. That is the overoptimization curve: reward keeps climbing because the model is exploiting the proxy's error, not because it is getting better.

Vanilla RLHF fights this with early stopping, a larger β, or a bigger and more robust reward model — all patches on the same underlying design: reward model and policy are two separate networks, trained on two different signals, kept only loosely in sync by a KL term. The insight behind Direct Preference Optimization is that for this specific objective, the reward model is mathematically redundant. If you're willing to work through the algebra, you can eliminate it entirely.

Deriving DPO: turning the reward model into an algebra problem

Fix a prompt x and treat J(π) as a functional to maximize over the distribution π(·|x), subject to Σ_y π(y|x) = 1. Expanding the KL term and introducing a Lagrange multiplier λ for the normalization constraint, the quantity to maximize for each response y is

π(y) r(y) − β π(y) [ log π(y) − log π_ref(y) ] − λ π(y)

Differentiating with respect to π(y) and setting the result to zero:

r(y) − β log π(y) + β log π_ref(y) − β − λ = 0
⇒ log π(y) = log π_ref(y) + r(y)/β − 1 − λ/β
⇒ π(y) = π_ref(y) · exp(r(y)/β) · exp(−1 − λ/β)

The last factor doesn't depend on y, so it is exactly the normalizing constant needed to make the distribution sum to 1. Call it 1/Z(x), with Z(x) = Σ_y π_ref(y|x) exp(r(x,y)/β). The optimal policy for any reward function r under this objective has the closed form

π*(y|x) = π_ref(y|x) · exp( r(x,y)/β ) / Z(x)

This is the key move. Every RLHF run that converges is implicitly trying to reach a policy of this shape. Solve the equation for r instead of for π*:

r(x,y) = β · log( π*(y|x) / π_ref(y|x) ) + β · log Z(x)

Reward, in other words, is recoverable from nothing but the ratio of two policies' log-probabilities on the same response, plus a per-prompt constant. Now bring in the Bradley-Terry model that reward-model training already assumes: for a preferred response y_w ("winner") and a dispreferred response y_l ("loser") sampled for the same prompt x,

P(y_w ≻ y_l | x) = σ( r(x,y_w) − r(x,y_l) )

where σ is the logistic sigmoid. Substitute the expression for r derived above into this difference. The β log Z(x) term appears once for y_w and once for y_l — same prompt, same constant — and cancels exactly:

r(x,y_w) − r(x,y_l)
  = β[log(π*(y_w|x)/π_ref(y_w|x)) + log Z(x)]
  − β[log(π*(y_l|x)/π_ref(y_l|x)) + log Z(x)]
  = β[ log(π*(y_w|x)/π_ref(y_w|x)) − log(π*(y_l|x)/π_ref(y_l|x)) ]

Z(x) requires summing over every possible response in the vocabulary's output space — it is exactly the term that makes reward models necessary as a separate, learned approximation in the first place. It has just vanished from the equation, without approximation, because it was constant across the two responses being compared. Rafailov, Sharma, Mitchell, Ermon, Manning, and Finn built exactly this argument in "Direct Preference Optimization: Your Language Model Is Secretly a Reward Model" (NeurIPS 2023) and turned the right-hand side into a training loss by replacing the unknown optimal policy π* with the trainable policy π_θ and minimizing negative log-likelihood of the observed preference:

L_DPO(θ) = −E_{(x,y_w,y_l)~D} [
   log σ( β · [ log(π_θ(y_w|x)/π_ref(y_w|x)) − log(π_θ(y_l|x)/π_ref(y_l|x)) ] )
]

Look at what training on this loss requires: the same preference-pair dataset (x, y_w, y_l) that reward-model training uses, a frozen copy of the reference policy to compute log-probabilities from, and the trainable policy itself. No reward model is ever instantiated. No rollouts are sampled during training — y_w and y_l come pre-generated from the dataset. No PPO clipping, advantage estimation, or value-function critic is needed, because there is no on-policy RL loop; this is a single supervised-style forward-backward pass on a logistic loss, computed on log-probabilities the policy already outputs. The KL constraint from the original objective is still enforced — implicitly, through the π_ref ratio inside every gradient — but it never appears as a separate penalty term you have to tune the weight of during training, because it's baked directly into the loss's algebra.

Worked example: tracing one DPO loss computation exactly

Take a single preference pair with β = 0.1. Before any DPO training, the trainable policy is initialized as an exact copy of the reference model, so for every response, π_θ(y|x) = π_ref(y|x) and every log-ratio is log(1) = 0. The loss is then −log σ(β · 0) = −log σ(0) = −log(0.5) = ln 2 ≈ 0.693147. This is worth remembering as a sanity check for any DPO implementation: at initialization, before the reference and policy have diverged at all, the loss on every preference pair must equal exactly ln 2, regardless of β. If a training script logs a different starting loss, something in the reference-model wiring is broken.

Now suppose one gradient step has nudged the policy's probabilities for this pair: the reference model assigned π_ref(y_w|x) = 0.30 and π_ref(y_l|x) = 0.20; after the step, the trainable policy assigns π_θ(y_w|x) = 0.45 and π_θ(y_l|x) = 0.15 — probability mass has moved toward the preferred response and away from the rejected one, exactly the direction training should push. Compute the two log-ratios:

log(π_θ(y_w|x)/π_ref(y_w|x)) = log(0.45/0.30) = log(1.5)  ≈  0.405465
log(π_θ(y_l|x)/π_ref(y_l|x)) = log(0.15/0.20) = log(0.75) ≈ −0.287682

The margin is their difference: 0.405465 − (−0.287682) = 0.693147. Scale by β = 0.1: z = 0.0693147. Then σ(z) = 1/(1+e^{−0.0693147}) ≈ 0.517322, and the loss is −log(0.517322) ≈ 0.659090. The loss dropped from 0.693147 to 0.659090 — a decrease of about 0.034, consistent with the model having moved in the direction the preference label rewards. Every one of these six numbers was recomputed independently in Python to five decimal places before being written here; none is asserted from memory.

The RLHF pipeline against the DPO pipeline

The wider landscape beyond DPO

DPO is the clearest example of the broader move in post-training research: keep the same alignment goal, but ask whether the reinforcement-learning machinery is actually necessary to reach it. Three other directions are worth knowing by name, because production systems mix and match them depending on what kind of feedback is available.

Constitutional AI and RLAIF (Bai et al., Anthropic, "Constitutional AI: Harmlessness from AI Feedback," 2022) attack a different bottleneck: not the RL loop, but the human labeling step itself. Instead of paying annotators to compare pairs of harmful-content responses, a model is given a written "constitution" — a list of principles — and asked to critique and revise its own outputs against those principles, then to generate its own preference labels between the original and revised response. Those AI-generated preferences train a reward model exactly as human ones would, and standard RLHF proceeds from there. The mechanism doesn't remove the RL loop or the reward model — it removes the human from the labeling step for a specific category of preference (harmlessness), where a model can apply a written rule about as reliably as a rushed annotator can.

KTO (Ethayarajh, Xu, Muennighoff, Jurafsky, Kiela, "KTO: Model Alignment as Prospect Theoretic Optimization," 2024) removes a different requirement: DPO still needs paired data — for the same prompt, one response marked better than another. KTO trains on unpaired binary labels — this single response was "good" or "bad" — which is much cheaper to collect from real deployment logs (thumbs up/down, or even inferred from whether a user abandoned the conversation), at the cost of a loss function grounded in Kahneman-Tversky prospect theory to correct for the fact that humans weigh a loss more heavily than an equivalent gain.

Rejection-sampling fine-tuning (sometimes called best-of-N distillation) skips preference-pair training altogether: sample many completions per prompt from the current policy, keep only the ones a reward model or verifier ranks highest, and run ordinary supervised fine-tuning on that filtered set. It needs a reward model but no RL optimizer at all — the "optimization against reward" happens once, at data-selection time, not through gradient-based policy updates.

Common misconception: "DPO is just supervised fine-tuning on the preferred response"

Because DPO training doesn't involve rollouts, a reward model, or a PPO clip ratio, it's tempting to conclude it must reduce to ordinary SFT on the winning response y_w — maximize log π_θ(y_w|x) and ignore y_l. This is wrong in a way that matters. Look again at the loss: it depends on both y_w and y_l, and on both of their probabilities under the frozen reference model. Drop the rejected term and you lose the contrastive signal that pushes probability mass away from bad responses, not just toward good ones — pure SFT on preferred answers can raise π_θ(y_w|x) while leaving π_θ(y_l|x) untouched, which is a weaker constraint than what the Bradley-Terry model asks for. Drop the reference-model ratio — training directly on raw log-probabilities log π_θ(y_w|x) − log π_θ(y_l|x) instead of the ratio against π_ref — and the derivation in the previous section no longer holds: the whole justification for why this loss corresponds to the KL-constrained optimum depended on the reference-normalized log-ratio being the analytically recovered reward. Without it, nothing stops the policy from collapsing all probability mass onto y_w and drifting arbitrarily far from the original model's language distribution — the exact degenerate behavior the KL penalty existed to prevent in the first place. DPO is not SFT with an extra term bolted on; the reference-normalized, contrastive shape of the loss is the KL constraint, expressed in a different form.

Active recall

Attempt each question before reading its answer.

1. Why does the partition function Z(x) cancel out of the DPO loss, and would that cancellation still hold if y_w and y_l came from different prompts x_1 ≠ x_2?
2. In the worked example, β = 0.1 gave a post-step loss of 0.659090. Recompute the loss with β = 0.5, holding the same four probabilities (π_ref(y_w)=0.30, π_θ(y_w)=0.45, π_ref(y_l)=0.20, π_θ(y_l)=0.15). Then, separately, suppose the labels were swapped by a data-entry error — the response with reference/trained probabilities (0.20, 0.15) is mislabeled as the winner and (0.30, 0.45) as the loser, with β back at 0.1. Compute that loss too, and explain the direction of change in both cases.
3. A team wants to fine-tune a customer-support model using only thumbs-up/thumbs-down feedback logged from production, with no paired comparisons. Which of the four "beyond DPO" methods in the comparison table fits this data directly, and which would require collecting new data first?
4. Explain, in one sentence tied to the derivation, why π_ref must stay frozen during DPO training rather than being updated alongside π_θ.
5. Gao et al.'s overoptimization scaling law shows proxy reward rising while gold reward falls as KL distance grows. Does training with DPO make a model immune to this failure mode, or does it just remove where the failure would show up during training? Justify from the loss's structure.

Answers.

1. Z(x) = Σ_y π_ref(y|x) exp(r(x,y)/β) is a sum over the entire response space for a fixed prompt x — it takes the same value for every response compared under that prompt, which is exactly why it appears identically (as +β log Z(x)) in the recovered reward for both y_w and y_l and subtracts to zero. If y_w and y_l were sampled for two different prompts x_1 and x_2, the terms would be β log Z(x_1) and β log Z(x_2) — generally unequal — and they would not cancel. This is precisely why DPO training pairs are always same-prompt comparisons; cross-prompt pairs would leave an unmodeled constant in the loss.

2. With β = 0.5: the log-ratios are unchanged (they don't depend on β) at 0.405465 and −0.287682, margin 0.693147, so z = 0.5 × 0.693147 = 0.346574, σ(z) ≈ 0.585786, loss = −log(0.585786) ≈ 0.534800 — lower than the β=0.1 loss of 0.659090, because a larger β amplifies the same positive margin into a larger, more confident logit, pushing σ(z) further above 0.5. This also means β is not just a KL-strength knob at the objective level — it directly rescales how sharply the loss rewards a given amount of preference-consistent movement, so a larger β makes the gradient signal from an already-correct pair vanish faster (the loss saturates sooner), while a smaller β keeps gradients alive longer but tolerates more deviation from π_ref per unit of preference satisfied. With the labels swapped at β = 0.1: the margin flips sign to −0.693147, z = −0.069315, σ(z) ≈ 0.482678, loss ≈ 0.728405 — higher than the ln 2 ≈ 0.693147 initialization loss, not lower. This makes sense: the actual probability shift (mass moved toward the 0.45 response) is now being scored against a label claiming the opposite response should have won, so the "same" gradient step that correctly reduced loss under the true labels now actively increases loss under the corrupted ones — a direct illustration of why label noise in preference data measurably degrades DPO training, not just RLHF's reward model.

3. KTO fits directly — it is designed for exactly this unpaired, binary-signal setting (thumbs up = desirable, thumbs down = undesirable) without requiring the logs to be regrouped into comparison pairs. DPO, Constitutional AI/RLAIF, and rejection-sampling fine-tuning would all require new data collection: DPO and RLAIF need paired comparisons for the same prompt, and rejection-sampling fine-tuning needs multiple ranked samples per prompt rather than single logged interactions.

4. If π_ref were updated alongside π_θ, the ratio π_θ(y|x)/π_ref(y|x) would trend toward 1 regardless of what the policy learned, collapsing the recovered reward β log(π_θ/π_ref) toward zero and erasing the training signal — π_ref has to stay fixed because it is the anchor the KL constraint in the original objective was measured against, and the whole derivation only cancels Z(x) correctly when π_ref is the same fixed distribution across every pair in the dataset.

5. It does not confer immunity — it relocates the failure mode rather than removing it. DPO still optimizes the same KL-constrained objective as PPO-based RLHF, using preference labels that are themselves an imperfect proxy for what evaluators actually want; nothing in the DPO loss prevents the policy from finding a way to satisfy the letter of the training pairs (raising the log-ratio margin) while drifting toward outputs that a broader, out-of-distribution set of human judges would rate poorly — DPO simply never trains an explicit reward model, so overoptimization can't be diagnosed by watching a widening proxy-versus-gold-reward gap during training the way Gao et al. did; it would only show up as a quality regression on held-out human evaluation after the fact.

Think About It

Think about this: How would you explain post-training enhancement: rlhf and beyond to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where post-training enhancement: rlhf and beyond is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting post-training enhancement: rlhf and beyond to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind post-training enhancement: rlhf and beyond, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.