RLHF: Reinforcement Learning from Human Feedback

Why teams stopped training a separate reward model

In October 2023 a team at Hugging Face released Zephyr-7B-beta, a 7-billion-parameter model that beat much larger PPO-tuned models on the MT-Bench conversational benchmark. The paper, Tunstall et al., "Zephyr: Direct Distillation of LM Alignment," described a training run that never touched a PPO rollout loop. There was no value network, no clipped surrogate objective, no reward model checkpoint saved separately from the policy. The preference data came from UltraFeedback (Cui et al., 2023), a set of GPT-4-ranked completions, and the entire alignment step was a single supervised-style loss computed directly on log-probabilities. The method behind it is Direct Preference Optimization, introduced by Rafailov, Sharma, Mitchell, Ermon, Manning, and Finn in "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (NeurIPS 2023).

The standard RLHF recipe — the one used in OpenAI's InstructGPT (Ouyang et al., 2022) — trains a reward model on human preference pairs, then runs Proximal Policy Optimization to push the language model toward high-reward outputs while a KL penalty keeps it close to its starting point. That pipeline is expensive in a very specific way: a full PPO step keeps four model copies resident in GPU memory at once — the policy being updated, a frozen reference copy used only to compute the KL penalty, the reward model, and a value/critic network that estimates expected future reward — and it must repeatedly sample new completions from the policy during training, then score them, before any gradient step happens. For a lab training an Indic-language model on a constrained GPU budget, that sampling loop is often the single most expensive part of the whole alignment stage. This chapter first derives exactly what those two RLHF-stage models are trained to do — the reward model's own loss, then PPO's clipped update — and works a numeric example through each by hand, before turning to DPO's algebra, which makes the reward model and the sampling loop disappear entirely without changing what the training procedure is mathematically trying to achieve.

From the RLHF objective to a closed-form policy

Every RLHF variant, PPO-based or not, is trying to solve the same constrained optimization problem for each prompt x:

maximize over π:   E_{y~π(·|x)}[ r(x,y) ]  −  β · KL( π(·|x) || π_ref(·|x) )

Read the two terms separately. The first term says: put probability mass on completions the reward model scores highly. The second term is a leash — it punishes the policy for drifting away from the reference distribution (usually the supervised-fine-tuned checkpoint), because a policy that ignores the leash entirely will find degenerate, reward-hacking text that scores well on the reward model but reads like nothing a human would call a good answer. β sets how tight the leash is.

This objective has a closed-form solution — you do not need to run any reinforcement learning to find the optimal π for a fixed, known reward function r. Treat the objective, for a single prompt x, as a functional of the distribution π(y|x) subject to the constraint that it integrates to 1, and introduce a Lagrange multiplier λ for that constraint:

L[π] = ∫ π(y|x) r(x,y) dy  −  β ∫ π(y|x) log( π(y|x) / π_ref(y|x) ) dy  −  λ( ∫ π(y|x) dy − 1 )

Taking the functional derivative with respect to π(y|x) and setting it to zero:

r(x,y)  −  β[ log( π(y|x) / π_ref(y|x) ) + 1 ]  −  λ  =  0

Solve for the log-ratio, then exponentiate:

π*(y|x)  =  π_ref(y|x) · exp( r(x,y) / β )  ·  exp( −1 − λ/β )

The last factor doesn't depend on y at all — it's just whatever constant makes the whole thing integrate to 1 over y. Call that normalizer Z(x):

π*(y|x)  =  ( 1 / Z(x) )  ·  π_ref(y|x)  ·  exp( r(x,y) / β )

This is the exact optimal policy for any reward function r. It says the aligned model is the reference model, reweighted exponentially by reward, then renormalized. Notice what is not in this formula: no gradient ascent, no rollouts, no PPO. If you already had r(x,y), you could in principle just compute this reweighting directly. The catch, obviously, is that Z(x) requires summing over every possible completion y, which is intractable for a language model's output space. This is exactly the wall that forces standard RLHF into PPO — PPO is a way to approximate climbing toward this same optimum via sampling, without ever computing Z(x).

How the reward model is actually trained

The objective above was written with an abstract, exact reward function r(x,y). Standard RLHF never has access to that function — it only has human preference labels — so its first move, before any PPO rollout happens, is to fit a parametric stand-in: the reward model r_φ(x,y), with its own parameters φ, entirely separate from the policy's parameters θ. Everywhere the objective above uses r(x,y), standard RLHF substitutes this learned r_φ(x,y).

r_φ is fit on the same kind of preference pairs (x, y_w, y_l) that DPO will later reuse directly, under the Bradley-Terry model of pairwise comparison: the probability a human prefers y_w over y_l is modeled as a sigmoid of the reward gap,

P(y_w ⊅ y_l | x)  =  σ( r_φ(x,y_w)  −  r_φ(x,y_l) )

and φ is fit by maximizing the log-likelihood of the human labels under this model — equivalently, minimizing its negation, the reward model's training loss:

L_RM(φ)  =  − E_{(x,y_w,y_l)} [ log σ( r_φ(x,y_w) − r_φ(x,y_l) ) ]

This is the loss InstructGPT (Ouyang et al., 2022) used to produce the reward model that PPO later optimizes against. Take a concrete pair: suppose the reward model, partway through its own training, scores r_φ(x,y_w) = 2.3 and r_φ(x,y_l) = 0.7. The gap is 1.6, so:

L_RM = −log σ(1.6) = log(1 + e^−1.6) ≈ 0.1839

import math
def rm_loss(r_w, r_l):
    diff = r_w - r_l
    return diff, math.log1p(math.exp(-diff))
diff, loss = rm_loss(2.3, 0.7)
print(diff, loss)
# 1.6 0.1839007409086053

Differentiate with respect to φ and the reward model's gradient has the same shape DPO's will later have — because DPO's loss is built directly out of this one:

∇_φ L_RM  =  −σ(−diff) · [ ∇_φ r_φ(x,y_w)  −  ∇_φ r_φ(x,y_l) ]

Here σ(−1.6) ≈ 0.1680: a small but nonzero push to raise r_φ(x,y_w) and lower r_φ(x,y_l) further, shrinking as the model gets more confident on this pair — the identical saturating-gradient shape the DPO worked example below will show. Once training stops, r_φ is frozen — saved as its own checkpoint, with no further gradient ever flowing into it during the PPO stage that follows. That frozen-ness is exactly what the four-model memory bill above is paying for.

PPO: climbing toward the closed-form optimum by sampling

With r_φ frozen, standard RLHF turns to the actual reinforcement-learning step. For an episode consisting of a prompt x and completion tokens a_1...a_T sampled from the current policy π_&theta_old (a snapshot taken before this round of updates), the per-token reward used to train the policy is not just the reward model's score — InstructGPT folds the KL leash from the RLHF objective directly into the reward signal, as a per-token penalty against the frozen reference model:

reward_t(x,y)  =  r_φ(x,y) · [t = T]  −  β · log( π_&theta(a_t|s_t) / π_ref(a_t|s_t) )

(the reward-model score lands only on the final token, since r_φ scores a whole completion; the KL term applies at every step). Because this reward depends on the whole future of the episode, PPO needs an estimate of how much better or worse an action turned out to be than the value network's own prediction — this is the advantage. With a learned critic V_ψ(s_t) estimating expected future reward from state s_t, generalized advantage estimation (GAE) builds the advantage from one-step prediction errors δ_t:

δ_t  =  reward_t  +  γ·V_ψ(s_{t+1})  −  V_ψ(s_t)
Â_t  =  δ_t  +  (γλ)·δ_{t+1}  +  (γλ)²·δ_{t+2}  +  …

Take a two-token completion, γ=1 (no discounting within a single response is standard in RLHF — the whole episode is a few hundred tokens, not a long-horizon game), λ=0.95, reward_1=0.5, reward_2=1.0 (this one carries the reward-model score), and critic estimates V(s_1)=0.3, V(s_2)=0.6, V(s_3)=0 (terminal state, nothing left to predict):

δ_1 = 0.5 + 1·0.6 − 0.3 = 0.8
δ_2 = 1.0 + 1·0   − 0.6 = 0.4
Â_1 = δ_1 + (1·0.95)·δ_2 = 0.8 + 0.95×0.4 = 1.18
Â_2 = δ_2 = 0.4

Â_1 says: the first token led to a better outcome than the critic expected, by 1.18 reward-units — that is the signal PPO uses to decide how hard to push up π_θ(a_1|s_1). Define the probability ratio for a token (written ρ_t here, not r_t, to keep it visually separate from the reward r already in use in this chapter):

ρ_t(θ)  =  π_&theta(a_t|s_t) / π_&theta_old(a_t|s_t)

and PPO's clipped surrogate objective, maximized via gradient ascent:

L_CLIP(θ)  =  E_t[ min( ρ_t(θ)·Â_t ,  clip(ρ_t(θ), 1−ε, 1+ε)·Â_t ) ]

Suppose, for token 1, the policy has moved since the rollout was sampled: π_&theta_old(a_1|s_1) = 0.10, and the policy now being updated assigns π_&theta(a_1|s_1) = 0.15. With ε=0.2 and Â_1=1.18:

ρ_1(θ) = 0.15/0.10 = 1.5
unclipped term  = 1.5 × 1.18 = 1.770
clip(1.5, 0.8, 1.2) = 1.2
clipped term    = 1.2 × 1.18 = 1.416
L_CLIP = min(1.770, 1.416) = 1.416

def ppo_clip(ratio, advantage, eps):
    unclipped = ratio * advantage
    clipped = max(min(ratio, 1+eps), 1-eps) * advantage
    return unclipped, clipped, min(unclipped, clipped)
print(ppo_clip(1.5, 1.18, 0.2))
# (1.77, 1.416, 1.416)

The clip caps the credit the objective gives for pushing ρ_t past 1+ε: even though the raw importance ratio says the update looks 1.77 units good, PPO reports only 1.416, because trusting an importance-weighted estimate too far from ρ=1 (the point where π_&theta and π_&theta_old agree) is unreliable. Since clip(ρ_t(θ), ·) is a flat constant once ρ_t leaves [1−ε, 1+ε], its gradient with respect to θ is zero there — so whenever the min selects the clipped branch, the policy stops receiving gradient signal to push that token's probability any further in that direction. This plays the same role DPO's σ(−margin) factor plays below: a built-in brake that turns off once a single update has moved far enough.

The sign of Â_t changes which side of the clip binds. Take a different token, Â_t′ = −0.4 (the critic judged this action worse than expected) with the same ρ = 1.5:

unclipped = 1.5 × (−0.4) = −0.600
clipped   = 1.2 × (−0.4) = −0.480
L_CLIP = min(−0.600, −0.480) = −0.600

Here the unclipped term is the smaller (more negative) one, so the min selects it and the clip does nothing — clipping only bites on the side that would otherwise let the objective run away in the direction the raw ratio is already pushing, for that sign of advantage. This asymmetry is easy to miss and is exactly why PPO's objective is written as a min over two terms rather than a single clipped expression.

Put together, one full PPO update repeats this for every token in a batch of freshly sampled rollouts, backpropagates through π_&theta only (the reward model, the reference model, and the critic are all either frozen or trained by a separate loss), and — unlike a DPO step — must re-sample new completions from the updated policy before it can take another gradient step, because π_&theta_old has to be refreshed to whatever π_&theta became. That resampling loop, repeated for every batch of every epoch, is the expensive part DPO is about to eliminate.

Inverting the closed form: the policy IS the reward model

DPO's move is to invert the equation instead of trying to sample around the wall. Take the closed-form solution and solve for r(x,y):

r(x,y)  =  β · log( π*(y|x) / π_ref(y|x) )  +  β · log Z(x)

Every reward function has a corresponding optimal policy, and now every optimal policy has a corresponding implied reward function — read straight off its log-probability ratio against the reference model. The intractable Z(x) is still sitting there, but watch what happens when you plug this expression into the Bradley-Terry preference model (the same P(y_w ⊅ y_l) = σ(r(x,y_w) − r(x,y_l)) used above to train the standard reward model): both r(x,y_w) and r(x,y_l) share the identical x, so they share the identical β log Z(x) term, and it cancels in the subtraction:

P(y_w ⊅ y_l | x)  =  σ( β log(π*(y_w|x)/π_ref(y_w|x))  −  β log(π*(y_l|x)/π_ref(y_l|x)) )

The partition function is gone. What remains is a probability of preference expressed purely in terms of two policies' log-probabilities on two known completions — nothing intractable left to compute. Replace the unknown optimal π* with a trainable π_θ, and fit it by maximum likelihood on real human preference pairs. That likelihood, negated, is the DPO loss:

L_DPO(θ) = − E_{(x,y_w,y_l)} [ log σ( β·log(π_&theta(y_w|x)/π_ref(y_w|x))  −  β·log(π_&theta(y_l|x)/π_ref(y_l|x)) ) ]

This is the paper's headline result: training the policy directly on this loss, with ordinary backpropagation, drives π_θ toward the exact same fixed point that PPO was approximating by sampling. No reward model is ever instantiated as a separate network. No completion is ever sampled during training — y_w and y_l come straight from the static preference dataset, the same pairs a reward model would have been trained on.

Worked example: one gradient step by hand

Take a single preference pair for a prompt x, with β = 0.1. Suppose the reference model (the SFT checkpoint, before any preference tuning) assigns these log-probabilities:

log π_ref(y_w|x) = −12.0     (preferred answer — more specific, so lower prior probability)
log π_ref(y_l|x) = −10.0     (rejected answer — generic, so higher prior probability)

This is realistic: before alignment, the SFT model often assigns higher probability to the bland, hedging answer than to the sharper, correct one — that gap is exactly what preference tuning exists to close. Now suppose the policy currently being trained, partway through DPO fine-tuning, assigns:

log π_&theta(y_w|x) = −8.0
log π_&theta(y_l|x) = −11.0

Compute the two log-ratios (call them Δ_w and Δ_l):

Δ_w = log π_&theta(y_w|x) − log π_ref(y_w|x) = −8.0 − (−12.0) = 4.0
Δ_l = log π_&theta(y_l|x) − log π_ref(y_l|x) = −11.0 − (−10.0) = −1.0

Δ_w is positive: the policy has raised the preferred answer's probability well above where the reference model put it. Δ_l is negative: the policy has lowered the rejected answer's probability below the reference. Both are moving the right direction. The margin is β times the difference:

margin = β·(Δ_w − Δ_l) = 0.1 × (4.0 − (−1.0)) = 0.1 × 5.0 = 0.50

and the loss is the negative log-sigmoid of that margin, equivalently the softplus of its negation:

L = −log σ(0.50) = log(1 + e^−0.50) ≈ 0.4741

Verify this with code rather than trusting the hand arithmetic:

import math

def dpo_loss(logp_theta_w, logp_ref_w, logp_theta_l, logp_ref_l, beta):
    diff_w = logp_theta_w - logp_ref_w
    diff_l = logp_theta_l - logp_ref_l
    margin = beta * (diff_w - diff_l)
    loss = math.log1p(math.exp(-margin))   # = -log(sigmoid(margin))
    return diff_w, diff_l, margin, loss

diff_w, diff_l, margin, loss = dpo_loss(-8.0, -12.0, -11.0, -10.0, beta=0.1)
print(diff_w, diff_l, margin, loss)
# 4.0 -1.0 0.5 0.4740769841801067

Running this produces exactly 4.0 -1.0 0.5 0.474..., matching the hand computation. Now look at the gradient, because it is the part that reveals what DPO is actually doing underneath the classification-shaped loss. Differentiating L with respect to θ (the π_ref terms vanish since the reference is frozen and carries no θ-dependence):

∇_θ L  =  −β · σ(−margin) · [ ∇_θ log π_&theta(y_w|x)  −  ∇_θ log π_&theta(y_l|x) ]

Descending this gradient raises log π_&theta(y_w|x) and lowers log π_&theta(y_l|x), exactly like a contrastive loss — but the step size is scaled by σ(−margin), the probability that the current policy still gets this pair wrong. Here σ(−0.50) ≈ 0.378: a moderate, still-meaningful update, because the model has made progress on this pair but hasn't fully separated the two answers yet. As training continues and the margin grows large and positive, σ(−margin) → 0 and the gradient on this example vanishes — DPO automatically stops spending gradient budget on pairs the policy has already learned, without any explicit early-stopping rule. That reweighting is the sense in which "the language model is secretly a reward model": the quantity β log(π_&theta(y|x)/π_ref(y|x)) behaves exactly like a learned reward score, and the loss is doing weighted maximum-likelihood training against it.

What the diagram shows

The figure below traces this computation end to end for the worked example: the same prompt and pair scored under two models, the two log-ratios, their combination into a margin, and the loss — with a dashed line showing that only the trainable policy receives a gradient. The frozen reference model is queried for its log-probabilities and nothing else.

The misconception: "no reward model" does not mean "no reward"

Students who follow the derivation up to the loss formula often walk away with the claim: "DPO removes the reward from RLHF — it's just supervised learning on preferences." That is wrong in a way that matters. DPO does not remove the reward; it removes the separate network that used to represent it. The reward is still there, sitting in closed form: r̂(x,y) = β log(π_&theta(y|x)/π_ref(y|x)) + constant. Anyone can pull an implicit reward score out of a DPO-trained model for any completion, just by running it through both the tuned policy and the frozen reference and taking the log-ratio — this is exactly how DPO-trained models get plugged into best-of-n reranking or reward-model evaluations after the fact, and it's why the paper's title is literally "your language model is secretly a reward model." The thing that disappeared is not the concept of reward, it's the cost of maintaining reward as a separately trained, separately stored, separately queried network. Confusing "no separate reward network" with "no reward" also hides a real consequence: because the reward is now entangled with the policy's own weights, you cannot swap in a different, cheaper policy architecture and reuse an old DPO-trained "reward" the way you can reuse a standalone reward model across several PPO runs.

Where DPO still runs into trouble

DPO is not a free upgrade. Because y_w and y_l are fixed, off-policy samples collected once from some other model (often the original SFT model or an even earlier system), the loss never lets the policy discover a third, better completion nobody ranked — PPO's rollout step, expensive as it is, at least explores the policy's own output distribution during training. DPO can also drive log π_&theta(y_l|x) toward very negative values quickly, and because there is no KL term computed against live samples (only against the two fixed completions in each pair), the policy can drift further from the reference distribution on inputs unlike anything in the preference dataset than a PPO run with an explicit, continuously-enforced KL penalty would allow. In practice this shows up as DPO models being more sensitive to the quality and coverage of the preference dataset — UltraFeedback's scale (over 60,000 prompts, each with multiple GPT-4-ranked completions) is part of why Zephyr worked, not just the algorithm. Follow-up work such as Identity Preference Optimization (Azar et al., 2023) modifies the loss specifically to control this overfitting-to-the-pair failure mode, which is itself evidence that the "just algebra, nothing lost" framing of DPO understates what PPO's sampling loop was actually buying.

Active recall

Attempt each question before reading its answer.

Q1. Why does DPO not need to sample new completions from π_θ during training, while PPO does?

Q2. Using the worked example's log-probabilities (Δ_w = 4.0, Δ_l = −1.0), recompute the margin and loss if β is raised from 0.1 to 0.3. State every downstream effect, not just the new loss value.

Q3. True or false, with justification: "DPO eliminates reward hacking because there is no reward model left to hack."

Q4. A different pair, same prompt, β = 0.1: log π_ref(y_w|x) = −9.0, log π_ref(y_l|x) = −9.5, log π_&theta(y_w|x) = −9.2, log π_&theta(y_l|x) = −7.0. Compute the margin and loss, and say in one sentence what went wrong with training on this example.

Q5. A classmate says: "The DPO loss looks exactly like binary cross-entropy on a classifier logit, so it must be training a separate classifier the same way a reward model is trained." What is the specific error in this statement?

Q6. After DPO training finishes, how would you extract an approximate scalar reward for a single new response, without training anything further?

A1. Because the closed-form derivation shows the optimal policy can be characterized entirely through its log-probability ratio against a frozen reference model, evaluated only on the fixed (y_w, y_l) pairs already collected by human annotators. PPO needs fresh rollouts because it is approximating that same optimum through sampling and a learned reward and value function, with no closed form available once the reward model replaces the true, unknown human reward function — and because its clipped objective is only valid relative to the policy that generated the current batch, π_&theta_old, which must be refreshed after every update.

A2. The immediate recompute: margin = 0.3 × (4.0 − (−1.0)) = 0.3 × 5.0 = 1.5, and loss = −log σ(1.5) = log(1+e^−1.5) ≈ 0.2014 — smaller than the β=0.1 loss of 0.474, because the same log-ratio gap now produces a larger margin, and the loss shrinks as the margin grows. But two further ripples follow from changing β, not just this one static recomputation. First, the gradient weight σ(−margin) drops from σ(−0.5)≈0.378 to σ(−1.5)≈0.182 — at β=0.3, a training run would take smaller effective steps on well-separated pairs sooner, since β scales how "confident" the same underlying log-ratio gap looks to the loss. Second, β is not just a loss-shape knob — it is literally the strength of the KL leash from the original RLHF objective. Raising it to 0.3 means every future gradient step will more strongly resist moving π_θ away from π_ref, so the log-probabilities themselves (currently −8.0 and −11.0) would evolve differently over subsequent steps than they did under β=0.1 — this recomputation is only valid as a snapshot; it does not predict where training converges.

A3. False. Reward hacking in RLHF means the model finds outputs that score well on the reward signal without being genuinely good — nothing in DPO's derivation removes the reward signal, it only removes the standalone network. The reward is still implicitly r̂(x,y) = βlog(π_&theta(y|x)/π_ref(y|x)), and DPO can just as easily overfit to spurious patterns in the preference data (for example, learning that longer answers were preferred more often, and inflating length rather than quality) as a PPO-trained reward model can. The absence of a separate reward network makes this failure mode harder to diagnose, if anything, since there's no standalone reward score to sanity-check against held-out data before deploying the policy.

A4. Δ_w = −9.2 − (−9.0) = −0.2. Δ_l = −7.0 − (−9.5) = 2.5. margin = 0.1 × (−0.2 − 2.5) = 0.1 × (−2.7) = −0.27. loss = −log σ(−0.27) = log(1+e^0.27) ≈ 0.837, noticeably higher than the well-behaved example. The negative margin means the current policy has moved in the wrong direction on this pair relative to the reference: it slightly lowered the preferred answer's probability (Δ_w < 0) while sharply raising the dispreferred one's probability (Δ_l = 2.5, large and positive) — training has regressed on this example, and the large loss correctly signals a large corrective gradient is needed.

A5. The error is in what receives the gradient. A trained reward model is a separate set of weights, frozen once trained, later queried during a different process (PPO rollouts) that does not backpropagate through it. In the DPO loss, the "logit" inside the sigmoid is recomputed fresh every step directly from π_&theta's own output probabilities, and the gradient of the loss flows straight into π_&theta's weights — there is no frozen classifier anywhere in the loop, and nothing is trained and then reused unchanged the way a reward model is.

A6. Run the response y through both the DPO-tuned policy and the original frozen reference model to get log π_&theta(y|x) and log π_ref(y|x), then compute β·(log π_&theta(y|x) − log π_ref(y|x)). This recovers the implicit reward up to the unknown additive constant β log Z(x), which is the same for every completion of a given prompt and therefore cancels whenever you're comparing or ranking multiple responses to that one prompt — exactly the operation used for best-of-n reranking with a DPO model.

Think About It

Think about this: How would you explain rlhf: reinforcement learning from human feedback to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind rlhf: reinforcement learning from human feedback, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.