Direct Preference Optimization: Learning from Preferences

A WhatsApp Refund Bot Learns to Ramble

A Bengaluru D2C startup fine-tunes its WhatsApp support bot with Direct Preference Optimization. Support agents label thousands of pairs: for the same customer message, one bot reply is marked "chosen" and one "rejected." The team trains, the held-out preference accuracy climbs past 80%, and they ship. Two weeks later, customers complain the bot has started writing paragraphs where a single line would do: "Your refund request has been received and is currently being processed by our team, and we want to assure you that we take this matter seriously..." for a question that only needed "Refund of ₹499 will land in 3-5 working days." Nothing in the training pipeline told the model to prefer length. Nobody labeled length as a criterion. Yet the model learned it anyway, because of how DPO's implicit reward is built out of raw sequence probabilities.

This chapter assumes you already know how to derive the DPO loss itself, starting from the KL-constrained RLHF objective, substituting the Bradley-Terry preference model, and cancelling the reward model and PPO rollout out of the picture entirely. That derivation lives in the companion RLHF chapter. Here the question is different: once you have that closed-form loss, what does the reward it implicitly learns actually behave like, where does it break in practice, and how do production systems built on it fix those breaks and fit inside real GPU budgets?

Fast Recap: The Object We're Analyzing

DPO's closed-form loss is a function of a policy π_θ, a frozen reference policy π_ref (almost always the SFT checkpoint the run started from), a preference pair (y_w, y_l) for prompt x, and a temperature β:

L_DPO(θ) = -E[(x,yw,yl)] log σ( β log(πθ(yw|x)/πref(yw|x)) - β log(πθ(yl|x)/πref(yl|x)) )

Define the implicit reward r_θ(x,y) = β log(π_θ(y|x)/π_ref(y|x)) and the margin Δ_θ(x,y_w,y_l) = r_θ(x,y_w) − r_θ(x,y_l). The loss is then just L_DPO = −log σ(Δ_θ). Everything below is about what this r_θ actually is, what it is not, and what happens when you optimize Δ_θ at scale on real data.

The Reward You Cannot Compare Across Prompts

The RLHF objective's optimal policy has a known closed form: π*(y|x) = π_ref(y|x) exp(r(x,y)/β) / Z(x), where Z(x) = Σ_y π_ref(y|x) exp(r(x,y)/β) is a per-prompt normalizing constant, a sum over every possible response the model could generate for that prompt. Inverting this relation to solve for the true reward in terms of the optimal policy gives r(x,y) = β log(π*(y|x)/π_ref(y|x)) + β log Z(x). DPO's r_θ(x,y) is exactly the first term; it never recovers β log Z(x), because that term is analytically intractable (it requires summing over the model's entire response space) and, more importantly, it does not need to be recovered: in the Bradley-Terry loss, y_w and y_l share the same prompt x, so Z(x) appears identically in both r_θ(x,y_w) and r_θ(x,y_l) and cancels exactly in the subtraction that forms Δ_θ. The loss is completely insensitive to it.

Common misconception: students who have just seen the DPO derivation often assume the model now has a genuine, general-purpose reward model bolted onto it, one that can be used the way a standalone reward model is used elsewhere in RLHF: score any completion for any prompt and rank them. That is false, and the algebra above shows exactly why. r_θ(x,y) is only correct up to an unknown additive constant that depends on x, and that constant differs from prompt to prompt in a way training never measures. Comparing r_θ(x₁, y₁) against r_θ(x₂, y₂) for two different prompts x₁ and x₂, say to rerank a mixed batch of generations from different customer queries by "which response is best overall," is comparing two numbers with different, unknown offsets baked in. It can look meaningful and still be wrong. The only valid use of r_θ is differencing two completions of the same prompt, which is exactly the operation the training loss performs and nothing more.

How Length Becomes a Confound

r_θ(x,y) = β log π_θ(y|x) − β log π_ref(y|x), and under teacher forcing log π_θ(y|x) is a sum of per-token conditional log-probabilities, one term for every token in y. A response with twice as many tokens contributes roughly twice as much raw log-probability mass to the sum, even if every token is exactly as well-predicted as before. Nothing in the DPO loss divides that sum by the number of tokens. So if the training data has any correlation between length and human preference, and it almost always does, because annotators tend to reward thoroughness even when a comparably good short answer exists, the implicit reward learns "longer minus shorter" as a usable, low-effort direction to increase Δ_θ on the training pairs, and that direction generalizes to prompts where brevity was actually the better answer. The refund bot did not learn "be helpful." It partly learned "produce more tokens raises your score."

Worked Example: Raw Reward vs. Length-Normalized Reward

Take the prompt x = "Refund of ₹499 hasn't arrived, what now?" and three candidate responses under a partially-trained policy π_θ and its frozen reference π_ref: A is a 4-token, correct, concise answer; B is an 8-token, equally correct but more elaborate answer; C is a 4-token, incorrect answer. To isolate the length effect cleanly, assign A and B the identical per-token quality: under π_θ every token in both averages −0.30 nats of log-probability (call this "equally good writing"), and under π_ref every token in both averages −0.45 nats (so training has genuinely improved both by the same +0.15 nats/token). C is worse: −0.55 nats/token under π_θ, still −0.45 under π_ref (same starting point, since training should be pushing its probability down, not up).

log πθ(A)  = 4 × (-0.30) = -1.20     log πref(A) = 4 × (-0.45) = -1.80
log πθ(B)  = 8 × (-0.30) = -2.40     log πref(B) = 8 × (-0.45) = -3.60
log πθ(C)  = 4 × (-0.55) = -2.20     log πref(C) = 4 × (-0.45) = -1.80

With β = 0.1, the raw DPO implicit reward for each response is r(y) = β(log π_θ(y) − log π_ref(y)):

r(A) = 0.1 × (-1.20 - (-1.80)) = 0.1 × 0.60  =  0.060
r(B) = 0.1 × (-2.40 - (-3.60)) = 0.1 × 1.20  =  0.120
r(C) = 0.1 × (-2.20 - (-1.80)) = 0.1 × (-0.40) = -0.040

r(B) is exactly double r(A), purely because B has exactly double the tokens; per-token quality is by construction identical. This is the length artifact, laid bare arithmetically, not asserted. Now check what the actual training pair, B preferred over C, produces. The margin is Δ = r(B) − r(C) = 0.120 − (−0.040) = 0.160. The DPO loss is L = −log σ(0.160). Computing σ(0.160) = 1/(1+e^−0.16) = 1/(1+0.8522) = 0.5399, so L ≈ −ln(0.5399) ≈ 0.616 nats, and the gradient weight on this pair, σ(−Δ) = 1 − 0.5399 = 0.460, is still large: the model has plenty of room left to learn from this pair. This is the healthy part of training. The problem surfaces only when you compare B against A, which is not even a labeled pair, just two equally good candidates the trained model might generate at inference time: raw reward ranks B strictly above A for no reason but length.

SimPO (Meng, Xia and Chen, 2024) removes both the reference model and the length artifact in one change: it defines the reward as the length-normalized average log-probability itself, r_SimPO(x,y) = (β/|y|) log π_θ(y|x), with no π_ref term at all, and trains with a loss that additionally requires the margin to clear a fixed target γ rather than just being positive. Applying the normalized reward to the same three responses:

r_simpo(A) = (0.1/4) × (-1.20) = 0.025 × (-1.20) = -0.030
r_simpo(B) = (0.1/8) × (-2.40) = 0.0125 × (-2.40) = -0.030
r_simpo(C) = (0.1/4) × (-2.20) = 0.025 × (-2.20) = -0.055

A and B now tie exactly, because average per-token quality was identical by construction, and C, the actually worse response, still ranks clearly lowest. Dividing by length did not just rescale the numbers; it removed length as a variable the reward can respond to at all, leaving only what the loss was always supposed to measure: how well each token is predicted.

Here is the corresponding training-time computation, the piece every DPO implementation actually runs per batch, teacher-forcing both completions through the policy and the frozen reference and reducing to the scalar loss above:

import torch
import torch.nn.functional as F

def sequence_logprob(logits, labels, loss_mask):
    """
    logits: (batch, seq_len, vocab)  -- model output BEFORE softmax
    labels: (batch, seq_len)         -- token ids, shifted so labels[t] is
                                         the token that follows position t
    loss_mask: (batch, seq_len)      -- 1.0 on response tokens, 0.0 on
                                         prompt/padding tokens
    returns: (batch,) summed log-prob of the response tokens only
    """
    log_probs = F.log_softmax(logits, dim=-1)               # stable, no log(softmax(.))
    token_logp = torch.gather(
        log_probs, dim=-1, index=labels.unsqueeze(-1)
    ).squeeze(-1)                                            # (batch, seq_len)
    return (token_logp * loss_mask).sum(dim=-1)              # (batch,)

def dpo_loss(policy_logp_w, policy_logp_l,
             ref_logp_w, ref_logp_l, beta=0.1):
    pi_ratio  = policy_logp_w - policy_logp_l
    ref_ratio = ref_logp_w  - ref_logp_l
    margin = beta * (pi_ratio - ref_ratio)
    return -F.logsigmoid(margin).mean()                      # -log σ(margin)

Feeding this function the B-versus-C numbers above, policy_logp_w=-2.40, policy_logp_l=-2.20, ref_logp_w=-3.60, ref_logp_l=-1.80, beta=0.1, gives margin = 0.1 × ((-2.40-(-2.20)) - (-3.60-(-1.80))) = 0.1 × ((-0.20) - (-1.80)) = 0.160, and -F.logsigmoid(0.160) evaluates to the same 0.616 computed by hand, confirming the two independent computation paths (reward-difference and ratio-difference) agree, as they must algebraically.

An Unbounded Loss: Why DPO Never Stops Pushing

−log σ(Δ) is a strictly decreasing function of Δ that only asymptotes to zero as Δ → ∞; it never reaches a minimum at any finite margin. That means, for every training pair, gradient descent on the DPO loss keeps rewarding a larger and larger margin between chosen and rejected forever, there is no point at which the loss says "this pair is separated enough, stop." In practice this shows up as policies that grow increasingly confident, increasingly close to deterministic, on the training distribution, especially once β is small or the preference labels are close to unambiguous: the KL term that was supposed to keep π_θ anchored to π_ref is only a soft penalty inside the loss's shape, not a hard constraint, and an unbounded loss can simply out-argue it during optimization.

Azar et al. (2023) diagnose this directly and propose ψPO / IPO as a fix: instead of a Bradley-Terry log-likelihood, minimize a bounded squared-error loss between the margin and a fixed target τ, L_IPO(θ) = (Δ_θ(x,y_w,y_l) − τ)², where τ plays a role similar to 1/β in controlling how confidently the model is allowed to separate the pair. This loss has a genuine minimum at Δ_θ = τ; overshoot it, and the loss starts rising again, actively pulling the policy back. The diagram below plots both loss shapes against the same margin axis using the exact functional forms above, and marks the two β = 0.1 and β = 0.2 points computed in the worked example on the DPO curve.

Read left to right, the red IPO curve is a parabola bottoming out at Δ = τ and rising on both sides: undershoot the target and the loss punishes you, overshoot it and the loss punishes you just the same, which is exactly the finite stopping point DPO lacks. The blue DPO curve keeps sliding toward zero the whole way to the right edge of the plot and beyond; there is no Δ at which it turns around. The two dots mark the actual β = 0.1 and β = 0.2 losses computed in the worked example: notice that doubling β moved the margin from 0.160 to 0.320 and dropped the loss from 0.616 to 0.546, sliding further down that same never-bottoming curve, not toward any target.

Building the Training Loop: Memory and Numerical Stability

Two implementation details in the code above are not cosmetic. First, F.log_softmax is used instead of torch.log(F.softmax(logits)); the latter can underflow to -inf for any token the model assigns very low probability, since it computes the softmax first and only takes the log afterward, while log_softmax computes the log-sum-exp directly and stays numerically stable for arbitrarily confident or unconfident logits. Second, F.logsigmoid(margin) replaces torch.log(torch.sigmoid(margin)) for the same reason: computed the naive way, very negative margins saturate sigmoid to exactly 0.0 in floating point and the subsequent log call is -inf; logsigmoid is implemented as the numerically stable -softplus(-margin) internally and never does.

The systems-level payoff of eliminating the explicit reward model and the PPO critic is GPU memory, and it is worth putting a number on it. A 7B-parameter model stored in bf16 takes 2 bytes per parameter, about 14 GB per copy, before counting optimizer state, activations, or KV cache. A PPO-based RLHF run of the kind used for InstructGPT-style training holds up to four separate networks resident at once: the policy being trained, the frozen reference for the KL penalty, the reward model, and the critic (value network), roughly 4 × 14 GB = 56 GB in weights alone. DPO needs exactly two: the policy and the frozen reference, 2 × 14 GB = 28 GB, half the footprint, because the reward model and critic never existed in the first place and the PPO rollout loop that needed the critic is gone too.

Production DPO trainers push this further with LoRA. If the policy is trained as a frozen base model plus a low-rank adapter, the reference model does not need a second full copy in memory at all: it is the same base weights with the adapter disabled. Toggling the adapter on and off around the two forward passes in sequence_logprob gets you π_θ and π_ref log-probs from one resident 14 GB base model, plus an adapter typically well under a gigabyte. A run that would have needed 56 GB of weights under PPO can fit its full policy-plus-reference DPO setup on a single 24-40 GB training GPU.

Active Recall

Attempt each question before reading its answer.

1. Why is it invalid to use a trained DPO model's r_θ(x,y) to rank completions of two different prompts against each other, even though it is perfectly valid to compare r_θ(x,y_w) against r_θ(x,y_l) for the same prompt?
2. The refund-bot dataset has annotators preferring longer answers by roughly 15% average length at equal quality. Name the fix from this chapter that removes the resulting bias, and state the one-line mechanism by which it does so.
3. Suppose β is doubled from 0.1 to 0.2 for the SimPO comparison in the worked example. Recompute r_SimPO(A), r_SimPO(B), r_SimPO(C), and separately recompute the raw DPO loss and gradient weight for the B-vs-C pair. Does the qualitative conclusion, A ties B and both beat C, survive?
4. In the worked example, response A is 4 tokens at -0.30 nats/token. If A were shortened to 2 tokens while keeping the same -0.30 nats/token quality, what happens to r_SimPO(A)? What happens to the raw DPO r(A)?
5. Why does the IPO loss have a well-defined minimum while the DPO loss does not, and what training-time symptom does this predict for DPO when preference labels are nearly deterministic?
6. A 13B model in bf16 takes about 26 GB per full copy. Give the total weight memory for a PPO-based RLHF run (policy, reference, reward model, critic) versus a full-finetuning DPO run (policy, reference), and name the one adaptation technique from this chapter that removes the reference model's memory cost almost entirely.

Answers

1. r_θ(x,y) = β log(π_θ(y|x)/π_ref(y|x)) is the true reward plus an unknown, prompt-dependent constant β log Z(x) that DPO's loss never estimates, because Z(x) cancels exactly when both terms in the margin share the same x. Compare two completions of the same prompt and the constant cancels again in your own comparison, so it is harmless. Compare completions of two different prompts and the two different unknown constants do not cancel, so the comparison is contaminated by an offset you cannot measure or correct for.

2. SimPO (Meng, Xia and Chen, 2024): it replaces raw summed log-probability with the length-normalized average, r_SimPO(x,y) = (β/|y|) log π_θ(y|x), and drops the reference model. Dividing by |y| means a longer response can no longer accumulate a larger reward purely by having more tokens; only the average per-token quality moves the score, exactly as shown numerically for A and B in the worked example, where the raw rewards differed by a factor of 2 (matching the length ratio) but the normalized rewards were identical.

3. SimPO side, scale by β=0.2 instead of 0.1: r_SimPO(A) = (0.2/4)×(-1.20) = -0.060, r_SimPO(B) = (0.2/8)×(-2.40) = -0.060, r_SimPO(C) = (0.2/4)×(-2.20) = -0.110. A and B still tie exactly and C is still lowest; doubling β is a uniform rescaling and cannot change a ranking. DPO side, the margin for B-vs-C becomes Δ = 0.2×1.60 = 0.320 (recall pi_ratio - ref_ratio = 1.60 from the original computation, unaffected by β). σ(0.320) = 1/(1+e^-0.32) = 1/1.7262 = 0.579, so L = -ln(0.579) ≈ 0.546 nats (down from 0.616), and the gradient weight σ(-Δ) = 1 - 0.579 = 0.421 (down from 0.460). So: the SimPO ranking is invariant to β, but the raw DPO loss value and its gradient weight both shrink as β grows, exactly the two points marked on the diagram's DPO curve.

4. r_SimPO(A at 2 tokens) = (0.1/2) × (2 × -0.30) = 0.05 × (-0.60) = -0.030, identical to the 4-token value, because the normalized reward is an average and averages of a constant-quality stream do not depend on how many terms you sum. The raw DPO reward, by contrast, r(A at 2 tokens) = 0.1 × ((2×-0.30) - (2×-0.45)) = 0.1 × (-0.60 - (-0.90)) = 0.1 × 0.30 = 0.030, exactly half the original r(A) = 0.060, because it scales linearly with token count. This is the same length sensitivity that made r(B) double r(A) earlier, now shown shrinking A instead of growing B, confirming it is a general property of the raw reward's definition, not an artifact of that one example.

5. IPO's loss is a squared distance to a fixed target τ, so moving Δ_θ past τ in either direction increases the loss; the parabola has a genuine minimum. DPO's loss is a monotonically decreasing function of Δ_θ with an infimum of zero that is never attained, so gradient descent always has a direction that lowers the loss further: push Δ_θ higher. When preference labels are close to deterministic (little annotator disagreement), this predicts the DPO-trained policy keeps growing more confident and drifting further from π_ref than the KL penalty was meant to allow, since nothing in the loss shape itself ever says "stop here."

6. PPO: 4 × 26 GB = 104 GB in weights (policy, reference, reward model, critic). DPO with full fine-tuning: 2 × 26 GB = 52 GB (policy, reference). LoRA removes the reference model's memory cost almost entirely: with the policy expressed as a frozen 13B base plus a small adapter, the reference model's log-probs come from the same resident base weights with the adapter switched off, so only one 26 GB base copy plus a sub-gigabyte adapter is needed in total.

Think About It

Think about this: How would you explain direct preference optimization: learning from preferences to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.