Reward Model Training: Learning Preference Prediction

A fintech support assistant handling UPI complaints gets asked, hundreds of times a day, some version of: "My payment failed but the amount was deducted from my account. What do I do?" Two candidate replies come out of the language model for the same query:

Response A: "This usually happens when the transaction times out at the bank's end after the amount has already been debited. Under NPCI rules, if the receiver hasn't been credited, the amount is auto-reversed to your account within T+1 working day. If it isn't reversed by then, raise a complaint on the UPI app's 'Raise Dispute' option or call your bank's grievance cell, quoting the UPI transaction reference number."

Response B: "Sorry to hear that. Please try the payment again after some time, or contact your bank for assistance."

Every engineer building this system agrees A is better. But "better" is not a loss function. You cannot backpropagate through a human's sense of helpfulness directly — there is no ground-truth label file that says response A deserves a score of 8.7 and response B deserves 3.1. This chapter is about the piece of machinery that closes that gap: a reward model, a neural network trained not to generate text but to output a single number that predicts which of two responses a human would prefer. It is the component that sits between "a language model that can talk" and "a language model that has been tuned toward what people actually want," and it is worth understanding on its own terms before you ever get to the reinforcement-learning loop (PPO) that consumes its output.

1. Why comparisons, not scores

The naive approach is to ask a human annotator to rate each response on a scale, say 1 to 10, and train a network to regress onto that number directly. This fails in practice for a measurable reason: absolute quality judgments are poorly calibrated. The same annotator rates a mediocre response a 6 in the morning and a 4 for an equally mediocre response after lunch, because the reference point in their head drifts. Different annotators disagree even more sharply — one person's 7 is another's 5. Psychometric studies on human judgment consistently find that people are far more reliable at relative comparisons ("which of these two is better?") than at producing stable absolute scores. This is precisely the reasoning behind the Bradley-Terry model, published in 1952 for ranking objects from pairwise comparisons, and it is exactly the model reused inside every modern RLHF pipeline, including the one described in OpenAI's InstructGPT paper (Ouyang et al., 2022), which is the direct ancestor of ChatGPT's alignment procedure.

You have already seen a working version of this idea if you follow cricket. The ICC does not ask a panel to assign each team an absolute "skill score" out of 100 and average their opinions. Instead, team ratings are built from the accumulated results of head-to-head matches: beat a higher-rated team, your rating rises more than if you'd beaten a weaker one. The rating is inferred entirely from wins and losses between pairs, and it is precisely this kind of pairwise-comparison model, mathematically, that a reward model learns from human preference data. The object being "ranked" is not a cricket team — it is a (prompt, response) pair — but the underlying mathematics is identical.

2. The Bradley-Terry model, derived

Suppose two items, i and j, have latent "strength" values s_i and s_j (a cricket team's rating, or a response's quality). The Bradley-Terry model assumes the probability that i "beats" j in a head-to-head comparison is proportional to i's strength relative to the combined strength of both:

P(i beats j) = e^(s_i) / (e^(s_i) + e^(s_j))

Divide numerator and denominator by e^(s_i):

P(i beats j) = 1 / (1 + e^(s_j - s_i)) = 1 / (1 + e^(-(s_i - s_j))) = σ(s_i - s_j)

where σ is the logistic sigmoid, σ(z) = 1/(1+e^(-z)). This is the same functional form used by the Elo rating system in chess and, with a base-10 scaling constant instead of base e, by the ICC's own team-rating methodology. The result only ever depends on the difference in strengths, never on their absolute values — a fact that matters a great deal in section 6.

For a reward model, replace "strength" with the scalar output of a neural network. The network is almost always the same transformer that was already pretrained and instruction-tuned as the base language model, but with its final layer swapped out: instead of an unembedding matrix that produces a probability distribution over the next token, it has a single linear "reward head" bolted onto the final hidden state, producing one real number per (prompt, response) pair. Call this function r_θ(x, y), where θ are the network's weights, x is the prompt, and y is a candidate response. Given a human-labeled pair where y_w ("winner") was preferred over y_l ("loser"), Bradley-Terry gives:

P(y_w preferred over y_l) = σ(r_θ(x, y_w) − r_θ(x, y_l))

3. The training objective

Reward-model training data is a set of triples (x, y_w, y_l): a prompt and two responses to it, with a human annotator's judgment of which one is better already resolved into an ordering. The training loss is the negative log-likelihood of the human's actual choice under the Bradley-Terry model above:

L(θ) = − E_(x,y_w,y_l)~D [ log σ( r_θ(x, y_w) − r_θ(x, y_l) ) ]

This is ordinary binary cross-entropy in disguise: the "label" is always 1 (y_w wins), and the "predicted probability" is σ(r_θ(x,y_w) − r_θ(x,y_l)). Minimizing it does exactly one thing — it pushes r_θ(x,y_w) up and r_θ(x,y_l) down whenever the current margin between them doesn't yet make the model confident that y_w wins. Once the reward model is trained on a large corpus of such comparisons (tens of thousands to millions of pairs, in production systems), it is frozen and reused as the reward signal that drives the later PPO fine-tuning stage — but the model itself never generates text; it only scores it.

4. Worked example: one gradient step by hand

A real reward model's final hidden state has hundreds or thousands of dimensions, which makes hand-tracing impossible. To see the actual mechanism without a computer, shrink it: pretend the transformer backbone has already reduced each response to a 2-dimensional feature vector φ(x,y) — stand-ins for whatever the real hidden state would encode, here loosely "does the response name a concrete resolution mechanism" (feature 1) and "how many distinct actionable steps it gives" (feature 2). The reward head is then just a linear layer: r(y) = w·φ(y) + b.

Take the UPI example from the opening. Response A (the human-chosen winner, y_w) and response B (the rejected one, y_l) get feature vectors:

φ(y_w) = [1, 3]   # names NPCI T+1 auto-reversal + dispute route; 3 actionable steps
φ(y_l) = [1, 0]   # vaguely acknowledges the issue; 0 actionable steps

Initialize the reward head with weights w = [1.0, 0.5] and bias b = 0. Forward pass:

r(y_w) = 1.0×1 + 0.5×3 + 0 = 1.0 + 1.5 = 2.5
r(y_l) = 1.0×1 + 0.5×0 + 0 = 1.0
d = r(y_w) − r(y_l) = 2.5 − 1.0 = 1.5
σ(d) = 1 / (1 + e^(−1.5)) = 1 / 1.22313 = 0.81757
L = −log σ(d) = log(1 + e^(−1.5)) = 0.20141

The model already assigns 81.8% probability to the correct ordering, so the loss is positive but small. Now find the gradient with respect to w. Since L = −log σ(d) and d = w·(φ(y_w) − φ(y_l)) (the bias term cancels because it's identical for both responses), the chain rule gives:

∂L/∂d  = −(1 − σ(d))              [standard derivative of −log σ]
∂L/∂w  = ∂L/∂d × (φ(y_w) − φ(y_l)) = −(1 − σ(d)) × (φ(y_w) − φ(y_l))

With φ(y_w) − φ(y_l) = [1−1, 3−0] = [0, 3] and 1 − σ(d) = 0.18243:

∂L/∂w = −0.18243 × [0, 3] = [0, −0.54728]

One gradient-descent step with learning rate η = 0.1:

w_new = w − η × ∂L/∂w = [1.0, 0.5] − 0.1×[0, −0.54728] = [1.0, 0.55473]

Only the second weight moved — the one attached to "number of actionable steps," the feature that actually differed between the two responses. The first weight's gradient was exactly zero because both responses had the same value (1) on that feature, so it carried no information about which one was better. Recomputing the forward pass with the updated weights:

r(y_w)_new = 1.0×1 + 0.55473×3 = 2.66418
r(y_l)_new = 1.0×1 + 0.55473×0 = 1.0
d_new = 1.66418
σ(d_new) = 0.84080
L_new = 0.17340

The loss dropped from 0.20141 to 0.17340 in a single step, and the margin between the two rewards widened from 1.5 to 1.664 — the network became more confident that the response with concrete, actionable content should score higher. This is the entire training loop, repeated over millions of comparison pairs with a real transformer backbone instead of a 2-dimensional stand-in: forward pass both responses, compute the Bradley-Terry loss on the margin, backpropagate, update.

This is verifiable directly in code — the block below reproduces every number above:

import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

def reward(phi, w, b):
    return sum(wi * pi for wi, pi in zip(w, phi)) + b

phi_w = [1, 3]   # winner: NPCI reversal + dispute steps
phi_l = [1, 0]   # loser: vague, no actionable steps
w = [1.0, 0.5]
b = 0.0

r_w = reward(phi_w, w, b)          # 2.5
r_l = reward(phi_l, w, b)          # 1.0
d = r_w - r_l                      # 1.5
p = sigmoid(d)                     # 0.81757...
loss = -math.log(p)                # 0.20141...

one_minus_p = 1 - p                # 0.18243...
diff_phi = [a - b_ for a, b_ in zip(phi_w, phi_l)]   # [0, 3]
grad_w = [-one_minus_p * x for x in diff_phi]        # [0.0, -0.54728...]

eta = 0.1
w_new = [wi - eta * gi for wi, gi in zip(w, grad_w)] # [1.0, 0.55473...]

r_w_new = reward(phi_w, w_new, b)  # 2.66418...
r_l_new = reward(phi_l, w_new, b)  # 1.0
loss_new = -math.log(sigmoid(r_w_new - r_l_new))     # 0.17340...

print(round(loss, 5), round(loss_new, 5))  # 0.20141 0.1734

5. The pipeline end to end

Once training converges over a large comparison dataset, the reward model θ is frozen. It stops learning and instead becomes a scoring function that a later stage — typically PPO — uses to grade responses the policy generates during reinforcement-learning fine-tuning, or to rank multiple sampled responses at inference time (best-of-n sampling). That later stage is a separate chapter's worth of machinery; the reward model's whole job ends the moment it can reliably predict which of two responses a human would have picked.

6. The misconception worth killing: "the reward is a quality score"

It's tempting to read r_θ(x,y) = 2.5 the way you'd read a mark out of ten: as an absolute, portable measure of how good response y is. This is wrong, and the Bradley-Terry derivation in section 2 shows exactly why. The loss function only ever depends on the difference r(y_w) − r(y_l), never on either value alone. Add any constant c to both rewards for a given prompt:

d' = (r(y_w) + c) − (r(y_l) + c) = r(y_w) − r(y_l) = d

The difference — and therefore σ(d), and therefore the loss — is completely unchanged. The gradient of the loss with respect to c is exactly zero: nothing in training ever pushes the reward model toward a particular zero-point or scale. Two reward models trained on the exact same comparison data, differing only in random initialization, can and typically do settle on entirely different absolute reward ranges — one might output values clustered around 0, another around −40 — while making identical preference predictions. This is precisely why raw Elo ratings or ICC ratings from different eras or player pools cannot be compared directly without anchoring: only differences within a shared reference frame carry meaning. The practical consequence for a reward model is sharper still: r_θ(x1, y) and r_θ(x2, y') for two different prompts x1, x2 are not comparable at all, even in principle, because they were never trained against each other in a single loss term. Only comparisons within the same prompt, between responses that were actually part of a training pair (or structurally equivalent to one), have any calibrated meaning. Treating the raw scalar as an interpretable "quality score" — averaging it across prompts, setting a global pass/fail threshold on it — is a category error the model's own training objective explicitly permits.

Active recall

Attempt these before reading the answers below.

1. Why do RLHF pipelines collect pairwise preference comparisons instead of asking annotators for an absolute 1-to-10 quality score?
2. Starting from P(i beats j) = e^(s_i) / (e^(s_i) + e^(s_j)), show algebraically that this equals σ(s_i − s_j).
3. A reward model gives r(y1) = 3.2 and r(y2) = 1.0 for two responses to the same prompt, and the human annotator preferred y1. Compute the model's implied probability that y1 is preferred, and the training loss for this one pair.
4. Take the pair in Q3. If you add 5.0 to both r(y1) and r(y2), what happens to the loss? Justify from the loss formula, not just by recomputing.
5. In the worked gradient step of section 4, w1 (the weight on "names a resolution mechanism") did not change at all, while w2 did. Why did the gradient assign zero update to w1?
6. What is the functional difference between the reward-model training stage described in this chapter and the PPO fine-tuning stage that typically follows it?

Worked answers

1. Absolute ratings are poorly calibrated: the same annotator's internal reference point drifts across a session, and different annotators anchor differently, so a "7" from one rater and a "7" from another aren't the same judgment. Pairwise comparisons ("which of these two is better") are a much easier and more consistent task for humans to perform reliably, which is why Bradley-Terry-style pairwise modeling — the same approach behind Elo and ICC cricket ratings — is used instead of direct regression onto a score.

2. Divide numerator and denominator of e^(s_i)/(e^(s_i)+e^(s_j)) by e^(s_i): this gives 1/(1 + e^(s_j)/e^(s_i)) = 1/(1 + e^(s_j − s_i)) = 1/(1 + e^(−(s_i − s_j))), which is exactly the definition of σ(s_i − s_j).

3. d = 3.2 − 1.0 = 2.2. σ(2.2) = 1/(1+e^(−2.2)) = 0.90025. Since the human did prefer y1 (the higher-scored response), the loss is −log(0.90025) = 0.10508. (Both values match a direct calculation: e^(−2.2) ≈ 0.11080, so 1/1.11080 ≈ 0.90025, and −ln(0.90025) ≈ 0.10508.)

4. Nothing changes. The loss depends only on d = r(y1) − r(y2), and adding the same constant to both rewards leaves d unchanged: (3.2+5) − (1.0+5) = 2.2, identical to before. So σ(d) and the loss stay exactly 0.90025 and 0.10508. This is the shift-invariance property from section 6 — the reward model's absolute scale is never constrained by training.

5. The gradient on each weight is −(1−σ(d)) × (φ(y_w) − φ(y_l)) component-wise. w1 is attached to the feature "names a resolution mechanism," and both y_w and y_l had value 1 on that feature (φ(y_w)[0]=1, φ(y_l)[0]=1), so the difference for that coordinate is 1−1=0. A feature that doesn't distinguish the winner from the loser in this particular pair carries no gradient signal for this pair — the update only touches weights on features where the two responses actually differed, here "number of actionable steps" (3 vs. 0).

6. Reward-model training (this chapter) is supervised learning on a fixed, static dataset of human-labeled comparison pairs, with a classification-style loss (Bradley-Terry negative log-likelihood) and no interaction with the policy model's own generation process. PPO fine-tuning is reinforcement learning: the policy model actively generates new responses, the frozen reward model scores them, and the policy's weights are updated to increase expected reward — a fundamentally different, on-policy training loop that consumes the reward model's output rather than training the reward model itself.

Think About It

Think about this: How would you explain reward model training: learning preference prediction to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind reward model training: learning preference prediction, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.