RLHF: Complete Pipeline from Human Feedback to Alignment

Karya, a Bangalore-founded data-annotation organization, employs rural Indian gig workers to do a very specific job for frontier AI labs: look at two model responses to the same prompt and click "this one is better." Multiply that click by tens of thousands of prompts, by several responses per prompt, by several annotators per comparison, and you get the raw material that reward models are built from. The interesting engineering problem is not the click itself, it is everything downstream of it. One annotator's click is noisy, cheap, and sometimes wrong. A trained, calibrated reward signal that a policy can be safely optimized against for thousands of gradient steps is expensive, fragile, and easy to break. This chapter is about the machinery that turns the first thing into the second, and about exactly how that machinery breaks when pushed too far.

Most treatments of RLHF jump straight from "humans have preferences" to the PPO objective and the KL penalty. That skips the part of the pipeline that actually costs the most money and causes the most production incidents: how raw pairwise clicks get aggregated into a single number per response, how many GPUs it takes to hold every model RLHF needs open at once, and how a reward model that looks like it is improving can be quietly making the policy worse. Those three problems, comparison aggregation, memory footprint, and reward overoptimization, are the focus here.

Why RLHF needs a pipeline, not just an algorithm

A single scalar reward per response would be easy to optimize with plain policy gradient. Humans cannot reliably produce that scalar directly: ask an annotator to score a response 1 to 10 and two annotators will disagree by several points, and even the same annotator will disagree with themselves an hour later. Ranking is far more stable: given two responses side by side, humans agree on which is better far more consistently than they agree on absolute scores. This single empirical fact, first exploited at scale by Christiano, Leike, Brown, Martic, Legg, and Amodei in Deep Reinforcement Learning from Human Preferences (NeurIPS 2017), is why every production RLHF pipeline collects comparisons, not scores, and reconstructs a scalar reward from them afterward. The reconstruction step is the Bradley-Terry model, and it is where a genuine amount of the pipeline's engineering effort lives.

The full pipeline, end to end, has nine moving stages: sourcing prompts, sampling multiple responses from the current policy, collecting a ranking from human annotators, decomposing that ranking into pairwise comparisons, fitting a Bradley-Terry model to those comparisons, training a reward model to generalize the fitted scores to unseen text, running PPO against that reward model with a KL constraint back to a frozen reference policy, deploying the result, and sampling fresh outputs from the new policy to restart the loop. The diagram below lays out all nine stages together with the frozen reference model that anchors the KL term.

Turning ranked lists into a scalar score

Suppose an annotator ranks three responses A, B, C to one prompt: A best, then B, then C. A single K-wise ranking with K = 3 decomposes into C(3,2) = 3 pairwise judgments (A beats B, A beats C, B beats C). With K = 4 that becomes C(4,2) = 6 pairs from a single ranking task; production pipelines typically use K between 4 and 9 for exactly this reason, one ranking task yields many training comparisons. Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin, Zhang, Agarwal, Slama, Ray, Schulman, Hilton, Kelton, Miller, Simens, Askell, Welinder, Christiano, Leike, and Lowe describe this exact design in Training language models to follow instructions with human feedback (Ouyang et al., 2022), where InstructGPT's reward model was trained on comparisons collected from about 40 contractors, using roughly 13,000 prompts for the supervised fine-tuning stage and roughly 33,000 prompts for the comparison-collection stage that fed the reward model.

The pairs from one ranking task are not independent draws, they all came from the same annotator looking at the same K responses once. Treating all C(K,2) of them as separate, shuffled, independently-epoched training examples lets the reward model overfit to that one annotator's idiosyncrasies faster than it learns anything general. The InstructGPT paper's fix is structural rather than statistical: all C(K,2) comparisons from a single ranking are placed in one training batch as a single gradient step, so the model sees the whole ranking's worth of signal at once rather than as scattered, repeatedly-visited fragments across an epoch. Aggregating labeler-labeler agreement rates confirms why this care is necessary at all: independent annotators looking at the same pair frequently disagree, so no individual comparison should be trusted as ground truth. The number the pipeline actually wants is not any single click but a latent per-response score that best explains the whole pattern of clicks across annotators and pairs. That is exactly what the Bradley-Terry model computes.

The Bradley-Terry model, worked by hand

Bradley and Terry's 1952 paper models the probability that item i beats item j in a pairwise comparison as a function of two latent strength scores:

P(i beats j) = σ(s_i − s_j) = exp(s_i) / (exp(s_i) + exp(s_j))

where σ is the logistic sigmoid. This is precisely the model reward models are trained under (the reward model's scalar output plays the role of s), so fitting Bradley-Terry to raw comparison counts is a smaller, transparent version of the same estimation problem the reward model performs at scale. Because only differences s_i − s_j enter the probability, the scale has one free additive degree of freedom; every implementation fixes one item's score, here s_C = 0, as an anchor.

Take three responses to one prompt, with comparisons pooled across five annotators, each of whom judged all three pairs (a complete design; Bradley-Terry handles an incomplete one just as well, when some annotators only see a subset of the pairs):

A beats B: 4   B beats A: 1
A beats C: 5   C beats A: 0
B beats C: 3   C beats B: 2

The maximum-likelihood scores are found by maximizing the log-likelihood of these observed counts under the model, with s_C fixed at 0:

import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def loglik(sA, sB):
    sC = 0.0
    return (4*math.log(sigmoid(sA-sB)) + 1*math.log(sigmoid(sB-sA))
          + 5*math.log(sigmoid(sA-sC)) + 0*math.log(sigmoid(sC-sA))
          + 3*math.log(sigmoid(sB-sC)) + 2*math.log(sigmoid(sC-sB)))

sA, sB, lr = 0.0, 0.0, 0.05
for _ in range(50000):
    gA = (loglik(sA+1e-6, sB) - loglik(sA-1e-6, sB)) / 2e-6
    gB = (loglik(sA, sB+1e-6) - loglik(sA, sB-1e-6)) / 2e-6
    sA += lr * gA
    sB += lr * gB

Running this gradient ascent to convergence gives s_A ≈ 2.599, s_B ≈ 0.705, s_C = 0 (fixed). Converting back to win probabilities with the fitted scores: P(A beats B) = σ(2.599 − 0.705) ≈ 0.869, P(A beats C) = σ(2.599) ≈ 0.931, P(B beats C) = σ(0.705) ≈ 0.669. Compare these to the raw empirical frequencies: A beat C in 5 of 5 direct comparisons (100%), yet the fitted model assigns only 93.1%. This is not an error, it is the point of using a latent-score model instead of raw counts. A's undefeated record against C is partly explained by A's high score alone; the model refuses to treat a small, undefeated sample as certainty and instead produces the estimate consistent with the whole comparison graph, including the indirect evidence that B (who lost to A 4 times but is not hopeless against C) beats C more often than a purely dominant A would predict. Every one of these three win probabilities is generated from just two numbers, s_A and s_B, which is exactly the compression a reward model performs across millions of comparisons: it does not memorize pairwise outcomes, it learns a scalar function whose sigmoid differences reproduce the observed comparison pattern.

The GPU memory a PPO round actually needs

RLHF's PPO stage is expensive for a reason that has nothing to do with the RL math: it needs four separate model instances resident at once, and two of them are being trained with Adam. Under standard mixed-precision training, each trainable parameter costs 16 bytes: 2 bytes for the bf16 weight, 2 for the bf16 gradient, 4 for the fp32 master weight, and 4 each for Adam's two moment estimates (m and v). A frozen parameter used only for a forward pass costs just 2 bytes (its bf16 weight; no gradient, no optimizer state, no fp32 master copy is needed).

Summing the ledger for a 7-billion-parameter model at every stage: policy 16 × 7,000,000,000 = 112 GB, value model another 112 GB, reference policy 2 × 7,000,000,000 = 14 GB, reward model another 14 GB. Total: 112 + 112 + 14 + 14 = 252 GB, and that is before counting activations from the forward and backward passes or the KV cache the policy needs while generating rollouts during sampling. A single H100 GPU has 80 GB of high-bandwidth memory, so this configuration needs at least four such GPUs purely to hold model state, before any of the actual training work happens. This is precisely why production RLHF frameworks (DeepSpeed-Chat, OpenRLHF, TRL with DeepSpeed ZeRO) shard the reference and reward models onto separate devices from the policy and value model, offload optimizer state to CPU memory, or replace one of the four models entirely (some pipelines reuse the reward model's backbone to initialize the value model and share weights during rollout generation to cut this footprint). The reward model derivation itself may be the same across labs; the memory engineering to run it at 7B, 70B, or larger scale is where most of the systems difficulty actually sits.

When the reward model lies: overoptimization

Common misconception: "the reward model was fit to real human comparisons, so pushing PPO to maximize it as far as possible should always produce a policy humans prefer more." This is false, and the failure mode has a name older than RLHF itself: Goodhart's law, once a measure becomes a target, it stops being a good measure. The reward model rφ is a noisy, finite-sample approximation of what humans actually want; PPO is a powerful enough optimizer to find policy outputs that score high on rφ's specific quirks (repeated reassurance phrases, excessive hedging, particular formatting the labelers happened to reward) without those outputs being genuinely better. Gao, Schulman, and Hilton's Scaling Laws for Reward Model Overoptimization (2022) measured this directly by training a large "gold" reward model to stand in for true human judgment and tracking it as a smaller proxy reward model was optimized against: the proxy reward climbs monotonically throughout training, but the gold reward rises, peaks, and then declines as the policy drifts further from the reference distribution the reward model was actually trained on.

The KL penalty term in the PPO objective, rφ(x,y) − β·KL(π ‖ π_ref), is not a regularizer added for training stability as an afterthought, it is the direct mechanical defense against this exact failure. KL divergence from the reference policy is a proxy for "how far has the policy wandered from the distribution the reward model was actually validated on," and Gao et al.'s finding is that gold reward is well predicted as a function of that KL distance specifically, not of training steps or proxy reward directly. This gives a concrete, checkable production signal: if the proxy reward model's score keeps climbing while the KL-from-reference keeps growing past the range the reward model was trained and validated on, the team should not read the climbing proxy score as good news. Raising β, stopping earlier, retraining the reward model on comparisons collected against the current policy (closing the distribution gap directly), or ensembling multiple reward models to reduce exploitable idiosyncrasy are the actual interventions, not simply running PPO longer.

Closing the loop

The reason InstructGPT-style pipelines are not a one-shot process is exactly the overoptimization problem above: a reward model trained on comparisons of the original SFT policy's outputs is validated on that policy's output distribution, and it becomes progressively less trustworthy as PPO pushes the policy away from that distribution. The fix built into the pipeline is the feedback arrow in the diagram: after deploying π_new, fresh outputs are sampled from it and sent back to annotators for a new round of K-wise ranking, producing new comparisons that are representative of where the policy currently is. The reward model for round r+1 is retrained (or fine-tuned) on this fresh data, closing the distribution gap that caused round r's overoptimization in the first place. This is why production RLHF is described as an iterated process across several policy versions rather than a single training run, each round buys a bounded amount of safe optimization before the reward model's blind spots need to be refreshed with real human judgment again.

Active recall

Attempt each question before reading its answer.

1. In the worked Bradley-Terry example, A beat C in all 5 direct comparisons, yet the fitted model gives P(A beats C) = 0.931, not 1.0. Why doesn't the model just report the empirical 100% rate?
2. If K (responses ranked per prompt) increases from 4 to 6, trace the full ripple: what happens to (a) pairwise comparisons harvested per ranking task, (b) reward model training-set size for a fixed prompt budget, (c) the correlation structure among those comparisons, and (d) what the pipeline does to compensate?
3. Using the memory ledger method (16 bytes/trainable parameter, 2 bytes/frozen parameter), compute the total GPU memory PPO needs if the policy and value model are 13B parameters each, but a team deliberately uses a smaller 7B reward model and 7B reference policy to save memory.
4. A team sees their proxy reward model's score rise steadily through PPO training while their human-rated quality plateaus around step 8,000 and then declines. Name the phenomenon, the training-time metric most directly implicated, and the intervention that targets it directly (not just "train less").
5. Why does the InstructGPT pipeline put all C(K,2) comparisons from one ranking into a single training batch instead of shuffling them independently across an epoch?
6. Explain in one or two sentences why RLHF is run as several iterated rounds (collect comparisons, train, deploy, collect again) rather than one long PPO run against a single fixed reward model.

Answers

1. Because Bradley-Terry does not fit each pair in isolation, it fits two scalar scores that must simultaneously explain every comparison in the graph, including B's record against both A and C. A's undefeated 5-of-5 sample against C is real evidence but a small one; the shared latent scale pulls the estimate toward what is jointly consistent with A's edge over B (4-1, not undefeated) and B's edge over C (3-2, not undefeated). The result, 0.931, is the model's best compromise between "A is clearly strong" and "the sample is too small to justify certainty," which is exactly the smoothing behavior that makes reward models resistant to a handful of noisy labels dominating the estimate.

2. (a) C(6,2) = 15 pairwise comparisons per ranking task, versus C(4,2) = 6 at K = 4, a 2.5× increase in comparisons per task. (b) For a fixed number of labeled prompts, the reward model's raw training-set size (in comparison pairs) grows by the same 2.5× factor, since each ranking task now yields more pairs, without requiring 2.5× more human labeling time (ranking 6 items takes longer than ranking 4, but not linearly so). (c) All 15 pairs from one ranking still come from a single annotator's single judgment, so the correlation among them is even higher than at K = 4, the risk of the reward model overfitting to one annotator's idiosyncratic ordering grows, not shrinks, with K. (d) The pipeline compensates the same way regardless of K: all C(K,2) pairs from one ranking are kept together in a single training batch (one gradient step sees the whole ranking at once) rather than being treated as independent, separately-epoched examples, which is the direct fix for the correlation problem identified in (c).

3. Policy: 16 × 13,000,000,000 = 208 GB. Value model: 16 × 13,000,000,000 = 208 GB. Reward model (frozen): 2 × 7,000,000,000 = 14 GB. Reference policy (frozen): 2 × 7,000,000,000 = 14 GB. Total: 208 + 208 + 14 + 14 = 444 GB, more than the 252 GB baseline case despite using a smaller reward model and reference policy, because the two trainable models dominate the ledger at 16 bytes/parameter each; shrinking the two frozen, forward-only models saves comparatively little (28 GB total either way at 7B) next to what shrinking the trainable policy or value model would save.

4. This is reward overoptimization (Goodhart's law applied to the reward model), as characterized empirically in Gao, Schulman, and Hilton's 2022 scaling-law study. The metric most directly implicated is the KL divergence between the current policy and the frozen reference policy, gold/human-judged quality is well predicted as a function of that KL distance, and it degrades once KL grows past the range the reward model was actually trained and validated on. The direct intervention is to increase β (the KL penalty coefficient) so the policy is constrained closer to the reference distribution, or to retrain the reward model on comparisons collected against the current policy to close the distribution gap, not simply to stop training earlier and hope for the best.

5. Because the pairs are not independent evidence, they are C(K,2) restatements of one annotator's single ranking decision on one occasion. Shuffling them independently across an epoch lets the model pass over the same underlying judgment many times as if it were many separate opinions, which accelerates overfitting to that one annotator's biases. Keeping them in one batch means a single gradient update absorbs the whole ranking's information content once, matching the actual amount of independent evidence collected.

6. A reward model trained on comparisons of one policy's outputs is only validated on that policy's output distribution; as PPO pushes the policy away from that distribution (measured by rising KL), the reward model's scores become progressively less trustworthy, which is the overoptimization failure from question 4. Running iterated rounds, deploy, sample fresh outputs, collect fresh comparisons, retrain the reward model, resume PPO, repeatedly closes that distribution gap so each round only has to extrapolate a small, bounded distance from data the reward model actually saw.

Think About It

Think about this: How would you explain rlhf: complete pipeline from human feedback to alignment to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where rlhf: complete pipeline from human feedback to alignment is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting rlhf: complete pipeline from human feedback to alignment to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind rlhf: complete pipeline from human feedback to alignment, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.