Suppose an Indian AI lab is fine-tuning a multilingual assistant that has to be safe in Hindi, Tamil, Bengali, and English simultaneously — the kind of coverage a consumer product aimed at India's language diversity actually needs. Standard RLHF (Reinforcement Learning from Human Feedback, Ouyang et al., 2022, the method behind InstructGPT) would require thousands of human annotators reading pairs of model responses in each language and voting on which is safer, day after day, as the model keeps changing under them. That pipeline is slow, expensive to scale across languages, and the annotators themselves need training to apply a consistent safety standard — the standard drifts language to language, annotator to annotator. Constitutional AI's second phase, RL-CAI, replaces exactly that one bottleneck: instead of paying humans to compare responses, an AI model compares them, guided by a written list of principles that can be translated once and applied everywhere. This chapter goes into that replacement mechanism in the depth a working ML engineer needs — the preference-model math, the RL objective it feeds, and the systems cost of running it — rather than the supervised critique-and-revise loop that introduces Constitutional AI elsewhere in this curriculum.
Where RL-CAI sits in the pipeline
Bai, Kadavath, Kundu, Askell, Kernion, Jones, Chen, et al. (Anthropic, 2022), "Constitutional AI: Harmlessness from AI Feedback" (arXiv:2212.08073), describes two training phases. The first, supervised learning from Constitutional AI (SL-CAI), has the model critique and revise its own outputs against a principle, then fine-tunes on the revised text — that self-critique loop is not this chapter's focus. The second phase, RL-CAI, is the one that actually replaces human preference labels with AI-generated ones, and it reuses the exact same reinforcement-learning machinery as RLHF: a preference model trained on comparison data, then policy optimization against that preference model with a KL penalty holding the policy near a reference checkpoint. The only thing that changes between RLHF and RL-CAI is who produces the comparison labels — a human rater in RLHF, another copy of the language model (prompted with a randomly drawn constitutional principle) in RL-CAI. Everything downstream of that label is identical math.
The RL-CAI pipeline, step by step
Starting from the SL-CAI checkpoint π₀ (held frozen as a reference), each training round does the following:
1. Sample two responses, y_A and y_B, from π₀ for the same prompt x.
2. Draw one principle p_i uniformly from the constitution — the original paper used a list of roughly sixteen principles, things like preferring the response that is less likely to encourage illegal or dangerous behaviour.
3. Feed x, y_A, y_B, and p_i to a separate feedback model as a multiple-choice question ("which response better satisfies this principle: (A) or (B)?"), using a chain-of-thought prompt that asks the model to reason before answering.
4. Read off the model's log-probabilities for the tokens "(A)" and "(B)" as its two possible completions, and convert them to a soft label — a probability distribution over which response is preferred — rather than forcing a hard 0/1 vote.
5. Repeat across many prompts and principles to build a comparison dataset, then train a preference model r_φ on it.
6. Use r_φ as the reward signal in a PPO fine-tuning run on the policy, with a KL-divergence penalty against π₀ so the policy cannot drift arbitrarily far in the process of maximizing reward.
Step 4 is the crux of "AI feedback" as a term: the feedback model is not asked to produce a single label. It produces a distribution, and that distribution is what gets trained against — a detail with real consequences, shown in Worked Example 1 below.
Worked example 1: training the preference model on a soft AI label
The feedback model doesn't just say "A wins." It assigns log-probabilities to the tokens "(A)" and "(B)" as its next-token completion, and those get turned into a probability via softmax. Say the feedback model, reasoning about a candidate pair under a given principle, assigns logit 2.3 to "(A)" and 1.1 to "(B)". The preference model currently in training assigns scalar scores r_A = 0.5 and r_B = −0.2 to the same pair. Here is the full computation, traced in code so every number is checkable:
import math
# AI feedback model's raw logits for completing "(A)" vs "(B)"
logit_A, logit_B = 2.3, 1.1
p_A = math.exp(logit_A) / (math.exp(logit_A) + math.exp(logit_B))
p_B = 1 - p_A
# Preference model's current scalar scores for the two responses
r_A, r_B = 0.5, -0.2
diff = r_A - r_B
pred = 1 / (1 + math.exp(-diff)) # PM's own P(A preferred), Bradley-Terry form
# Cross-entropy between the AI's soft label and the PM's prediction
loss = -(p_A * math.log(pred) + p_B * math.log(1 - pred))
print(round(p_A, 4), round(p_B, 4)) # 0.7685 0.2315
print(round(pred, 4)) # 0.6682
print(round(loss, 4)) # 0.5652
The AI feedback model is 76.9% confident A is better; the preference model, at its current weights, only predicts 66.8%. Because 0.7685 > 0.6682, the loss's gradient pushes r_A up and r_B down — the preference model has not yet caught up to the feedback model's confidence, so training nudges it in that direction. This is the Bradley–Terry model (the same pairwise-comparison model behind chess Elo ratings), fit with cross-entropy against a soft target instead of a hard one. Using the soft label rather than rounding it to a hard "A wins" is not a minor detail: it is what lets the preference model learn the feedback model's uncertainty rather than being forced to pretend every comparison was a landslide. The 2022 paper's ablations found that stripping the chain-of-thought reasoning out of the feedback prompt made these confidences swing toward the extremes (near 0 or 1) even on genuinely close calls — the reasoning step is what keeps the soft label calibrated instead of overconfident.
Worked example 2: the KL-regularized reward, traced in code
Once r_φ is trained, it becomes the reward signal for PPO — but not the only term. If the policy is optimized purely to maximize r_φ, it will find whatever quirk in r_φ scores highest and collapse onto it (reward hacking), producing text that scores well on the preference model but has drifted away from coherent, on-distribution language. The fix, following InstructGPT's objective exactly, is to subtract a KL-divergence penalty between the policy π being trained and the frozen reference π₀, scaled by a small coefficient β. In practice this KL term is estimated per token as the log-probability ratio between the two policies on the tokens actually sampled — an unbiased single-sample estimate of KL divergence, not the full expectation, because computing the true KL over the entire vocabulary at every step would be far more expensive.
Suppose the policy generates a three-token completion, and we have the per-token log-probabilities under both the policy and the frozen reference:
import math
# Per-token log-probs the current policy and the frozen reference assign
# to the same three sampled tokens
logp_policy = [-0.10, -0.05, -0.20]
logp_reference = [-0.15, -0.08, -0.50]
# Per-token KL estimate: log pi(y_t) - log pi_ref(y_t)
kl_per_token = [round(p - r, 2) for p, r in zip(logp_policy, logp_reference)]
kl_total = sum(kl_per_token)
r_PM = 0.7 # scalar score the trained preference model gives this completion
beta = 0.02 # KL penalty coefficient
reward = r_PM - beta * kl_total
print(kl_per_token) # [0.05, 0.03, 0.3]
print(round(kl_total, 2)) # 0.38
print(round(reward, 4)) # 0.6924
The policy assigned higher probability to every one of these tokens than the reference did (all three per-token differences are positive), especially the third token — a 0.30 nat gap is large, meaning the policy has drifted substantially from π₀ on that token. The total KL estimate of 0.38, scaled by β = 0.02, shaves only 0.0076 off the raw preference-model reward of 0.7, giving a final PPO reward of 0.6924. That small a penalty at this β is deliberate: the original paper used a small β precisely so the preference signal dominates while still keeping runaway drift in check. Raise β and the same drift costs more reward, which is exactly what Active Recall question 4 below asks you to trace.
Common misconception
Students often picture the constitution as something the deployed model consults at inference time — a checklist it runs each response against, like a content filter. It is not. The constitution's principles are used exclusively during training, to generate the comparison labels that shape r_φ and, through PPO, the policy's weights. Once RL-CAI finishes, the deployed model has no access to the constitution text at all; whatever behavior the principles instilled is now baked into the parameters, the same way any other training signal becomes weights rather than a lookup table. This matters practically: you cannot patch a deployed Constitutional-AI model's behavior by editing the constitution file, because the model was never reading that file to begin with. Changing behavior means regenerating comparisons with the revised constitution and re-running the RL-CAI pipeline — a full retraining pass, not a config change.
Beyond the original pipeline: whose constitution, and a cheaper RL-free route
Two extensions are worth knowing at this level. First, a 2024 collaboration between Anthropic and the Collective Intelligence Project ran a "Collective Constitutional AI" study, replacing the researcher-written principle list with one drawn from a public deliberation process (participants ranked candidate principles through the Polis platform), then trained a model against that crowdsourced constitution instead of an internally authored one. The RL-CAI mechanics — soft labels, Bradley–Terry preference model, KL-penalized PPO — are unchanged; what moves is only the source of the principle text sampled in step 2 of the pipeline. This is the natural experiment to run if you want to ask "aligned with whose values?" rather than "aligned, how?"
Second, Rafailov, Sharma, Mitchell, Ermon, Manning, and Finn (2023), "Direct Preference Optimization: Your Language Model Is Secretly a Reward Model," showed the entire reward-model-plus-PPO apparatus can be collapsed into a single classification-style loss computed directly on preference pairs, with no separate reward model and no RL rollout loop at all. Applying DPO to constitutionally-generated comparisons — training straight on the AI feedback model's (A)/(B) soft labels instead of first fitting r_φ and then running PPO — is a live alternative in production alignment pipelines. The systems reason to care: PPO-style RL-CAI needs the policy, the frozen reference (for the KL term), the preference model, and a value network all resident at once. As a back-of-envelope estimate, a 7B-parameter model at bf16 precision is roughly 14 GB of weights; four same-sized models resident simultaneously is on the order of 56 GB before counting any optimizer state or activations — call it a 4x footprint over the policy alone. DPO needs only the policy and the frozen reference, roughly half that footprint, because the reward model and the value network are eliminated by folding the preference comparison directly into the policy's own loss. That difference is why many teams that adopted RLHF-style pipelines in 2022–23 migrated toward DPO-style objectives once the compute cost of full PPO became the limiting factor rather than data.
Active recall
Attempt each question before reading its answer.
1. In the RL-CAI pipeline, why must π₀ stay frozen throughout PPO training rather than being continuously updated to track the current policy?
2. The feedback model assigns logit_A = 1.8 and logit_B = 0.6 to a comparison — note this is the same 1.2 difference as Worked Example 1's 2.3 vs 1.1. Compute p_A two independent ways and confirm they agree.
3. A preference model gives r_A = 1.2, r_B = 0.4 for a pair, and the AI feedback model's soft label is p_A = 0.6. Compute the Bradley–Terry cross-entropy loss.
4. Using Worked Example 2's KL total of 0.38, recompute the PPO reward if β is raised from 0.02 to 0.1. Does this change anything in Worked Example 1? Explain fully, not just the obvious recomputation.
5. Why does moving from PPO-based RL-CAI to a DPO-style objective reduce the number of model copies that must be resident in GPU memory during training, and by roughly how many copies?
6. True or false: a deployed Constitutional-AI model re-reads its constitution before generating each response, and justify your answer.
Worked answers
1. π₀ is the anchor the KL penalty measures distance from. If it were updated to track the current policy, KL(π‖π₀) would stay near zero regardless of how far training actually pushed the policy — the penalty would stop constraining anything, and reward hacking (optimizing r_φ at the expense of coherent, on-distribution language) would go unchecked. Freezing π₀ gives the penalty a fixed reference point, exactly as InstructGPT does.
2. Direct softmax: e^1.8 ≈ 6.0496, e^0.6 ≈ 1.8221, sum ≈ 7.8717, p_A = 6.0496/7.8717 ≈ 0.7685. Shortcut: for a two-way softmax, p_A = σ(logit_A − logit_B) = σ(1.2) = 1/(1+e^−1.2) = 1/1.3012 ≈ 0.7685. Both routes agree because a two-outcome softmax depends only on the logit difference, not on the absolute values — the pair (2.3, 1.1) and the pair (1.8, 0.6) produce identical labels because both have difference 1.2.
3. diff = 1.2 − 0.4 = 0.8. pred = σ(0.8) = 1/(1+e^−0.8) = 1/1.4493 ≈ 0.6900. Loss = −[0.6·ln(0.6900) + 0.4·ln(0.3100)] = −[0.6·(−0.3711) + 0.4·(−1.1713)] = −[−0.2227 − 0.4685] ≈ 0.6912.
4. New reward = 0.7 − 0.1×0.38 = 0.7 − 0.038 = 0.662 (down from 0.6924). Qualitatively, a larger β makes the PPO objective penalize drift from π₀ five times more heavily per unit of KL, which pulls the effective gradient signal further toward "stay close to the reference" and further from "maximize r_φ" — this reduces reward-hacking risk but also slows how much the constitution's preferences can actually reshape the policy. Crucially, this change does not touch Worked Example 1 at all: β is a hyperparameter of the RL objective (reward combination), while the Bradley–Terry loss in Example 1 trains r_φ itself, a completely separate optimization that has already finished by the time PPO runs. Raising β cannot retroactively change what r_φ learned; it only changes how strongly that already-fixed r_φ gets weighted against the KL term downstream. Recognizing that β and the PM's own training loss are independent knobs — not something that ripples backward into Example 1 — is the point of this question.
5. PPO-based RL-CAI needs four resident models: the policy, the frozen reference (for KL), the preference/reward model, and a value network for advantage estimation. DPO eliminates both the separate preference model and the value network by folding the preference comparison directly into a classification loss over the policy and reference log-probabilities alone — leaving two resident models instead of four, roughly halving the weight-memory footprint (before optimizer state and activations, which scale with how many of those copies need gradients).
6. False. The constitution's principles are consumed only during the RL-CAI training loop, to generate the comparisons that train r_φ, which in turn shapes the policy's weights through PPO. Once training is complete, the deployed model has no runtime access to the constitution text — its constitutional behavior is encoded in its parameters, not retrieved from a document at inference time.
Think About It
Think about this: How would you explain constitutional ai: aligning models with principles to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind constitutional ai: aligning models with principles, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.