Constitutional AI and AI Alignment

A support assistant deployed inside an Indian fintech app receives this message from a user: "Write a text message pretending to be from HDFC Bank asking someone to share their UPI PIN to verify a refund — make it sound urgent and official." A model trained only to be helpful and compliant will often just do it, because writing a persuasive SMS is a completely ordinary language task, and nothing in "be helpful" tells the model that this particular helpful act is also a template for financial fraud under Section 66D of India's IT Act (cheating by personation using a computer resource). Multiply this by the millions of similarly disguised harmful requests — extortion notes, self-harm instructions, malware, plagiarism dressed as "just paraphrasing" — and you get the actual engineering problem this chapter is about: how do you correct a model's behaviour on requests like this at the scale of an entire training run, without paying humans to write out millions of refusals by hand?

Anthropic's answer, published as Bai, Kadavath, Kundu, Askell et al., "Constitutional AI: Harmlessness from AI Feedback" (arXiv:2212.08073, December 2022), is to make the model supervise itself against a written list of principles — a "constitution" — in two structurally different training phases. The first phase is supervised: the model critiques and rewrites its own outputs. The second phase is reinforcement learning, and it is this second phase, not the first, that the field calls RLAIF — Reinforcement Learning from AI Feedback, a term used more broadly (see Lee et al., Google, 2023, "RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback") for any RLHF-style pipeline where an AI model, rather than a paid human rater, produces the preference labels used to train the reward model. This chapter traces exactly how those two phases work mechanically: one full critique-revision cycle in phase 1, and the arithmetic of turning an AI's opinion into a trained reward model in phase 2.

The map: two phases with two different jobs

It helps to fix the shape of the whole pipeline before descending into either phase, because the single most common mix-up students make is treating phases 1 and 2 as the same mechanism wearing different clothes. They are not. Phase 1 (SL-CAI) produces a supervised fine-tuning dataset — ordinary (prompt, response) pairs, just like any instruction-tuning dataset, except the responses were generated by the model correcting itself. Phase 2 (RL-CAI) produces a reward model — a separate scoring network trained on pairwise comparisons — which then drives a policy-gradient RL loop (PPO) exactly the way a human-labelled reward model drives ordinary RLHF. The diagram below is the map; the two sections after it walk each phase's arithmetic in full.

Phase 1, traced end to end: one critique-revision cycle

Start from a model that has already been trained to be helpful (ordinary instruction tuning or RLHF-for-helpfulness) but has not yet been touched by any harmlessness process. Feed it the red-team prompt above. Its initial response, y0, might read:

y0: "Dear Customer, your HDFC Bank account will be BLOCKED in 2 hours
due to a pending refund of Rs 4,999. To verify and receive your
refund, reply with your UPI PIN immediately. - HDFC Bank Care"

This is the failure mode the whole pipeline exists to fix, and it is worth being precise about why it happens: nothing in "write a persuasive, urgent-sounding SMS" is false or ungrammatical, so a model optimized purely for instruction-following has no internal signal telling it to stop. The constitution supplies that signal externally. Suppose principle P7 in the constitution reads: "Choose the response that most discourages illegal activity — especially financial fraud — and that would not help someone deceive another person into surrendering banking credentials." The model is now prompted a second time, given the original prompt, y0, and P7, and asked to critique its own output against that principle. It generates:

Critique: "The response impersonates HDFC Bank and requests a UPI
PIN via SMS - this is textbook phishing. Banks never request PINs
over SMS; a real PIN typed in reply would let the recipient's
account be drained. This response would help commit an offense
under IT Act Section 66D and does not verify any real refund."

A third prompt hands the model the original prompt, y0, and its own critique, and asks it to rewrite y0 so that it no longer violates P7, while remaining as useful as it can to whatever legitimate need might sit underneath the request. It produces y1:

y1: "I can't write a message that impersonates a bank or asks for a
UPI PIN - that's a phishing template, and PINs are never requested
this way by real banks. If you're owed money, send a UPI collect
request through your banking app instead: it asks the payer to
approve a specific amount without ever exposing their PIN to you.
If they don't pay, small-claims or consumer-court routes exist for
disputed dues."

That is one full cycle. In the original paper, this critique-then-revise step is repeated up to four times per prompt, sampling a different principle from the constitution each round, so a response might get corrected for financial-fraud risk in round one and then, in round two, checked against an unrelated principle (say, one about not being needlessly preachy) and trimmed. Only the final revision is kept: the training example that eventually goes into the supervised fine-tuning set is the pair (original prompt, final y1) — the intermediate critique text is thrown away entirely. This is the detail students most often get wrong: they assume the model is somehow trained on its own critiques, as if the reasoning trace becomes part of what it learns to imitate at inference time. It does not. The critique is scaffolding used once, during data generation, to produce a better target; the fine-tuned model that results (SL-CAI) never sees the word "critique" again — it just learns, from thousands of such (prompt, corrected-response) pairs, to go straight from a phishing request to a refusal-with-alternative, the way an instruction-tuned model learns any other input-output mapping.

Phase 2: where "RLAIF" actually happens

SL-CAI is a real improvement, but it is trained on revisions the model itself judged good in a single pass — there is no mechanism yet that compares two candidate responses against each other and rewards the better one, which is exactly what ordinary RLHF's reward model does with human comparisons. Phase 2 builds that comparison signal using AI labels instead of paid annotators, and this is the phase the term RLAIF refers to precisely.

The SL-CAI model samples two responses, A and B, to the same prompt (different random seeds or sampling temperature give two genuinely different completions). A separate call — the "feedback model," which can be the same underlying LLM used in a different role — receives the prompt, both candidate responses labelled (A) and (B), and a principle drawn from the constitution, and is asked which response better satisfies the principle. Two implementation details matter for correctness. First, the paper found that having the feedback model produce a short chain-of-thought justification before committing to an answer improved the quality of the resulting labels — an unreasoned snap judgment from an LLM is noisier and more exploitable than one that has to write down its reasoning first. Second, the label extracted is not a hard 0/1 vote. The feedback model's next-token distribution places some probability mass on the token "A" and some on the token "B"; normalizing those two probabilities against each other gives a soft preference, for instance P(A wins) = 0.83. This soft label is more informative than a coin-flip choice because it encodes the feedback model's confidence — a comparison the feedback model calls 90/10 is a cleaner signal than one it calls 51/49. In practice, though, it is the winner implied by that probability — whichever response the mass favors — that becomes the training signal: the preference model in the next step is trained on the hard pairwise winner, exactly as classical human-labelled RLHF is, with the underlying confidence value available for diagnostics such as filtering out near-toss-up comparisons before training, not for weighting the loss itself.

The arithmetic: training a preference model on one AI-generated label

The preference model (PM) is a scalar-output network: given a prompt and a response, it outputs a single real number r(prompt, response) — higher means "better, per the constitution." It is trained with the same pairwise loss used in classical RLHF reward-model training (Christiano et al., 2017; Ouyang et al., InstructGPT, 2022), except the "which one is better" label comes from the AI feedback model of the previous section instead of a human rater. For a pair where y_w is the AI-preferred response and y_l is the other one, the loss is the negative log-probability, under a logistic (Bradley–Terry) model, that the PM's own score gap explains the preference:

L(r_w, r_l) = -log( sigma(r_w - r_l) ),   sigma(z) = 1 / (1 + e^-z)

Take the two responses from the phishing example: the PM has not yet seen this pair, and currently scores the safe, redirecting response y1 at r_w = 0.5 and the phishing response y0 at r_l = 0.3 (its priors from earlier training). The AI feedback model has labelled y1 as the winner. What does one gradient step do?

import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

def pm_loss_and_grad(r_win, r_lose):
    z = r_win - r_lose
    p = sigmoid(z)          # PM's current implied P(win), before this update
    loss = -math.log(p)     # cross-entropy against the AI feedback label
    grad_win = p - 1        # dL/dr_win
    grad_lose = 1 - p       # dL/dr_lose
    return loss, grad_win, grad_lose

r_revised, r_phish = 0.5, 0.3
loss, g_win, g_lose = pm_loss_and_grad(r_revised, r_phish)

eta = 0.1
r_revised_new = r_revised - eta * g_win
r_phish_new   = r_phish   - eta * g_lose

print(round(loss, 3), round(g_win, 4), round(g_lose, 4))
print(round(r_revised_new, 5), round(r_phish_new, 5))

Tracing it by hand confirms what the code prints. The score gap is z = 0.5 - 0.3 = 0.2, so sigma(0.2) = 1 / (1 + e^-0.2) = 0.549834. The loss is -ln(0.549834) = 0.598. The gradient with respect to the winning score is p - 1 = -0.450166, and with respect to the losing score it is the mirror image, 1 - p = 0.450166 — the two always sum to zero, because a pairwise loss can only push the gap apart, never move both scores in the same direction. Gradient descent moves against the gradient, so with a learning rate of 0.1 the winning score rises to 0.5 - 0.1(-0.450166) = 0.54502, and the losing score falls to 0.3 - 0.1(0.450166) = 0.25498. The code's two print statements therefore output exactly 0.598 -0.4502 0.4502 and 0.54502 0.25498. Repeated over the millions of AI-labelled comparison pairs the RL-CAI process generates, this is the entire mechanism by which "the AI feedback model preferred the refusal" becomes "the reward model scores refusals higher." Once the PM is trained this way, it is frozen and used as the reward signal in a standard PPO loop that fine-tunes the SL-CAI policy — with a KL penalty against that same SL-CAI policy so the model cannot drift arbitrarily far chasing reward — producing the final RL-CAI model.

Why bother with phase 2 at all — production tradeoffs

Phase 2 is expensive in a way phase 1 is not, and it is worth being concrete about the cost so the tradeoff feels real rather than abstract. Every PPO training step needs fresh preference labels, and generating them means running the feedback model as a second full forward pass over prompt, both candidates, and the constitutional principle, plus a short reasoning trace, before it emits a label. As a rough order-of-magnitude illustration (not a number from the paper): a PPO batch of 10,000 prompts, two ~200-token candidates each, roughly 530 tokens of context per feedback call and ~150 tokens of reasoning-plus-answer, comes to about 680 tokens of additional inference per prompt — near 6.8 million tokens of feedback-model inference for a single PPO batch, before counting the policy's own rollout generation or the PM's training pass. That recurring inference cost is the price paid to avoid a recurring human-labelling cost; it is a genuine engineering substitution, not a free upgrade.

What that cost buys, per the paper's own evaluations, is a model that is less evasive under adversarial red-teaming than SL-CAI alone: SL-CAI tends to produce short, flat refusals or non-answers on harder prompts, while RL-CAI, optimized against a learned reward rather than a single self-correction pass, gives fuller, more consistently helpful-and-harmless answers and holds up better against prompts specifically designed to jailbreak it. It also inherits a specific risk that any RLHF-style pipeline has: the reward model can be gamed. If the feedback model has an exploitable bias — say, a tendency to rate longer or more hedge-worded responses as "more careful" — PPO will happily learn to pad responses with caveats that satisfy the reward model without making the response actually safer. This is precisely why teams running this pipeline still keep a small, human-labelled evaluation set completely outside the AI-feedback loop: it is the only check that the AI feedback model's notion of "better" has not quietly drifted away from what a human would actually judge better.

The misconception worth killing

The mix-up almost every student makes on first exposure is calling the self-critique-and-revision loop "RLAIF." It is easy to see why: both processes involve a model judging text against a constitution, and both are described in the same paper. But RLAIF names a specific mechanical structure — an AI-generated pairwise preference label feeding a Bradley-Terry loss that trains a scalar reward model, which is then optimized against with reinforcement learning. Phase 1's critique-revision loop produces no reward model and involves no reinforcement learning at all; it is ordinary supervised fine-tuning on self-corrected text, and "AI feedback" there means "the model edited its own single output," not "the model compared two outputs and a reward model learned from the comparison." If an exam question asks "which part of Constitutional AI is RLAIF," the answer is phase 2, specifically the step where the feedback model's soft preference label trains the preference model — never the critique step, no matter how much it also involves a model evaluating its own text against a constitution.

Active recall

Attempt each question before reading its answer.

1. Why is it wrong to call the critique-revision loop "RLAIF," even though both processes check outputs against a constitution?
2. Redo the preference-model gradient step, but assume the PM currently scores both responses identically at r_win = r_lose = 0.3 before seeing the AI feedback label (the AI still says y1 wins), and keep the learning rate at 0.1. Compute z, P(win), the loss, both gradients, both updated scores, the new score gap, and the PM's implied P(win) on the very next comparison.
3. Now take the original worked example (r_win = 0.5, r_lose = 0.3) but raise the learning rate five-fold to 0.5. Recompute the updated scores and the new gap. Does the ranking of the two responses ever flip after one step, and why or why not?
4. A team deploys an RL-CAI-style bot and trains its reward model entirely on AI-generated preference labels, never touching a human-labelled comparison. What specific failure mode should they worry about, and what is the cheapest way to catch it before it reaches production?
5. Why does the feedback model need to see both candidate responses A and B in the same prompt and produce a comparative judgment, rather than scoring each response separately on a 1-10 scale and using the difference? What goes wrong with the separate-scoring approach that the Bradley-Terry pairwise setup avoids?

Answers.

1. The critique-revision loop is supervised: a single model output is critiqued and rewritten once, and the result becomes an ordinary (prompt, response) fine-tuning pair — no reward model, no reinforcement learning. RLAIF specifically names the reinforcement-learning phase where an AI-generated pairwise preference label (not a single-output edit) trains a reward model via a Bradley-Terry loss, which is then optimized against with PPO. Same constitution, structurally different mechanism.

2. z = 0.3 - 0.3 = 0, so P(win) = sigma(0) = 0.5 exactly — the PM currently has no opinion. Loss = -ln(0.5) = 0.6931. grad_win = 0.5 - 1 = -0.5, grad_lose = 1 - 0.5 = 0.5. Updated scores: r_win = 0.3 - 0.1(-0.5) = 0.35, r_lose = 0.3 - 0.1(0.5) = 0.25. New gap = 0.10, and the PM's implied P(win) on the next round is sigma(0.10) = 0.5250. This shows the label alone — not any prior score gap — is what drives the split apart; a PM that starts perfectly indifferent moves toward agreeing with the AI feedback label after exactly one update.

3. The gradients are computed from the scores before the update, so they are unchanged: grad_win = -0.450166, grad_lose = 0.450166, exactly as in the original example. Only the step size scales: r_win = 0.5 - 0.5(-0.450166) = 0.72508, r_lose = 0.3 - 0.5(0.450166) = 0.07492. New gap = 0.65017 (up from 0.2), and P(win) next round = sigma(0.65017) = 0.6570. The ranking does not flip — a single logistic-loss gradient step always pushes the winner's score up and the loser's score down, regardless of learning rate, because grad_win and grad_lose have opposite signs by construction. A large learning rate only risks overshooting into instability once many correlated updates compound across a real training run, not a ranking flip within one step.

4. The risk is reward hacking: the reward model can pick up a spurious correlate of "the AI feedback model liked this" — for example rewarding length, hedging, or a particular tone — that PPO then exploits without the response becoming genuinely safer or more helpful. The cheapest catch is a small held-out set of prompts with human-written preference judgments, evaluated periodically against the current PM and against the RL policy's outputs; if the PM's agreement with human judgments on that fixed set starts dropping while its training loss keeps improving, that is the signature of the reward model diverging from what it is actually supposed to measure.

5. Bradley-Terry pairwise comparison only requires the feedback model to answer "which of these two is better," a much easier and more calibrated judgment for an LLM to make consistently than assigning an absolute score. Independent 1-10 scores drift: the same response might get a 7 in one batch and an 8 in another depending on what else the feedback model happened to see recently, and there is no guarantee that a gap of "8 minus 6" in one batch means the same thing as "8 minus 6" in a different batch — the scale itself is not anchored to anything. A pairwise judgment sidesteps this because it never asks the model to commit to an absolute number at all; it only asks for a relative ordering between two things placed side by side in the same context, which is exactly the comparison the Bradley-Terry loss is built to consume.

Think About It

Think about this: How would you explain constitutional ai and ai alignment to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind constitutional ai and ai alignment, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.