Constitutional AI: Making AI Systems Harmless and Honest

The bottleneck no one wants to staff

Picture a fintech building a Hindi/Tamil/Telugu support model for UPI dispute resolution, or a state agriculture department shipping a farmer helpline bot that will field lakhs of queries in a dozen languages. Before either ships, the safety team runs the standard playbook: reinforcement learning from human feedback (RLHF). Human raters look at pairs of model responses to adversarial prompts, things designed to elicit self-harm instructions, communal-tension bait, illegal-lending advice, and mark which response is the less harmful one. To train a reward model that actually generalizes, that comparison has to happen tens of thousands of times, redone whenever a new attack pattern surfaces, a new language rolls out, or a policy changes. Every single comparison requires a person to read genuinely harmful text and hold it in their head long enough to judge it. This is not a hypothetical cost. Content-moderation labor at platforms like Meta has been documented, including in litigation from outsourced review hubs, to cause real psychological harm to the people doing it, and RLHF's harmlessness half of the pipeline is structurally the same job: read harmful text for a living, at scale.

That is the specific bottleneck this chapter addresses, not RLHF cost in general, but the harmlessness-labeling cost specifically, which scales with the product of adversarial prompts, languages, principles, and human hours, and which puts a person in the loop for the most unpleasant part of the pipeline. In December 2022, Anthropic (lead author Yuntao Bai, with a large team including Jared Kaplan and the Anthropic alignment group) published a method that keeps humans exactly where their judgment is irreplaceable, deciding what "helpful" and "harmless" should mean, and removes them from the part that is mechanical execution of that judgment across millions of completions. The method is Constitutional AI (CAI), and the mechanism that replaces the human labelers is called Reinforcement Learning from AI Feedback (RLAIF).

From RLHF to RLAIF, from first principles

Standard RLHF has three stages: supervised fine-tuning on demonstrations, training a reward model on human A/B comparisons, then running PPO to push the policy toward whatever the reward model scores highly. Two things go wrong when the comparisons being labeled are about harm rather than helpfulness. First, the labeling cost described above. Second, a subtler failure mode documented in Anthropic's earlier work on RLHF for a helpful-and-harmless assistant (Bai et al., April 2022): models trained this way tend toward evasion. A rater looking at two completions and clicking whichever one feels safer is optimizing for "doesn't sound bad," and a model that learns to satisfy that signal learns to refuse anything that pattern-matches to a sensitive topic, including entirely reasonable requests, without ever being forced to say which rule it thinks it is following. The rater's click carries no reasoning, so the model never has to produce any.

Constitutional AI attacks both problems with the same idea: replace the human's gut-reaction comparison with the model applying an explicit, written-down principle to its own output, and force the model to write out its reasoning against that principle during training, not just react to a label. "Constitutional" is the right word for this: a constitution states general principles that get interpreted case by case, rather than an exhaustive rulebook enumerating every situation in advance. The model plays the role of the interpreter, generalizing a short list of principles across the enormous space of prompts a legislature could never have enumerated by hand.

Phase 1: SL-CAI, self-critique and self-revision

The pipeline starts from a model that is already helpful, typically the output of an ordinary helpful-only RLHF run, since CAI builds harmlessness on top of an existing helpfulness capability rather than training both from scratch at once. Feed this helpful model red-team prompts engineered to elicit harmful completions, and sample its response. Then, in a second turn on the same conversation, sample one principle at random from the constitution (Anthropic's version drew on outside sources including the UN Declaration of Human Rights and Apple's developer terms of service, combined with principles Anthropic wrote itself, on the order of ten to twenty explicit rules in total) and prompt the model to critique its own prior response specifically against that principle. In a third turn, ask it to rewrite the response given its own critique. Repeat this critique-then-revise cycle for several rounds, up to four in the original paper, sampling a fresh principle each round so a single response gets pressure-tested against a spread of different rules rather than just one.

Only the final revision from each chain is kept; the intermediate critique and revision text is discarded scaffolding, not training data in its own right. These (prompt, final-revision) pairs, mixed with ordinary helpfulness data so the resulting model does not regress on tasks it was already competent at, become a standard supervised fine-tuning set applied to a fresh pretrained model. The output is called the SL-CAI model. The mechanical point worth sitting with: nowhere in Phase 1 does a human read a harmful completion. The only human labor is upstream, writing the constitution once, and that cost amortizes across every prompt, every language, and every future re-run of the pipeline.

Phase 2: RLAIF, AI-generated preference labels

Supervised fine-tuning alone tends to reproduce its training distribution rather than optimize a reward signal, so it generalizes less well to novel adversarial prompts than an RL-tuned policy does. Phase 2 reproduces the RLHF recipe but swaps out the harmlessness half of the comparison data.

Sample the SL-CAI model twice on the same red-team prompt to get two candidate responses, A and B. Show a separate copy of the model, acting as a feedback model, the pair plus one randomly sampled constitutional principle, formatted as a forced-choice question: which response better follows this principle, (A) or (B)? Rather than taking the argmax as a hard 0-or-1 label, extract the feedback model's own probability mass on the "(A" token versus the "(B" token and renormalize the two so they sum to one. This renormalized value is the AI-generated soft preference label, and it is more informative than a hard label because it encodes the feedback model's confidence, not just its top pick. These AI-labeled harmlessness comparisons are pooled with ordinary human-labeled helpfulness comparisons (deciding which of two helpful answers is better is not the psychologically costly half of the job, so humans keep doing it), and a single preference model (PM) is trained on the combined set with the standard Bradley-Terry pairwise loss. Finally, PPO runs against the PM as the reward signal, producing the RL-CAI model, the system the paper reports on.

Why "harmless" and "honest" run through the same machinery

This chapter's title names two properties, harmless and honest, though the original paper itself is titled only "Constitutional AI: Harmlessness from AI Feedback" — honesty is not in its title even though honesty principles appear in its constitution. It is worth being precise about why one pipeline handles both anyway. The constitution is not exclusively a list of harm-avoidance rules about violence, illegal activity, or discrimination. It also includes honesty principles: avoid asserting claims the model is not actually confident in, express calibrated uncertainty rather than false confidence, do not mislead the user about the model's own nature or limitations. Critique-and-revise does not care which kind of principle it is applying, a critique step that catches an overconfident factual claim looks mechanically identical to one that catches a harmful instruction, just with a different principle sampled that round. The same holds for the RLAIF comparison step: the feedback model can just as easily be asked "which response is more honest about its own uncertainty" as "which response is less harmful." The generality is the actual innovation. A pipeline that only knew how to reduce harm would need a second, separately engineered system to improve honesty; a pipeline organized around applying arbitrary written principles handles both because both properties are expressible as principles a model can apply to its own text.

Worked example: from soft label to preference-model gradient

The step most students gloss over is exactly how a probability distribution over two tokens becomes a training target. Trace it with real numbers.

The feedback model is shown the pair (A, B) and one constitutional principle, and asked to answer with the single token "(A" or "(B". Its output distribution, restricted to just those two tokens, assigns probability mass p_A = 0.86 and p_B = 0.05 (these do not sum to 1 because the rest of the vocabulary's probability mass, punctuation, whitespace, other tokens, absorbs the remainder). The soft label renormalizes within the two-choice subset:

soft_label_A = p_A / (p_A + p_B) = 0.86 / 0.91 = 0.9451
soft_label_B = 1 - soft_label_A = 0.0549

This is the target the preference model (PM) is trained against, using a Bradley-Terry model: if the PM assigns scalar rewards r_A and r_B to the two responses, its own implied probability that A is better is P_PM(A ≻ B) = sigmoid(r_A − r_B). Suppose the PM, mid-training, currently scores A only 1.0 logit above B. Then P_PM(A ≻ B) = sigmoid(1.0) = 0.7311, short of the 0.9451 target, so cross-entropy loss is:

import math

def constitutional_soft_label(p_choice_a: float, p_choice_b: float) -> float:
    """Normalize the feedback model's probability mass on the '(A'
    token vs. the '(B' token into a soft preference label for A,
    as in Bai et al. 2022."""
    return p_choice_a / (p_choice_a + p_choice_b)

def preference_model_loss(soft_label_a: float, reward_gap: float) -> float:
    """Binary cross-entropy between the AI soft label and the PM's
    own Bradley-Terry estimate P(A > B) = sigmoid(reward_gap)."""
    p_pm_a = 1 / (1 + math.exp(-reward_gap))
    soft_label_b = 1 - soft_label_a
    return -(soft_label_a * math.log(p_pm_a) + soft_label_b * math.log(1 - p_pm_a))

p_a, p_b = 0.86, 0.05
soft_a = constitutional_soft_label(p_a, p_b)
print(f"soft label P(A better) = {soft_a:.4f}")          # 0.9451

loss_before = preference_model_loss(soft_a, reward_gap=1.0)
print(f"loss at reward_gap=1.0: {loss_before:.4f}")       # 0.3682

loss_after = preference_model_loss(soft_a, reward_gap=1.5)
print(f"loss at reward_gap=1.5: {loss_after:.4f}")        # 0.2838

Running this produces exactly the three printed values in the comments: a soft label of 0.9451, a loss of 0.3682 when the PM's current gap is 1.0, and a lower loss of 0.2838 once a gradient step has widened the gap to 1.5. That drop confirms the training signal points the right way: because the AI feedback model was confident A was better (0.9451, well above the PM's implied 0.7311), gradient descent pushes r_A − r_B upward, exactly the direction that narrows the gap between the PM's belief and the AI feedback model's. This is the entire mechanism by which a probability the feedback model happened to assign to a token becomes, several arithmetic steps later, a nudge on the policy's weights during the subsequent PPO run.

Common misconception

Students who hear "constitution" and "principles" often assume Constitutional AI means the deployed model consults a rulebook at inference time, a filter bolted onto the outputs, checking each response against a checklist before it goes out. That is not what happens. The constitution is used exclusively during training, in the critique-and-revision loop of Phase 1 and in generating comparison labels in Phase 2. Once RL-CAI is trained, the principles are gone from the runtime path entirely; harmlessness and honesty are baked into the model's weights, the same way any other trained behavior is, and the model at inference time is answering from a single forward pass with no rulebook lookup happening anywhere. If it declines a request or hedges a claim, that is a learned disposition, not a triggered rule.

A production risk worth naming

Phase 2's feedback model and the policy being trained are frequently close relatives, sometimes literally the same checkpoint used in two different roles. Any systematic blind spot or self-favoring quirk in one is then baked directly into the reward signal shaping the other, since the "judge" and the "student" share the same biases by construction. This is a real, general risk with AI-feedback methods, not specific to Anthropic's implementation, and it is one reason the original paper reports checking how well the AI feedback model's soft labels agreed with separate human-crowdworker judgments on the same comparisons before trusting the pipeline at scale: agreement with an independent, non-relative judge is the check against the pipeline quietly grading its own homework. Any team deploying an RLAIF-style pipeline in production, not just Anthropic's original one, needs an equivalent held-out human check, precisely because nothing inside the pipeline itself would surface a shared blind spot.

Active recall

Attempt each question before reading its answer.

1. Why does the labeling bottleneck this chapter opens with apply specifically to harmlessness comparisons and not to helpfulness comparisons in standard RLHF?
2. In Phase 1, why fine-tune only on the final revision of each critique-revise chain, discarding the intermediate critique and revision turns, and why mix in ordinary helpfulness data rather than training purely on the revised harmlessness examples?
3. Redo the worked example with p_A = 0.6 and p_B = 0.6 instead of 0.86 and 0.05. Compute the soft label, and explain what a preference model trained on many examples like this one would (and would not) learn from it, compared to training on confident examples like the original.
4. A classmate claims "Constitutional AI removes humans from AI safety." Identify every point in the two-phase pipeline where a human is still required, and explain why the claim is wrong as stated.
5. Why does the original paper keep human labels for the helpfulness comparisons in Phase 2 instead of generating those with AI feedback too, the same way harmlessness comparisons are generated?
6. If the feedback model in Phase 2 is the SL-CAI model judging pairs produced by that same SL-CAI model, what specific failure mode should you worry about, and what does the paper do to check for it?

Answers.

1. Harmlessness comparisons require a rater to read genuinely harmful content, self-harm instructions, violent content, illegal-activity advice, repeatedly, which carries a real psychological cost documented in content-moderation labor more broadly. Helpfulness comparisons ask a rater to judge which of two benign, on-topic answers is better, which is comparatively low-cost cognitive work with no exposure harm. The bottleneck is specific to the content being read, not to comparison labeling in general.

2. The critique and revision turns are scaffolding that produces the final revision; they are prompts and intermediate reasoning used to get to a better response, not themselves examples of the target behavior a deployed model should imitate (a deployed model isn't supposed to narrate a self-critique before every answer, in this setup). Training only on the final revision teaches the model to produce that improved response directly, on the first turn, without the scaffolding. Mixing in helpfulness data prevents the fine-tune from overfitting to the harm-avoidance distribution of the critique-revise dataset; without it, the resulting model would over-index on caution and get measurably worse at the ordinary helpful tasks it could already do, since supervised fine-tuning has no mechanism to preserve capabilities it doesn't see training examples for.

3. soft_label_A = 0.6 / (0.6 + 0.6) = 0.5, soft_label_B = 0.5, an exact tie. A preference model trained on many ties like this one learns nothing about ranking A versus B specifically, because the target label matches exactly what an uninformed model (one outputting P = 0.5 for everything) would already produce; the cross-entropy loss at that point is minimized by staying at 0.5, so this example contributes close to zero gradient. This is a feature, not noise: it means the soft-label scheme automatically down-weights comparisons the AI feedback model itself was unsure about, unlike a hard-label scheme, which would have forced a coin-flip pick to (A) or (B) and fed the PM a confident-looking label for what was actually an ambiguous case.

4. Humans are required in at least three places: writing the constitution's principles in the first place (nothing in either phase generates principles, both phases only apply and sample from a fixed human-authored list); producing the helpfulness comparison labels used to train the PM alongside the AI-generated harmlessness labels; and, per the paper's own validation methodology, providing an independent held-out check on whether the AI feedback model's judgments track human judgment. What RLAIF removes is specifically the human labor of reading and comparing harmful completions at scale, not human oversight of the system as a whole. The claim conflates "removes humans from one labor-intensive labeling task" with "removes humans from the loop," which is a much stronger and false claim.

5. Human labels for helpfulness stay because that comparison isn't the psychologically costly one, so there is no labor-welfare or scaling argument for automating it away, and because helpfulness judgments (which answer is more useful, more complete, more correctly reasoned) plausibly benefit from the breadth of human judgment about what real users actually find useful in a way that is harder to specify as a short written principle than "don't help someone synthesize a weapon" is. The paper's design choice reflects that RLAIF was built to solve a specific bottleneck, not to replace human feedback wherever it can be replaced.

6. The risk is self-preference bias, or more generally shared blind spots: if the feedback model and the policy being trained share weights, checkpoints, or training history, a systematic error one makes (a specific attack pattern it fails to recognize as harmful, a category of overconfident claim it doesn't flag as dishonest) can go completely undetected, because the "judge" evaluating a response and the "student" that produced it share the same error. Worse, RL could actively learn to exploit whatever the feedback model rewards, including features that a truly independent judge would penalize, a form of reward hacking specific to using a correlated model as the reward source. The paper's stated check is comparing the AI feedback model's soft labels against separate human-crowdworker judgments on the same comparisons, treating agreement with an independent, non-relative judge as the validation that the pipeline hasn't drifted into grading its own homework.

Think About It

Think about this: How would you explain constitutional ai: making ai systems harmless and honest to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where constitutional ai: making ai systems harmless and honest is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting constitutional ai: making ai systems harmless and honest to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind constitutional ai: making ai systems harmless and honest, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.