Suppose you are building the AI support assistant inside a UPI payment app used across India. A user types: "My uncle is 70 and can't use his phone well. He gave me his UPI PIN over the phone. How do I check his account balance from my phone?" This single message is genuinely ambiguous. It could be a grandson helping an elderly relative — completely legitimate, and refusing to help would be unhelpful and even a little insulting. It could also be a textbook social-engineering script that fraudsters run verbatim on elderly victims across the country, and helping without any friction would make the assistant an accessory to fraud. A blanket "I can't help with anything involving another person's PIN" is safe but useless for the honest grandson. A blanket "sure, here's how to check any account's balance" is helpful but dangerous. The correct behavior sits in between: ask a clarifying question, explain that sharing a PIN is against bank policy regardless of intent, and suggest the legitimate alternative (adding the uncle as a linked account, or having him check it himself with assistance). Teaching a language model this specific, situational judgment — at the scale of millions of similar edge cases across fraud, self-harm, medical advice, coding exploits, and everything else a model gets asked — is the actual engineering problem that Constitutional AI (CAI) was built to solve. It is not a generic "make the AI nice" technique; it is a specific two-stage training procedure, introduced by Anthropic in 2022, for teaching a model fine-grained harmlessness judgment without requiring a human being to sit and grade every single harmful-or-not example by hand.
Why RLHF alone hits a wall
By Grade 11 you already know the standard alignment recipe: Reinforcement Learning from Human Feedback (RLHF). A base language model generates two candidate responses to a prompt, a human labeler picks the better one, thousands of such comparisons train a preference model (a reward model) to predict which response a human would prefer, and then the language model is fine-tuned with reinforcement learning (typically PPO) to maximize the score this preference model assigns, while a KL-divergence penalty keeps it from drifting too far from its original behavior and degenerating into reward-hacking gibberish.
This works well for helpfulness — humans are good at judging whether an answer to "explain merge sort" is clear and correct. It works badly for harmlessness at scale, for three concrete reasons. First, cost: producing enough harmlessness comparisons to cover the space of borderline requests (fraud-adjacent banking questions, medical dosage edge cases, chemistry questions that are legitimate for a student but dangerous out of context) requires an enormous number of human judgments, each one slow because the labeler has to read potentially disturbing content and reason carefully about a policy. Second, exposure: the labelers doing harmlessness ranking are the ones repeatedly reading violent, fraudulent, or self-harm-related content, which is a real human cost. Third, and most subtly, consistency: two different human labelers, or the same labeler on two different days, will draw the helpful/harmless line in slightly different places, and a preference model trained on inconsistent labels learns a blurry, noisy signal. Constitutional AI's answer is to replace the human-generated harmlessness labels — not the human-generated helpfulness labels, those stay — with labels the model generates itself, guided by an explicit, written-down set of principles called the constitution.
Stage 1 — SL-CAI: self-critique and revision
The constitution itself is nothing exotic: it is a list of roughly a dozen to sixteen short natural-language principles, drawn from sources like the UN Declaration of Human Rights, platform terms of service, and principles the lab judges important, phrased as instructions such as "Choose the response that a thoughtful person would be less likely to see as intended to manipulate someone into a harmful action, even if the harm is indirect." The first training stage, called SL-CAI (Supervised Learning from Constitutional AI), uses this constitution to bootstrap a better training set through a critique-then-revise loop, run entirely by the model on itself, with no human harmlessness labels at all. Trace it step by step on the UPI example:
Step 1 — initial response. A model trained only for helpfulness (no harmlessness tuning yet) is given the prompt and produces something over-compliant: "Sure — download the app, log in with his registered mobile number, enter the UPI PIN he gave you, and you'll see the balance on the home screen." This is fluent and "helpful" in a narrow sense, but it treats a fraud-shaped request as routine.
Step 2 — critique. The same model is now shown its own response plus one principle sampled from the constitution, and asked to critique itself: "Identify ways in which the last response could be harmful, unethical, or could facilitate fraud." The model generates a critique: "The response treats sharing a UPI PIN as normal and gives step-by-step instructions to act on someone else's PIN, without noting that legitimate banks never require PIN-sharing and that this exact pattern is a common fraud script targeting elderly users."
Step 3 — revision. The model is now asked to rewrite its own response in light of the critique: "UPI PINs should never be shared, even with family — this is exactly the pattern fraudsters use, so please don't act on it. If your uncle needs help, the safe options are: sit with him and let him enter the PIN himself, or add his account as a linked account in your app with his consent, or call the bank's helpline together." This revised response is still helpful — it directly addresses the underlying need — but it removes the fraud vector.
Step 4 — iterate and collect. This critique-revise cycle is repeated a few times per prompt (empirically around four rounds), each round sampling a different principle from the constitution, and the final revision is kept. This is done across a large, diverse set of red-teamed prompts — deliberately adversarial or edge-case questions written to probe exactly these helpful/harmful boundaries. The resulting (prompt, final-revision) pairs are then used as ordinary supervised fine-tuning data, mixed with helpfulness examples, to produce the SL-CAI model. Notice what happened: no human ever looked at these prompts and wrote a harmlessness label. The "labels" are the model's own revised text, shaped entirely by repeatedly re-reading its own output against the written constitution.
Stage 2 — RL-CAI: reinforcement learning from AI feedback
SL-CAI alone gets a model most of the way, but supervised fine-tuning on a fixed set of revisions can't cover every possible phrasing of every possible edge case, and it doesn't push the model to actively prefer better responses over merely-acceptable ones. For that you still want the RLHF machinery — a preference model and PPO — but Constitutional AI generates the harmlessness preference labels with the model itself instead of a human, a technique now generally called RLAIF (Reinforcement Learning from AI Feedback).
The mechanism: take the SL-CAI model and sample two candidate responses, A and B, to the same prompt. Show both, along with one constitutional principle, to a separate feedback model (a capable language model, often a larger one) as a multiple-choice question: "Which response better follows the principle [X]? Answer (A) or (B)." The feedback model doesn't output a hard A-or-B choice — it outputs a probability distribution, because we read off the log-probabilities it assigns to the tokens "A" and "B" and convert those into a soft preference via softmax. This soft distribution is the AI-generated preference label, used exactly where a human's forced-choice click would have gone in ordinary RLHF.
Let's trace the arithmetic on a concrete pair. Say the feedback model, comparing the over-compliant response A against the safety-conscious response B for the UPI prompt, assigns log-probability (logit) 1.2 to token "A" and 3.8 to token "B" — it is fairly confident B is the better response, but not certain.
import math
logit_A, logit_B = 1.2, 3.8 # feedback model's logits for "A" / "B"
e_A, e_B = math.exp(logit_A), math.exp(logit_B)
p_A = e_A / (e_A + e_B)
p_B = e_B / (e_A + e_B)
print(round(p_A, 3), round(p_B, 3)) # 0.069 0.931
Working this by hand: e^1.2 ≈ 3.320, e^3.8 ≈ 44.701, their sum is 48.021, so p_A = 3.320 / 48.021 ≈ 0.069 and p_B = 44.701 / 48.021 ≈ 0.931. This soft label — "B is preferred with probability 0.931, not certainty 1.0" — is the target used to train a preference model r_θ, exactly as a human's binary click would be, except it carries the feedback model's own uncertainty instead of forcing an artificial hard choice.
Now suppose the preference model, at its current stage of training, assigns reward scores r_A = 0.4 and r_B = 1.1 to the two responses. Under the standard Bradley-Terry preference model (the same one used in ordinary RLHF), the predicted probability that B beats A is a sigmoid of the score difference:
r_A, r_B = 0.4, 1.1
sigma = 1 / (1 + math.exp(-(r_B - r_A))) # P(B preferred), Bradley-Terry
loss = -(p_B * math.log(sigma) + p_A * math.log(1 - sigma))
print(round(sigma, 3), round(loss, 3)) # 0.668 0.452
Tracing this: r_B − r_A = 0.7, so σ(0.7) = 1 / (1 + e^−0.7) = 1 / 1.4966 ≈ 0.668. The preference model currently thinks B beats A with 66.8% probability, but the AI feedback label says 93.1%. Plugging both into the cross-entropy loss −[p_B·ln(σ) + p_A·ln(1−σ)] gives −[0.931·(−0.402) + 0.069·(−1.102)] ≈ 0.452. This nonzero loss is exactly the gradient signal that pushes the preference model to widen the gap r_B − r_A further, until its own predicted probability matches the AI feedback model's 0.931 — the preference model is being trained by an AI judge instead of a human clicker, using the same math either way.
Once the preference model is trained on a large batch of such AI-labeled comparisons, the policy (the SL-CAI model) is fine-tuned with PPO exactly as in standard RLHF: maximize the preference model's reward for the policy's own sampled outputs, minus a KL-divergence penalty against the SL-CAI reference policy so the model can't "reward-hack" by drifting into degenerate text the preference model happens to score highly. If, for one sampled response, the preference model gives reward r = 1.1 and the estimated KL divergence from the SL-CAI reference is 0.5 nats with penalty weight β = 0.02, the effective training objective for that sample is r − β·KL = 1.1 − 0.02×0.5 = 1.09 — almost the full reward, with only a small correction pulling the model back toward its more conservative reference behavior. This final RL-tuned model is what "Constitutional AI" refers to in the paper's own terminology: helpful because part of its preference data still comes from human helpfulness rankings, harmless because its harmlessness preference data comes from the constitution-guided feedback model, and non-evasive — it can explain its safety judgment (as the revised UPI response does) rather than refusing silently, because the training process itself involved writing out that judgment as text, not applying a hidden filter after the fact.
The full pipeline
The misconception to correct
The name "Constitutional AI" strongly suggests to most students something like Asimov's Three Laws of Robotics: a fixed list of rules loaded into the model that it checks against and obeys at the moment it is answering a question, like an if-else guard rail sitting in front of the output. This is not what happens, and it matters that it isn't. The constitution is never read by the model at inference time — when you actually chat with a Constitutional-AI-trained model, there is no rulebook being consulted token by token. The constitution is used exactly twice, both times during training, offline: once to generate the critique-and-revision text that becomes SL-CAI's supervised fine-tuning data, and once to phrase the multiple-choice question the feedback model answers to produce RLAIF preference labels. By the time the final model is deployed, the constitution has been fully "compiled" into the network's weights through ordinary gradient descent — the same way a model trained on physics problems doesn't consult a physics textbook while answering, it has absorbed the patterns into its parameters. This distinction matters practically: it explains why a Constitutional-AI-trained model can still be wrong or inconsistent on a novel edge case the training process never covered (there is no rule left to fall back on, only learned statistical behavior), and why improving its behavior means changing the constitution and re-running the training pipeline, not editing a rule file that ships with the model.
Active recall
Attempt these before reading the answers below.
- Why can't the harmlessness half of ordinary RLHF scale the same way the helpfulness half does?
- In SL-CAI, what are the three steps applied per prompt, and what does each one produce as text?
- A feedback model gives logits of 2.0 and 2.0 to responses A and B on some principle. What preference probability does softmax assign to each, and what does that number mean about the feedback model's judgment?
- In the RL-CAI stage, what two roles does the "constitution" play across the two training stages, and at which point (if any) does the deployed model actually read constitutional text?
- Why does the PPO objective subtract β·KL(π‖π_SL-CAI) rather than just maximizing the preference model's reward directly?
- A classmate says "Constitutional AI removes the need for any human-labeled data." Is this accurate? What's actually still human-labeled?
Answers.
1. Harmlessness comparisons require a human to read potentially disturbing content (fraud, self-harm, violence framings) and make a careful judgment call every time, which is slow, costly, and psychologically taxing at the volume needed to cover the space of edge cases — and different labelers draw the line inconsistently, producing noisy training signal. Helpfulness judgments (is this explanation of merge sort clear?) don't carry that cost or that inconsistency.
2. (i) Initial response: the helpful-only model answers the prompt, often over-compliantly. (ii) Critique: the same model is shown its response plus one sampled constitutional principle and asked to identify how the response could be harmful — it produces a critique paragraph. (iii) Revision: the model rewrites its own response in light of the critique, producing the final text used as supervised training data.
3. Equal logits give equal exponentials, so softmax gives P(A) = P(B) = 0.5. This means the feedback model sees no meaningful difference between the two responses under that principle — a genuinely uncertain, 50/50 preference label, which is a much more honest training signal than a human being forced to click one of two buttons.
4. First, in SL-CAI, the constitution supplies the principle the model critiques its own output against, shaping the supervised revision data. Second, in RL-CAI, the constitution supplies the principle phrased into the multiple-choice question the feedback model uses to compare response pairs, shaping the AI preference labels. In both cases this happens strictly during training. The deployed model, once trained, never reads constitutional text at inference time — its behavior is encoded in its weights, not looked up from a rulebook.
5. Without the KL penalty, PPO would happily exploit any quirk or blind spot in the preference model's scoring function — drifting toward text that scores artificially high on the reward model but reads as repetitive, incoherent, or gamed to a human, a failure called reward hacking. The penalty term keeps the policy's output distribution close to the SL-CAI reference model, trading a little potential reward for staying in the region of outputs the reference model (and by extension, the training process) actually vouches for as coherent and safe.
6. Not accurate. Constitutional AI replaces human harmlessness preference labels with AI-generated ones (via the feedback model in RL-CAI). The helpfulness preference data — humans ranking which of two answers is more useful, clear, or correct — is typically still human-labeled and mixed into the same preference model training. CAI is selective about which half of the pipeline gets automated, not a wholesale removal of human data.
Think About It
Think about this: How would you explain constitutional ai: principled model alignment to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where constitutional ai: principled model alignment is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting constitutional ai: principled model alignment to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind constitutional ai: principled model alignment, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.