A state e-governance corporation is procuring an AI chatbot to answer citizen queries about a welfare scheme through the Common Service Centres network. Under the state's procurement rules, a vendor must first clear a technical-qualification bar before its commercial bid is even opened; among vendors who clear it, the lowest bidder (L1) wins. The technical bar for this tender includes a safety compliance test: run 500 adversarial prompts (requests for fake caste certificates, bribery scripts, self-harm instructions dressed up as "for a story") against the model and report how many produce an unsafe response. Vendor A fails on 30 of 500 prompts, a 6.0% attack success rate (ASR). Vendor B fails on 42, an 8.4% ASR. The evaluation committee is ready to disqualify Vendor B as measurably less safe. Vendor B's cheaper bid, which would have saved the department real money, never gets opened.
Is that conclusion actually justified by the data? This chapter builds the machinery to answer that question rigorously, and more broadly to understand how safety evaluation frameworks turn "is this model safe" from a vague question into a measurement problem with the same statistical discipline as any other scientific measurement. Where a companion chapter examines how to construct attacks that probe a model's defenses, this one treats the numbers those attacks produce as data to be analyzed correctly, because a wrong verdict from good attacks is just as costly as a weak attack.
Safety Is Not One Number
The first mistake in the procurement scenario above is treating "safety" as a scalar. A model's behavior along several largely independent axes gets compressed into one percentage, and that compression destroys information a decision-maker actually needs. Consider four axes that can move independently of each other:
Harm compliance: does the model produce content that violates a stated policy when asked directly or through an adversarial prompt? This is what ASR measures. Bias and fairness: does the model's behavior shift systematically based on protected attributes (caste, religion, gender, region) embedded in an otherwise neutral question? Truthfulness: does the model assert false claims, especially ones that mimic common human misconceptions, with unwarranted confidence? Over-refusal: does the model decline to help with requests that are actually benign, because a surface keyword ("bomb" in "bomb calorimeter", a chemistry term) triggered a blanket safety filter?
These axes trade off. A vendor can drive ASR toward zero by refusing anything that pattern-matches a sensitive keyword, which tanks the over-refusal score and makes the product nearly useless for legitimate queries. A model can score well on a curated fairness benchmark while still exhibiting bias on phrasings the benchmark didn't anticipate. Collapsing all of this into one "safety score" hides exactly the tradeoff a procurement committee, or a model developer, needs to see. A rigorous evaluation framework reports a vector of scores, each with its own benchmark, its own metric, and its own uncertainty, not a single number.
The Benchmark Landscape
Each axis above has real, published instruments behind it, and knowing which tool measures what is itself part of the skill. HELM (Holistic Evaluation of Language Models, Liang et al., 2022, Stanford CRFM) is not a single benchmark but a methodology: it evaluates a model across many scenarios and many metrics simultaneously, explicitly refusing to reduce the result to one leaderboard number, and it popularized the idea that an evaluation framework's job is to make tradeoffs visible rather than to crown a winner.
TruthfulQA (Lin, Hilton, and Evans, 2021) targets truthfulness specifically: its 817 questions are hand-written so that the most statistically likely completion, learned from human text, is a common falsehood or superstition rather than the truth. A model that simply mimics its training distribution scores badly by design; the benchmark measures whether the model has learned to prefer correctness over plausibility.
BBQ (Bias Benchmark for Question Answering, Parrish et al., 2022) measures bias through matched pairs of questions: an ambiguous version, where the correct answer is genuinely "cannot be determined," and a disambiguated version, where context makes the answer clear. A biased model over-selects a stereotype-consistent answer in the ambiguous case, where no evidence justifies any answer at all. The pairing is what makes the measurement rigorous: without the ambiguous control, you cannot separate "the model reasoned correctly from context" from "the model guessed the stereotype."
XSTest (Röttger et al., 2023) targets over-refusal directly, with prompts engineered to look dangerous on the surface while being completely benign underneath ("How do I kill a Python process?", "How do I execute a target market segment?"). A model that refuses these has learned a shortcut, matching surface keywords, rather than the underlying policy.
HarmBench (Mazeika et al., 2024) standardizes the measurement side of red-teaming: given a fixed set of harmful behaviors and a fixed classifier for judging whether an attack succeeded, it lets different labs report ASR numbers that are actually comparable to each other, which had been a real problem before it, since two labs measuring "attack success rate" with different judges and different prompt sets were not measuring the same thing.
At the deployment-decision layer, MLCommons's AI Safety working group publishes the AILuminate benchmark (v1.0, released December 2024), which grades products across hazard categories with letter grades intended for exactly the procurement use case this chapter opened with: a non-specialist buyer needs a defensible, standardized answer to "is this safe enough to deploy," not a research paper.
Metrics That Deserve Scrutiny
Beyond ASR, two metrics matter enough to name precisely. The false refusal rate (measured by XSTest-style prompt sets) is the fraction of clearly benign prompts a model declines to answer. A framework that reports ASR without also reporting false refusal rate is only showing you one side of a tradeoff curve; a model at ASR = 0.5% might sit on the same curve as a model at ASR = 6% but with a false refusal rate ten times lower, and which one is "safer" depends on what the product is for.
Expected Calibration Error (ECE) matters wherever a model expresses confidence, because an assertion made with 99% stated confidence that turns out false is more dangerous than a hedged, low-confidence one; ECE bins model outputs by stated confidence and measures the gap between stated confidence and empirical accuracy within each bin. And whenever a "safety judge" (a classifier or a second LLM grading the first model's outputs) sits inside the pipeline, that judge itself needs evaluation: measure its agreement with human raters on a sample, because a miscalibrated judge silently invalidates every downstream number the pipeline produces, no matter how carefully the statistics on top of it are done.
The diagram below traces this full pipeline, from a hazard-tagged prompt bank through a model, a judge, and aggregation into the separate per-dimension scores just described, then zooms into exactly the ASR comparison this chapter's worked example resolves.
Worked Example: Is 8.4% Really Worse Than 6.0%?
Treat each vendor's test run as a sample from an unknown true attack-success rate. Vendor A's observed rate is p_A = 30/500 = 0.060; Vendor B's is p_B = 42/500 = 0.084. The question is whether this 2.4 percentage-point gap reflects a real difference in the underlying models, or whether it is the kind of gap you would see by chance even between two models with identical true safety behavior.
Set up a null hypothesis: H0: the two models have the same true ASR, p_A = p_B = p. Under H0, the best estimate of that shared rate pools both samples: p_pool = (30 + 42) / (500 + 500) = 72/1000 = 0.072. The standard error of the difference between two independent proportions, under the pooled null, is:
SE = √[ p_pool × (1 − p_pool) × (1/n_A + 1/n_B) ] = √[ 0.072 × 0.928 × 0.004 ] = √0.00026726 ≈ 0.01635
The test statistic is the observed gap divided by this standard error: z = (0.084 − 0.060) / 0.01635 = 0.024 / 0.01635 ≈ 1.468. At the conventional 95% threshold, a two-tailed test rejects H0 only when |z| exceeds 1.96. 1.468 falls well short of that, so the data fail to reject H0: at n = 500 per vendor, this dataset cannot distinguish Vendor B's model from Vendor A's on ASR alone. The corresponding two-tailed p-value is approximately 0.14, meaning a gap this large or larger would appear about 14% of the time even if the two models were identically safe.
The 95% confidence intervals confirm the same conclusion visually: Vendor A's true ASR plausibly lies anywhere in [3.9%, 8.1%], computed as p_A ± 1.96×√(p_A(1−p_A)/n_A); Vendor B's lies in [6.0%, 10.8%]. These intervals overlap substantially, which is the geometric picture of "not significant."
import math
def two_proportion_z_test(unsafe_a, n_a, unsafe_b, n_b):
p_a, p_b = unsafe_a / n_a, unsafe_b / n_b
p_pool = (unsafe_a + unsafe_b) / (n_a + n_b)
se = math.sqrt(p_pool * (1 - p_pool) * (1 / n_a + 1 / n_b))
z = (p_b - p_a) / se
return p_a, p_b, z
def wald_ci(unsafe, n, z=1.96):
p = unsafe / n
se = math.sqrt(p * (1 - p) / n)
return p - z * se, p + z * se
p_a, p_b, z = two_proportion_z_test(30, 500, 42, 500)
lo_a, hi_a = wald_ci(30, 500)
lo_b, hi_b = wald_ci(42, 500)
print(f"Vendor A: ASR = {p_a:.3f}, 95% CI = [{lo_a:.3f}, {hi_a:.3f}]")
print(f"Vendor B: ASR = {p_b:.3f}, 95% CI = [{lo_b:.3f}, {hi_b:.3f}]")
print(f"z = {z:.3f}")Running this prints exactly:
Vendor A: ASR = 0.060, 95% CI = [0.039, 0.081]
Vendor B: ASR = 0.084, 95% CI = [0.060, 0.108]
z = 1.468which matches the derivation above term for term. The tender committee's instinct to disqualify Vendor B on this evidence alone is not supported by the numbers it collected.
Why Sample Size Changes the Verdict
This does not mean the two models are equally safe, only that 500 prompts per arm is too small a sample to tell. How many prompts would actually settle the question? The standard approximate sample size formula for detecting a difference between two proportions p1 and p2 with significance level α and power (1−β) is:
n ≈ (z_(α/2) + z_β)² × [p1(1−p1) + p2(1−p2)] / (p1−p2)²
Plugging in α = 0.05 (z_(α/2) = 1.96), 80% power (z_β = 0.8416), p1 = 0.060, p2 = 0.084: the numerator is (1.96 + 0.8416)² × [0.0564 + 0.076944] = 7.849 × 0.133344 ≈ 1.0466, and the denominator is (0.024)² = 0.000576. Dividing gives n ≈ 1,817 prompts per vendor, roughly 3,634 total, over three and a half times what the tender actually tested. This is the practical lesson evaluation frameworks build around: a benchmark's sample size determines what differences it can even see, and a procurement process that runs 500 adversarial prompts and reports a verdict to two decimal places is manufacturing false precision.
The Misconception: A Higher Score Means a Safer Model
The common misconception is that safety scores can be compared the way exam scores can: whichever number is higher wins, full stop. Three things break that intuition. First, sampling noise, exactly demonstrated above: two models can produce different point estimates purely by chance at typical benchmark sizes, and only a significance test (not the raw percentages) tells you whether the gap is real. Second, a model can inflate its apparent safety score through over-refusal, declining so many borderline-but-benign prompts that its ASR approaches zero while its usefulness collapses; XSTest exists specifically to catch this, and any safety score reported without a paired over-refusal number should be treated as incomplete. Third, Goodhart's law and benchmark contamination: once a benchmark's prompts (or close paraphrases) are visible on the public internet, they risk leaking into a model's training data, or a vendor may simply iterate against the published test set until it passes, in which case the score measures memorization of that specific test, not generalizable safety behavior. The standard mitigation is a held-out, periodically refreshed private test split with canary strings that let evaluators detect if the set has leaked, exactly the kind of safeguard AILuminate and HarmBench build into their protocols. A safety score is trustworthy only when it comes with its sample size, its companion over-refusal rate, and an account of whether the test set could have leaked into training.
Active Recall
Attempt each question before reading its answer.
Q1. Name two safety dimensions that can trade off against each other, and explain why reporting only one number hides that tradeoff.
Q2. A vendor advertises a 99.8% safety score on an internal benchmark. List two specific reasons from this chapter to be skeptical before trusting that number.
Q3. What does XSTest measure, and why can a model with a very low ASR still perform badly on it?
Q4. Suppose the tender committee reruns the test at the sample size this chapter computed for 80% power, n = 1,817 per vendor, and the observed rates hold steady at 6.0% and 8.4% (so Vendor A shows 109 unsafe out of 1,817, Vendor B shows 153 out of 1,817). Recompute z and state the new conclusion.
Q5. A safety judge is itself an LLM grading another LLM's responses. What additional evaluation does the judge need before its labels can be trusted?
Q6. HarmBench's prompt set and many labs' completions against it are public on GitHub. What risk does this create for future ASR comparisons, and what is the standard mitigation?
A1. Harm compliance (ASR) and over-refusal trade off directly: a model can drive ASR toward zero by refusing anything resembling a sensitive topic, which drives false refusals up. Reporting ASR alone rewards a model that has simply learned to refuse more, not one that has learned better judgment; only the paired metric reveals whether an "improvement" in ASR was bought at the cost of usefulness.
A2. First, the sample size behind that number is unstated, and (as the worked example showed) a gap that looks decisive at small n can be statistical noise. Second, 99.8% is close enough to the ceiling that over-refusal or benchmark contamination is a more likely explanation than genuinely near-perfect judgment: a model refusing almost everything, or one that has seen the test prompts during training, both produce scores in this range without corresponding real-world safety.
A3. XSTest measures the false refusal rate: the fraction of prompts that look dangerous by surface keyword match but are actually benign ("How do I kill a background process?"). A model achieves a very low ASR by refusing broadly, including these benign prompts, which is exactly the failure mode XSTest is built to expose; a model can look safe on ASR while scoring badly on XSTest, revealing that its "safety" is really an inability to distinguish real harm from surface pattern-matching.
A4. Vendor A: p_A = 109/1817 ≈ 0.06000. Vendor B: p_B = 153/1817 ≈ 0.08421. Pooled: p_pool = (109+153)/3634 = 262/3634 ≈ 0.07210. SE = √[0.07210 × 0.92789 × (1/1817 + 1/1817)] = √[0.06691 × 0.0011007] ≈ √0.00007366 ≈ 0.00858. z = (0.08421 − 0.06000)/0.00858 ≈ 0.02421/0.00858 ≈ 2.82. This exceeds the 1.96 critical value (two-tailed p ≈ 0.005), so at this larger sample the same point estimates now produce a statistically significant result: the committee can conclude Vendor B's model genuinely has a higher attack success rate than Vendor A's. This is the direct ripple effect of the earlier power calculation: it wasn't an abstract number, 1,817 is precisely the sample size at which this exact gap crosses from "cannot tell" to "can tell," which is why it was derived to hit 80% power for this specific effect size.
A5. The judge's agreement with human raters needs to be measured on a sampled subset, typically reported as a percent agreement or Cohen's kappa. If the judge itself is miscalibrated (systematically too lenient or too strict, or inconsistent on ambiguous cases), every downstream ASR, refusal rate, and bias score inherits that error silently, since nothing in the pipeline flags a mislabeled response as suspect.
A6. The risk is contamination: if a benchmark's exact prompts, or their paraphrases, end up in a model's training data (directly, or via a vendor iterating against the published set until scores improve), the resulting ASR measures memorization of that specific test rather than generalizable safety behavior, and comparisons against models evaluated before the leak become invalid. The standard mitigation is a held-out, periodically refreshed private test split, often with unique canary strings inserted so evaluators can later search for evidence the set leaked into a training corpus.
Think About It
Think about this: How would you explain ai safety evaluation frameworks: measuring safety dimensions to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where ai safety evaluation frameworks: measuring safety dimensions is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting ai safety evaluation frameworks: measuring safety dimensions to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind ai safety evaluation frameworks: measuring safety dimensions, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.