A mid-size Indian bank is choosing an LLM to power a customer-support assistant that will draft replies to loan queries in English and Hindi-English code-mix. The vendor deck for Model A shows a leaderboard screenshot: 86.1% on some public benchmark, beating Model B's 83.4%. The bank picks Model A. Three weeks after launch, Model A is quietly worse at the actual job — it fabricates EMI figures on ambiguous queries, mishandles code-mixed input, and occasionally refuses benign requests it misreads as fraud. Model B, the "lower-scoring" one, would have done better. Nothing about the leaderboard number was fake. The number was real, and it was still the wrong basis for the decision — because a benchmark score is not a scalar measurement of "how good the model is." It is the output of a specific, narrow measurement procedure, and the procedure the vendor ran shares almost nothing with the bank's actual task distribution. Understanding how that procedure is built — what MMLU actually measures, what HumanEval actually measures, what HELM does differently, and why red-teaming exists as a separate practice from benchmarking altogether — is what turns "which model scored higher" into "which model will actually work here."
A benchmark is four design decisions, not one number
Every benchmark is fully specified by four choices: a task distribution (what set of problems, sampled how), a protocol (how the model is prompted — zero-shot, few-shot, with what system prompt, at what temperature), a scoring function (how one response is judged right or wrong), and an aggregation rule (how per-item scores combine into the one headline number). Two benchmarks that sound similar — "a knowledge test" versus "a coding test" — differ in all four dimensions, and even two runs of the same benchmark can produce different numbers if the aggregation rule changes while the raw per-item results stay identical, as the worked example below shows. Keeping these four knobs explicit is the discipline this chapter builds, because it is exactly the discipline the bank's procurement team skipped.
MMLU: breadth of knowledge, and the aggregation trap
MMLU (Massive Multitask Language Understanding), introduced by Hendrycks, Burns, Basart, Zou, Mazeika, Song, and Steinhardt at ICLR 2021, tests 57 subjects spanning STEM, humanities, social science, and professional domains — from elementary mathematics to abstract algebra to professional law — using four-option multiple-choice questions sourced from real exams and textbooks, typically evaluated few-shot. The task distribution is knowledge recall and reasoning across a huge subject spread; the scoring function is exact-match on the chosen option letter. The subtle part is the aggregation rule, and it is where most students' mental model breaks.
Suppose a model is tested on three MMLU subjects of very different sizes: Elementary Mathematics (200 questions, 180 correct), Abstract Algebra (100 questions, 30 correct), and Formal Logic (50 questions, 20 correct). There are two ways to turn these three numbers into one score. Micro-averaging pools every question together and divides total correct by total questions. Macro-averaging computes each subject's accuracy separately, then averages those subject-level accuracies with equal weight, regardless of how many questions each subject contributed.
subjects = {
"Elementary Math": (180, 200),
"Abstract Algebra": (30, 100),
"Formal Logic": (20, 50),
}
correct_total = sum(c for c, n in subjects.values())
n_total = sum(n for c, n in subjects.values())
micro = correct_total / n_total
macro = sum(c / n for c, n in subjects.values()) / len(subjects)
print(f"micro accuracy = {micro:.4f}")
print(f"macro accuracy = {macro:.4f}")
This produces micro accuracy = 0.6571 and macro accuracy = 0.5333 — a 12.4-percentage-point gap from the aggregation choice alone, on identical underlying performance. The gap exists because Elementary Mathematics contributes 200 of the 350 total questions (57% of the pool) and the model is very strong at it (90%), so pooling lets that one easy, oversized subject drag the overall number up. MMLU's official protocol uses macro-averaging — per-subject accuracy averaged with equal subject weight, further grouped into four top-level categories — specifically so a model cannot buy a high headline score by being excellent at one large, easy subject while being near the guessing floor (25%, for four-option MCQ) on smaller, harder ones. This is not a cosmetic choice: it can flip which of two models ranks higher. Construct Model A (Elementary Math 195/200, Abstract Algebra 5/100, Formal Logic 3/50) and Model B (Elementary Math 100/200, Abstract Algebra 60/100, Formal Logic 40/50). Model A's micro accuracy is 58.0% versus Model B's 57.1% — A wins on pooled accuracy. But A's macro accuracy is 36.2% versus B's 63.3% — B wins decisively once every subject counts equally. A reviewer who only sees "pooled score" would deploy the worse-rounded model.
HumanEval and pass@k: correctness you can run, not correctness you can match
HumanEval, introduced by Chen et al. (OpenAI, 2021) alongside the Codex model, evaluates a fundamentally different kind of task: given a function signature and a docstring, generate a Python function body, then execute it against a hidden suite of unit tests. The scoring function is not string similarity or token overlap — it is binary, functional: does the code actually pass every hidden test when run? This distinction matters because two syntactically different implementations (a loop versus a list comprehension, different variable names, a completely different algorithm) can both be perfectly correct, while a completion that reads as "almost identical" to a reference solution but mishandles one edge case is simply wrong. You cannot grade this by comparing text; you have to execute it.
The aggregation metric HumanEval reports is pass@k: the probability that at least one of k independently sampled completions for a problem passes all tests, averaged over all 164 problems. The naive way a student might compute this — generate exactly k completions per problem and check whether any pass — is a common misconception, and the Codex paper explicitly warns against it. That estimator is biased and high-variance: at low pass rates, which specific k samples you happened to draw dominates the result, so the same model can score very differently across two identical runs. The paper's actual method is to sample n completions per problem, with n ≫ k, count how many of those n pass (call it c), and then compute pass@k analytically for any k ≤ n using a combinatorial estimator, without resampling:
from math import comb
def pass_at_k(n: int, c: int, k: int) -> float:
"""Unbiased estimator, Chen et al. (2021), Eq. 1.
n = samples drawn per problem, c = samples among them
that pass every hidden unit test, k = the k in pass@k."""
if n - c < k:
return 1.0
return 1.0 - comb(n - c, k) / comb(n, k)
n, c = 10, 4
for k in (1, 5, 10):
print(f"pass@{k} = {pass_at_k(n, c, k):.4f}")
The logic: pick k of the n samples at random without replacement. The probability that none of them pass is the probability of drawing k items entirely from the (n − c) failing samples, which is C(n−c, k) / C(n, k) — the number of ways to choose k failures divided by the number of ways to choose any k samples. pass@k is one minus that. With n=10 and c=4, this code prints pass@1 = 0.4000, pass@5 = 0.9762, pass@10 = 1.0000. The first value is a sanity check: pass@1 must equal the raw success rate c/n = 4/10 = 0.40, and it does. pass@10 is a second sanity check: if k equals n, all samples are "drawn," so pass@10 must be 1.0 whenever c ≥ 1, which it is. The estimator's real value shows at pass@5: instead of resampling five completions repeatedly and averaging noisy 0/1 outcomes, one fixed batch of 10 completions yields an exact, reproducible 0.9762 for every possible choice of 5-sample subset, because the formula sums over all C(10,5) = 252 possible subsets analytically rather than sampling a handful of them.
This estimator also explains why pass@k saturates as k approaches n, and why that saturation can mask a real capability drop. Suppose a team later discovers that 3 of the original 12 passing completions in a larger n=20 batch only passed because a hidden test had a bug (it accepted any output) — the corrected c becomes 9, and pass@1 falls from 12/20 = 0.60 to 9/20 = 0.45, and pass@5 falls from 0.9964 to 0.9702 — both real, visible drops. But pass@15 on that same corrected batch stays exactly 1.0, because with only n − c = 11 samples now failing, any 15 samples drawn from 20 must include at least 15 − 11 = 4 passers by pigeonhole, regardless of how bad c gets, as long as c ≥ 4. High-k pass rates are structurally less sensitive to true quality once k gets close to n; this is precisely why reported benchmarks pick k values matched to the realistic use case — pass@1 approximates "does the one completion a developer accepts on the first try work," which is the number that actually predicts deployment experience, while a saturated pass@100 mostly measures whether the model can produce a correct answer at all given unlimited tries, a much weaker claim.
HELM: making the blind spots visible instead of hiding them
MMLU and HumanEval each fix all four design choices to one task type and report one number. HELM (Holistic Evaluation of Language Models), introduced by Liang et al. at Stanford's Center for Research on Foundation Models in 2022, starts from a different premise: no single scenario-metric pair can characterize a model, and worse, papers that each report a different ad hoc slice make models incomparable, because a strong number on one paper's chosen metric says nothing about a dimension that paper never measured. HELM's structural fix is to define an explicit matrix — scenarios (task, domain, and demographic combinations: question answering, summarization, toxicity-sensitive generation, across multiple languages and subject populations) crossed with metrics that are not just accuracy:
- Accuracy — task-appropriate correctness, as in MMLU or HumanEval.
- Calibration — does the model's stated confidence match its actual correctness rate?
- Robustness — does performance survive small input perturbations (typos, paraphrases, code-mix)?
- Fairness — does accuracy hold steady across demographic subgroups within a scenario?
- Bias and toxicity — do generations encode stereotypes or produce harmful content unprompted?
- Efficiency — inference cost in FLOPs, latency, and energy for a given quality level.
Every model is run on every applicable (scenario, metric) cell, and the raw prompts and completions are published, not just the aggregate numbers. The point of the matrix is not that every cell gets filled — many don't, because not every metric applies to every scenario — but that the gaps become visible as gaps. If a model has never been evaluated for toxicity on medical-QA-style prompts, HELM's coverage table shows an empty cell, forcing the reader to notice "this dimension is simply unmeasured" rather than silently assuming a good accuracy score implies good behavior everywhere else. This is the direct fix for the bank's mistake in the opening scenario: Model A's 86.1% was very likely a single accuracy number on a single scenario nothing like loan-query support, with zero measured robustness to code-mixed input and zero measured calibration on ambiguous queries — both of which turned out to be exactly where it failed in production.
Red-teaming: adversarial evaluation that benchmarks cannot do
MMLU, HumanEval, and HELM all share one property: the test set is fixed in advance and sampled once. That is precisely their weakness against a specific failure mode — a sufficiently capable or sufficiently motivated user does not sample uniformly from the benchmark distribution; they search for the input that breaks the model. Red-teaming is evaluation by adversarial search rather than fixed sampling: humans or models actively try to elicit disallowed outputs (harmful instructions, PII leakage, jailbroken refusal bypasses) and the metric reported is attack success rate — the fraction of adversarial attempts that succeed, tracked per harm category.
Ganguli et al. (Anthropic, 2022) ran large-scale manual red-teaming: crowdworkers were paid to have adversarial conversations with a model and rate how successfully they elicited harmful output, producing tens of thousands of labeled attack attempts and, crucially, showing how attack success rate changes across model scale and across RLHF fine-tuning stages — the same measurement discipline as a benchmark, applied to an adaptive rather than a fixed distribution. Perez et al. (DeepMind, 2022) automated the search: one language model is prompted to generate candidate adversarial test cases at scale, a target model responds, and a classifier scores whether the response is harmful, replacing expensive human labor with a model-generates-model-evaluates loop that can run orders of magnitude more attempts than manual red-teaming.
Attack success rate resists the contamination problem that plagues static benchmarks (test questions leaking into a later model's pretraining data, inflating scores without real capability gain) because new adversarial prompts are generated fresh at evaluation time rather than reused from a fixed public file. But it has its own instability: a jailbreak technique that works today gets patched once discovered, so the same nominal "attack success rate" metric measured six months apart is not comparing the same underlying vulnerability surface — the attacker population, the known techniques in circulation, and the model's patches have all moved. Attack success rate is a snapshot of an ongoing arms race, not a fixed property of the model, which is why it must be tracked as a time series and re-measured continuously, feeding back into further RLHF and safety fine-tuning rather than being reported once and considered settled.
The pass@k pipeline, traced
Why a leaderboard rank is not a deployment decision
Every mechanism above is a special case of the same warning: once a number becomes the target that determines funding, publication, or a procurement decision, pressure builds to raise that specific number by any means available — including means that do not raise the underlying capability the number was meant to proxy. Contamination (benchmark questions leaking into pretraining data) inflates MMLU or HumanEval-style scores without new capability. Fine-tuning specifically toward a benchmark's exact answer format inflates accuracy without improving robustness on paraphrased versions of the same questions — precisely the gap HELM's robustness metric is designed to expose. And a model can be heavily red-teamed and safety-tuned against known attack patterns while remaining vulnerable to attack patterns nobody has generated yet, which is why attack success rate is reported as an evolving series, never a settled score. The bank's mistake was treating one accuracy number, on one unstated scenario, as sufficient evidence for a decision that actually depended on robustness to code-mixed input and calibration on ambiguous queries — two dimensions that number never measured at all.
Active recall
Q1. A team generates n = 20 completions per HumanEval problem; c = 12 pass all hidden tests. Compute pass@1 and pass@5 using the unbiased estimator, showing every step.
Q2. A grader reports MMLU accuracy by pooling all 57 subjects into one accuracy number instead of averaging per-subject accuracies. Which of the four benchmark design choices (task distribution, protocol, scoring function, aggregation rule) does this change, and construct two models where the ranking flips between the pooled and the per-subject-averaged score.
Q3. Explain why "the model scored 91% on a public benchmark, therefore it will perform well on our production task" is not a valid inference. Name two distinct failure modes.
Q4. A safety team compares a chatbot's red-teaming attack success rate measured today against the same model's attack success rate measured six months ago. Why is the raw number not directly comparable even though the model itself hasn't changed?
Q5 (ripple). Reusing Q1's setup (n = 20, c = 12): the team discovers 3 of the 12 "passing" completions only passed because a hidden test had a bug that accepted any output — they were not truly correct. Recompute the corrected c, then recompute pass@1, pass@5, and pass@15. Does pass@15 change? Explain why or why not, tracing the full effect rather than assuming every pass@k value moves the same way.
Q6. In HELM's scenario-by-metric matrix, a model is evaluated for accuracy on medical question-answering but never evaluated for toxicity or robustness on that same scenario. What does HELM's design make visible here that a single-leaderboard paper reporting only the accuracy number would hide?
A1. pass@1 = 1 − C(8,1)/C(20,1) = 1 − 8/20 = 1 − 0.40 = 0.60. Sanity check: this must equal c/n = 12/20 = 0.60, and it does. pass@5 = 1 − C(8,5)/C(20,5). C(20,5) = 15,504 and C(8,5) = 56, so pass@5 = 1 − 56/15,504 = 1 − 0.00361 = 0.9964.
A2. This changes the aggregation rule only — the task distribution, protocol, and scoring function are unchanged, since each individual question is still scored exactly the same way. Using the three-subject setup from the chapter (Elementary Math n=200, Abstract Algebra n=100, Formal Logic n=50): Model A scores 195/200, 5/100, 3/50 — pooled accuracy 203/350 = 58.0%, per-subject average (97.5% + 5% + 6%)/3 = 36.2%. Model B scores 100/200, 60/100, 40/50 — pooled accuracy 200/350 = 57.1%, per-subject average (50% + 60% + 80%)/3 = 63.3%. Pooled accuracy ranks A above B (58.0% vs 57.1%); per-subject averaging ranks B far above A (63.3% vs 36.2%). The ranking flips purely from the aggregation choice, because A's advantage comes entirely from dominating the largest subject while being near-random on the two smaller ones — exactly the pattern macro-averaging exists to penalize.
A3. First, distribution mismatch: the benchmark's task distribution (e.g., MMLU's exam questions, or a generic public QA set) may share almost no structural features with the production task (e.g., ambiguous, code-mixed, domain-specific customer queries), so high accuracy on one says nothing about the other — this is the HELM insight that accuracy is only one cell in a much larger matrix, and an unmeasured cell (robustness to code-mix, calibration on ambiguous input) can fail even when the measured cell is strong. Second, contamination or narrow overfitting: the 91% may reflect the benchmark's exact question format having leaked into training data, or fine-tuning specifically toward that benchmark's answer style, inflating the number without a corresponding gain in general capability — Goodhart's law, where optimizing the proxy stops tracking the target.
A4. Attack success rate measures an adversarial equilibrium, not a fixed model property: the "known attack techniques in circulation" six months ago are different from today's (new jailbreak methods get discovered and shared publicly), and the model itself has likely been patched or re-fine-tuned against the attacks known at each point in time. A stable or lower attack success rate today could mean the model genuinely got safer, or it could mean this month's attackers happened to try techniques the model was already patched against — the metric is a snapshot of a moving arms race between attackers and defenders, so a raw comparison across time conflates model change with attacker-population change.
A5. Corrected c = 12 − 3 = 9 (n stays 20). pass@1 = 1 − C(11,1)/C(20,1) = 1 − 11/20 = 0.45 (down from 0.60). pass@5 = 1 − C(11,5)/C(20,5) = 1 − 462/15,504 = 1 − 0.0298 = 0.9702 (down from 0.9964). pass@15: here n − c = 11 and k = 15 > 11, so it is mathematically impossible to draw 15 samples from 20 without including at least one of the 9 passers (by pigeonhole, any 15-sample draw excludes at most 11 items, and there are only 11 failing items to exclude) — so C(11,15) is /zero and pass@15 = 1.0, completely unchanged by the correction. The ripple is uneven: pass@1 and pass@5 drop meaningfully because they are sensitive to the true pass rate, while pass@15 is saturated and insensitive to a correction that removed a quarter of the "passing" samples. This is exactly why low-k pass rates, not high-k ones, are the metric that tracks real quality.
A6. A traditional paper reporting only "91% accuracy on medical QA" gives no signal about whether the model produces toxic or biased content when discussing sensitive medical topics, or whether that 91% survives a paraphrased or code-mixed version of the same questions — the reader has no way to distinguish "this dimension is fine" from "this dimension was never checked." HELM's explicit scenario-by-metric matrix makes an unmeasured cell visibly empty rather than silently absent, forcing anyone reading the results to see the coverage gap as a gap, not to assume good behavior on dimensions the accuracy number never touched — which is the exact blind spot that caused the opening scenario's deployment failure.
Think About It
Think about this: How would you explain ai evaluation metrics: mmlu, humaneval, helm, benchmarks, and red-teaming to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind ai evaluation metrics: mmlu, humaneval, helm, benchmarks, and red-teaming, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.