Agentic AI Evaluation Frameworks: Testing Autonomous Systems

The refund bot that graded itself wrong

Picture an autonomous agent deployed by an IRCTC-style ticketing platform to handle refund disputes. Call it Nyaya. A user's train is cancelled, they file a claim, and Nyaya is given three tools: verify_pnr, check_balance, and issue_refund. It runs unsupervised, end to end, thousands of times a day. Before this system goes live, someone has to answer a question that sounds simple and is not: how do you test it?

If Nyaya were a plain text classifier — "is this a refund request, yes or no" — you would build a labelled test set and report accuracy. That recipe does not survive contact with an agent. Nyaya does not emit one label. It takes an action, observes the result, takes another action conditioned on that result, and keeps going until it reaches a terminal state. Two runs on the same ticket can legitimately follow different paths and still both be correct. A run can also follow a path that looks fluent and confident at every individual step while producing an outcome that is quietly wrong, or worse, unsafe — refunding a ticket it had no authority to refund. Grading that requires watching the whole trajectory, checking the state it leaves the world in, and doing this cheaply enough to run thousands of times. That is what an agentic evaluation framework is built to do, and it is the subject of this chapter.

Why a trajectory is not a sentence

Formally, an agent interacting with an environment produces a trajectory τ = (s₀, a₀, o₀, s₁, a₁, o₁, …, s_T), where sₜ is the environment's state at turn t, aₜ is the action the agent's policy π chooses (a tool call, a command, a message), and oₜ is the observation the environment returns. A single-turn language model benchmark — MMLU-style multiple choice, HumanEval's one function per problem — collapses this entire object into a single input-output pair: one prompt, one completion, one comparison against a reference. An agentic benchmark cannot do that collapse, because the thing under test is π itself, evaluated over the full distribution of states it can reach, not over one fixed prompt.

This has three concrete consequences that every agentic evaluation framework has to engineer around. First, grading must be a function of the final environment state (or the full trajectory), not of the agent's last utterance — a correct outcome can be reached through many different action sequences, and an incorrect one can be dressed up in perfectly plausible-sounding language. Second, because tool calls, retrieval, and sampling all introduce stochasticity, the same task run twice can diverge, so a framework must support repeated, independent rollouts of the same task rather than one deterministic pass. Third, because a real deployment cares as much about how an agent fails as whether it fails — did it politely decline, or did it hallucinate a refund and corrupt a ledger — grading needs multiple lenses (automated checks, a judge model, a human), not one scalar.

Anatomy of an evaluation harness

Stripped to its essential parts, every modern agentic evaluation framework — whether it is SWE-bench running a coding agent against a GitHub repository, or WebArena running a browsing agent against a simulated shopping site — assembles the same five pieces: a sandboxed task/environment that the agent cannot escape or corrupt, the agent under test running its observe-reason-act loop against that sandbox, a trajectory log that records every state, action, and observation for later inspection, a set of evaluators that turn a logged trajectory into a score, and an aggregator that rolls per-episode scores into a scorecard across many rollouts and many tasks. The diagram below traces that pipeline for Nyaya.

Notice that the environment sits between the agent and the grader: the harness never lets the agent see or touch the scoring code, and the scoring code only ever reads the trajectory log, never the agent's internal reasoning trace. That separation matters for the same reason exam answer sheets are graded by someone who did not write the question paper — an agent that can inspect its own grading function can learn to satisfy the grader instead of solving the task, a failure mode called reward hacking or specification gaming (Amodei et al., 2016, "Concrete Problems in AI Safety"). Keeping the environment sandboxed and the evaluator downstream-only is not a coding convenience; it is the control that prevents Nyaya from, say, discovering it can call an internal debug endpoint that marks every ticket as refunded without actually touching the balance.

Four real frameworks, four different graders

The "environment, log, evaluator" skeleton is general, but what counts as a correct final state is completely domain-specific, which is why production-grade agent benchmarks each build their own grading logic rather than sharing one metric.

The pattern across all four: none of them grade the agent's prose. SWE-bench does not read the agent's explanation of its fix, it runs the tests. WebArena does not read the agent's final chat message, it queries the browser's DOM and the shop's backend. Grading an agent means writing a checker that understands the task's environment, not a checker that understands English.

Worked example: from per-step reliability to pass@k

Suppose an IRCTC-booking version of the agent must complete six sequential, dependent tool calls to succeed: log in, search trains, select a train, select a berth, apply a concession code, and pay. Historical logs show each individual step succeeds independently with probability p = 0.90 — a respectable number for any one call. Assuming independence between steps, what is the probability the entire six-step trajectory completes successfully?

P(success) = p⁶ = 0.9⁶. Computing it: 0.9² = 0.81, 0.9³ = 0.9² × 0.9 = 0.81 × 0.9 = 0.729, and 0.9⁶ = (0.9³)² = 0.729² = 0.729 × 0.729. Breaking that multiplication down: 0.729 × 0.7 = 0.5103, and 0.729 × 0.029 = 0.021141, so 0.729² = 0.5103 + 0.021141 = 0.531441. A "90%-reliable" agent, measured step by step, only completes the full task about 53.1% of the time end to end. This is precisely why an evaluation framework that only reports a step-level accuracy number — the kind you would get from testing each tool call in isolation — dramatically overstates how often the agent actually finishes the job, and why trajectory-level success rate has to be reported as its own, separately measured number.

Trajectory-level success is one axis; the other is how many independent attempts you are willing to credit before calling the task solved. Chen et al. (2021), introducing Codex, defined the now-standard unbiased estimator for this: given n independent rollouts of a task of which c succeed, the probability that at least one of k randomly chosen rollouts (drawn without replacement from the n) succeeds is

pass@k = 1 − C(n−c, k) / C(n, k)

where C(a,b) is "a choose b". Take n = 10 rollouts of the same booking task with c = 6 successes. For k = 3: C(n−c, k) = C(4, 3) = 4 (choosing 3 of the 4 failing rollouts), and C(n, k) = C(10, 3) = 120 (choosing any 3 of the 10). So pass@3 = 1 − 4/120 = 1 − 0.03333… = 0.96667, roughly 96.7%. Even though a single rollout only succeeds 60% of the time, an agent that gets three independent attempts — with a supervisor accepting any one of them — clears the task almost 97% of the time. Framework designers report both numbers side by side precisely because they answer different deployment questions: pass@1 tells you what happens with no retries; pass@k tells you what happens if you can afford k of them, and how quickly that safety margin degrades as k shrinks.

Building a minimal harness and tracing it

The five-part anatomy from the diagram compiles down to surprisingly little code. Below is a deterministic, fully self-contained harness for the three-step refund flow (verify_pnr → check_balance → issue_refund) that you can trace by hand.

from math import comb

class RefundEnv:
    """Deterministic mock ticket-refund environment."""
    def __init__(self, seed):
        self.balance_ok = seed < 6   # seeds 0-5: refundable, 6-9: not
        self.state = "start"

    def step(self, action):
        if self.state == "start" and action == "verify_pnr":
            self.state = "verified"
            return "PNR verified", False
        if self.state == "verified" and action == "check_balance":
            self.state = "balance_checked"
            return ("balance_ok" if self.balance_ok else "balance_low"), False
        if self.state == "balance_checked" and action == "issue_refund":
            if self.balance_ok:
                self.state = "done"
                return "refund_issued", True
            self.state = "escalated"
            return "escalated_to_human", True
        return "invalid_action", True


def agent_policy(state):
    """Fixed 3-step policy: verify, then check, then refund."""
    script = {"start": "verify_pnr", "verified": "check_balance",
              "balance_checked": "issue_refund"}
    return script.get(state, "noop")


def run_episode(seed):
    env = RefundEnv(seed)
    trajectory = []
    for _ in range(5):
        action = agent_policy(env.state)
        obs, done = env.step(action)
        trajectory.append((env.state, action, obs))
        if done:
            break
    return trajectory, env.balance_ok


def evaluate_naive(trajectory):
    _, _, final_obs = trajectory[-1]
    return final_obs == "refund_issued"


def pass_at_k(n, c, k):
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)


results = [run_episode(s) for s in range(10)]
successes = sum(evaluate_naive(traj) for traj, _ in results)
print(successes, len(results))
print(round(pass_at_k(10, successes, 3), 4))

Trace run_episode(0) by hand: balance_ok = 0 < 6 = True. Turn 1: state "start" → action "verify_pnr" → state becomes "verified", observation "PNR verified", not done. Turn 2: state "verified" → action "check_balance" → state "balance_checked", observation "balance_ok" (since balance_ok is True), not done. Turn 3: state "balance_checked" → action "issue_refund" → balance_ok is True, so state "done", observation "refund_issued", done — loop breaks after exactly 3 recorded steps. evaluate_naive reads the last tuple's observation, "refund_issued", and returns True.

Now trace run_episode(6): balance_ok = 6 < 6 = False. The first two turns are identical in shape (verify, then check), except the second observation is "balance_low" instead of "balance_ok". Turn 3: action "issue_refund", balance_ok is False, so the environment sets state "escalated" and returns "escalated_to_human", done True. evaluate_naive checks the last observation against "refund_issued" — it is "escalated_to_human" instead — and returns False.

Running the loop over seeds 0 through 9: seeds 0–5 have balance_ok True and finish with "refund_issued" (6 successes); seeds 6–9 have balance_ok False and finish with "escalated_to_human" (4 non-matches under the naive check). So successes = 6, len(results) = 10, and the script prints 6 10, then pass_at_k(10, 6, 3): n−c = 4 ≥ k = 3, so it returns 1 − comb(4,3)/comb(10,3) = 1 − 4/120 = 0.96667, rounded to 0.9667 — matching the hand derivation above exactly.

The misconception: "success" is not "matches the expected final action"

The natural assumption, carried over from grading a classifier or a QA model, is that testing an agent means comparing its last output to one expected answer — here, "did the agent's final action equal issue_refund". The traced run for seed 6 shows exactly why that assumption breaks: the ground truth for that ticket is balance_ok = False, meaning the correct behavior is to decline and escalate, not to refund. Nyaya did the right thing — it protected the platform from paying out a refund it was not authorized to give — and evaluate_naive still scored it as a failure, because the checker only ever looks for one fixed string.

The fix is not a smarter string match; it is a checker that is a function of both the agent's action and the environment's initial condition: success = (final_obs == "refund_issued" and balance_ok) or (final_obs == "escalated_to_human" and not balance_ok). Applied to the same ten trajectories, this state-conditioned checker scores all ten episodes as correct — the naive version was undercounting the agent by 40 percentage points, entirely from mis-grading, with zero change to the agent's behavior. This is exactly why WebArena, AgentBench, and SWE-bench each write a bespoke correctness predicate per task instance rather than sharing one global comparison rule: the "expected output" of an agentic task is not a fixed string, it is a condition on the state the environment ends up in given the state it started in.

Metrics beyond success, and the judge that grades itself

A scorecard that reports only pass@1 is already an improvement over a single string-match test, but production frameworks track several axes side by side because a deployment decision depends on all of them together. Step efficiency — how many turns and tool calls the agent needed relative to a minimal solution — matters because every extra tool call costs latency and money and is another chance to touch something it shouldn't. Safety-violation rate — how often the agent takes an irreversible or unauthorized action, like issuing a refund above its mandate — matters independently of whether the task was ultimately completed, because a single catastrophic action can outweigh a thousand successful ones. Cost per task, in tokens and wall-clock seconds, determines whether a given success rate is even economical to deploy at scale. None of these show up if the harness reports one pass/fail number per episode; they require the trajectory log to be kept, not discarded after grading.

Where a task's correct output genuinely cannot be reduced to a programmatic assertion — grading the quality of a written explanation, say, rather than whether a refund fired — frameworks fall back to an LLM-as-judge: a second, usually stronger model scores the trajectory against a rubric (Zheng et al., 2023). This buys flexibility at the cost of a new failure mode: judge models measurably favor longer answers, favor whichever response is shown first in a pairwise comparison, and can be gamed by an agent that has learned what judge-pleasing phrasing looks like, independent of whether the underlying action was correct. That is precisely why the harness diagram routes trajectories through three separate channels rather than trusting the judge alone — a programmatic checker anchors the score in something the agent cannot rhetorically talk its way past, and human review exists specifically to catch the cases where the judge and the checker disagree, or where neither one can express what "safe" means for a given ticket. The same trajectory-versus-string-match problem recurs at the frontier: organizations evaluating whether an agent can autonomously carry out long, open-ended tasks — the kind of work METR's public autonomy evaluations study — cannot grade by matching a final message either; they have to instrument the entire multi-hour trajectory and check, step by step, whether the agent stayed within its intended scope.

Active recall

Attempt each question before reading its answer.

Q1. In one sentence, why can't a benchmark that scores single LLM responses (like an MMLU-style accuracy test) be reused unmodified to grade an autonomous agent?

Q2. An IRCTC-booking agent needs 6 sequential, dependent tool calls to succeed, each independently reliable at p = 0.90. Assuming independence, what is the end-to-end trajectory success probability? Show the arithmetic.

Q3. Using pass@k = 1 − C(n−c,k)/C(n,k), with n = 10 rollouts and c = 6 successes, compute pass@3.

Q4. Show that pass@1 computed from the same formula, for n = 10 and c = 6, equals the naive success rate c/n.

Q5 (ripple). Suppose RefundEnv's script changes to balance_ok = seed < 8 instead of seed < 6, with run_episode, agent_policy, and the state-conditioned evaluator unchanged. Recompute c out of n = 10, then pass@3, and state precisely what changed and what did not.

Q6. For seed = 6, the trajectory ends with final_obs == "escalated_to_human" and evaluate_naive marks it a failure. Explain why this grading is wrong and state the fix.

A1. An agent's output is a multi-step trajectory of actions against a changing environment, not one string — what must be graded is whether that sequence reaches an acceptable environment state (and how safely/efficiently), which requires reading environment state, not comparing text.

A2. 0.9⁶. 0.9² = 0.81, 0.9³ = 0.81 × 0.9 = 0.729, 0.9⁶ = 0.729² = 0.531441 ≈ 53.1%. A per-step-reliable agent still completes barely half of a 6-step task end to end, which is why frameworks must report trajectory-level success separately from step-level accuracy.

A3. C(4,3) = 4, C(10,3) = 120. pass@3 = 1 − 4/120 = 1 − 0.0333 = 0.9667 (96.7%).

A4. pass@1 = 1 − C(4,1)/C(10,1) = 1 − 4/10 = 0.6, and c/n = 6/10 = 0.6. They match because choosing k = 1 sample and asking "did it pass" is definitionally the empirical success rate — a useful self-check whenever you implement this estimator.

A5. Under the state-conditioned evaluator — the one the question specifies stays unchanged — nothing moves: c = 10 out of n = 10, at threshold 8 exactly as at threshold 6, so pass@3 = 1 − C(0,3)/C(10,3) = 1 − 0/120 = 1.0 (100%), also unchanged. The reason is structural, not numerical: agent_policy always attempts verify_pnr → check_balance → issue_refund regardless of seed, and RefundEnv.step deterministically returns "refund_issued" exactly when balance_ok is True and "escalated_to_human" exactly when it is False. The predicate success = (final_obs == "refund_issued" and balance_ok) or (final_obs == "escalated_to_human" and not balance_ok) is therefore satisfied on every seed no matter where the balance_ok threshold sits — it credits both a correct refund and a correct escalation, and this policy always produces one or the other. What changed: nothing, under the specified evaluator — c stays 10, pass@3 stays 1.0, matching what the chapter's own "misconception" section already established for threshold = 6. What did not change: the same, for the same reason — a policy that always acts correctly scores perfectly under a state-conditioned checker regardless of how the underlying task distribution shifts. (Only the naive evaluator — the fixed-string match this chapter's misconception section just finished repudiating — would show any movement, its count shifting from 6/10 to 8/10; that movement is precisely the naive-matching fragility the chapter argues against, not a legitimate ripple effect of the threshold change.)

A6. The checker treats "success" as matching the single fixed string "refund_issued", but the ground truth for seed 6 is balance_ok = False, meaning the correct action is to decline and escalate. The agent did exactly that — correct, safety-preserving behavior — and was still marked a failure because the checker never consulted the environment's actual condition. The fix is a state-conditioned predicate: success = (final_obs == "refund_issued" and balance_ok) or (final_obs == "escalated_to_human" and not balance_ok), scoring the action jointly with the environment's ground truth rather than against one hardcoded string — the same approach WebArena and AgentBench use, writing a distinct correctness predicate for each task instance.

Think About It

Think about this: How would you explain agentic ai evaluation frameworks: testing autonomous systems to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where agentic ai evaluation frameworks: testing autonomous systems is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting agentic ai evaluation frameworks: testing autonomous systems to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind agentic ai evaluation frameworks: testing autonomous systems, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.