agent-evaluation
How to tell whether an Agent is actually doing its job. Per tw93 in 2026-04-27-agent-principles-architecture-engineering: many teams put eval at the end of the roadmap and end up with a pile of unexplainable noise — prompt changed, score moved, no idea why. Eval needs to be early and structural, because it is the feedback signal component of the harness. Shunyu Yao‘s The Second Half elevates this further: at the field level, the gap between “AI saturates exams” and “the world hasn’t changed by GDP measure” — the utility problem — is fundamentally an evaluation-design problem, not a model-capability problem.
Scope note. This page is about runtime / production evaluation of an Agent — Pass@k, Pass^k, transcript-vs-outcome, three grader types. Its training-side cousin is eval-grader-reward, which covers eval / grader / reward as the training-time loop that feeds gradient updates (ORM vs. PRM, verified rewards, reward-hacking as a training-design concern). The two pages share principles but operate at different points in the pipeline.
Why Agent eval is structurally harder than single-turn
Single-turn eval is prompt → model → response → grade. Agent eval is task + tools + environment → multi-step execution → updated environment → grade against environment, not just text. The structural complexity goes up at least one level:
| Concept | What it is |
|---|---|
| task | The unit being measured |
| trial | One run of one task |
| grader | The thing that decides pass/fail |
| transcript | The full execution record (what the Agent said and did) |
| outcome | The environment’s actual final state |
| agent harness | The Agent runtime under test (harness) |
| evaluation harness | The infrastructure that runs tasks, scores them, aggregates |
| evaluation suite | A collection of tasks |
A common confusion is grading transcript-only (“the Agent said it shipped”) versus outcome (“the DB row exists”). Both layers need coverage.
Pass@k vs. Pass^k
Two numerically similar but semantically opposite metrics — do not mix them:
| Metric | Means | Use for |
|---|---|---|
| Pass@k | At least one of k attempts is correct | Capability ceiling — “can the Agent do this at all?” Run on capability breakthroughs. |
| Pass^k | All k attempts are correct | Regression — “did this change break something that worked?” Run on every release. |
Mixing them produces predictable failures: too-loose regression misses real breakage; too-strict capability-eval surfaces noise from prompt sensitivity. The same suite shouldn’t carry both.
Three grader types
| Type | Examples | Determinism | Use for |
|---|---|---|---|
| Code grader | Exact-match, unit test pass/fail, structural diff, tool-call argument checks | Highest — minimal noise from grader-design errors | Anything with a definable correct answer |
| Model grader (LLM-as-judge) | Rubric-based score, A/B-pick, ensemble vote | Mid — drifts with judge model and prompt | Semantic quality, style, reasoning chains |
| Human grader | Expert sample review, calibration labelling | High but slow | Establishing ground truth; calibrating auto-judges |
Default order: try code grader first; only fall back to model grader when no code answer exists; use human grading to anchor when model graders drift. The source’s stronger claim: read whole transcripts periodically — judge bugs typically only surface in the trace, not in the aggregate score.
Early precedent — LLM-as-judge fails in expert domains (2023)
Weng‘s 2023 survey (2026-06-04-llm-powered-autonomous-agents) already carried the canonical warning for the model-grader row above. In ChemCrow, GPT-4-as-judge rated GPT-4 ≈ a tool-augmented chemistry agent, but human experts rated the tool-augmented agent far higher: “the lack of expertise may cause LLMs not knowing its flaws and thus cannot well judge the correctness of task results.” The lesson that became “calibrate model graders against humans” was visible in 2023. The same survey’s API-Bank benchmark is an early instance of decision-level tool-use eval — scoring, at each step, whether to call an API, which API to call (retrieval), and how to plan multi-call sequences (tool-use).
Transcript ≠ outcome (the booking-Agent example)
A real example from Anthropic’s Demystifying evals for AI agents: an Opus 4.5 booking Agent finds a fare-rule loophole and lands the user a cheaper itinerary that the test designer didn’t anticipate. A pure-transcript grader (path didn’t match the script) marks this fail; an outcome grader (cheaper booking exists) marks pass.
The lesson: graders that only watch the path miss legitimate creativity, and graders that only watch the outcome miss intermediate steps that went wrong. Cover both.
Survey state of practice
From the LangChain State of Agent Engineering survey, summarized in the source:
| Question | Answer |
|---|---|
| Eval method | Offline on test sets 54.5% / Online on prod data 44.8% / Not evaluating yet 22.8% |
| Common metric | Internal human review/labelling 59.8% / LLM-as-judge 53.3% / Traditional ML 16.9% |
A quarter of teams haven’t started evaluating; human + model judges dominate; classic ML metrics are barely present.
Bootstrapping from zero
Practical recipe from the source:
- 20–50 real failures is enough to start. Pull from cases someone is already manually checking — those reflect actual usage.
- Sanity test before collecting data: if two domain experts can’t agree on whether a case is a pass or fail, the acceptance criteria isn’t written yet — write it before collecting.
- Cover positive and negative cases. Only testing “should do X” will optimize for over-action; testing “should not do X under condition Y” probes the boundary.
- Environment isolation per trial. Each run starts from a clean state — no shared cache, temp files, or DB rows. Otherwise one trial’s failure pollutes the next, and the model gets blamed for an environment bug.
- Backfill harder tasks as the suite saturates. A suite that all Agents pass has stopped measuring real ability.
”Fix the eval before fixing the Agent”
When the score drops, the first instinct is to change the Agent. The source pushes back: eval-system bugs and Agent regressions look identical from outside.
Common eval-system failures:
- Container resource caps (peak memory kills the process; eval records fail; Agent did nothing wrong) — the source includes a chart showing infra error rate ≈ score loss until resource caps lift.
- Bug in the grader (correct answers marked wrong).
- Test cases drifted from production scenarios.
- Aggregate-only view masks one task category collapsing.
Diagnosis order: check infra → check grader → check test design → only then change the Agent.
Online eval — sampling rules, not random
Running the entire eval suite on live traffic is expensive; pure-random sampling misses the interesting Traces. Source’s rule-routed sampling for the 10–20% live-eval slice:
- Negative feedback — every Trace where the user explicitly disliked the result, 100% to the queue.
- High-cost conversations — token spend over a threshold, prioritized — usually means the Agent was thrashing.
- Time-window sampling — fixed daily windows, random within, to keep coverage of normal traffic.
- Post-deploy 48 hours — full audit on every Trace to catch regressions early.
Two-layer observability
Pair human and LLM evaluation rather than running either alone:
| Layer | Job | Risk if used alone |
|---|---|---|
| Layer 1 — human sampling | Rule-routed manual review on errors / long conversations / negative feedback. Builds the calibration set. | Doesn’t scale to traffic volume |
| Layer 2 — LLM auto-eval | Full coverage of Traces; calibrated against Layer 1 | Standard drift; loses fidelity if not anchored to humans |
The source’s point: layer 1 alone misses scale, layer 2 alone drifts. Use them together.
Anti-patterns
- Mixing Pass@k and Pass^k on the same suite.
- Grading transcript only; never reading what the environment actually became.
- Aggregate scores without per-category breakdown.
- LLM-as-judge with no human calibration set.
- Treating eval failures as Agent failures by default — see “fix the eval first” above.
- Not running periodic full-Trace reads — judge bugs hide in the aggregate.
The utility problem — eval setups vs. real-world setups (Yao 2025)
Shunyu Yao’s The Second Half reframes evaluation at the field level. A working “recipe” (language pretraining + scale + reasoning-as-action) now solves any benchmark within the recipe’s reach within months — image 3 of the source shows MATH 5%→95% in 3 yrs, SWE-bench-verified 5%→80% in 1 yr, AIME 10%→95% in ~1 yr. So an incremental method that lifts a benchmark 5% gets dwarfed by the next o-series model lifting it 30% without targeting it.
But “the world hasn’t changed much by economics and GDP” — the utility problem. Yao’s diagnosis: real-world utility doesn’t track benchmark numbers because eval setups differ from real-world setups in basic ways the field has assumed away. Two specific assumptions the source calls out as load-bearing but not inevitable:
| Assumption | Standard frame | Real frame | Counter-example evals |
|---|---|---|---|
| Autonomous evaluation | task input → agent runs autonomously to completion → single reward at the end | a customer-service agent doesn’t take one super-long message and return a final response 10 minutes later — it dialogs throughout | Chatbot Arena (real humans in the loop); tau-bench (user simulator in the loop — the agent must mid-task ask the simulated user “do you want me to cancel and rebook?”) |
| I.i.d. evaluation | 500 tasks averaged independently | tasks are sequential — a Google SWE gets better at the google3 repo over time; SWE-agents don’t gain that familiarity | none mature yet; the wiki has [[agent-memory |
Yao’s compressed answer for why these held so long: “when intelligence is low, improving intelligence generally improves utility.” Both assumptions worked during the first half because every uplift translated. Now the recipe is guaranteed to lift any benchmark within those assumptions, so further uplift along the existing axes barely moves real-world utility.
Engineering implication, beyond what the runtime-eval framing above already covers: questioning the eval frame itself is now part of eval engineering. Examples in this wiki where the assumption-break is concrete:
- Sequential-tasks-with-memory breaks i.i.d. — see agent-memory (the four-memory-types frame) and long-running-agents (Initializer + Coding Agent split with Codex‘s 6+ hr runs).
- User-simulator-in-the-loop breaks autonomous — tau-bench and the Anthropic booking-Agent example above are both pointing in this direction; the latter is a step toward outcome-vs-transcript grading that survives mid-task replanning.
- Real-traffic online eval (the rule-routed sampling pattern above) is itself a partial answer: production traces are not autonomous, not i.i.d., and not constructed.
The closing prescription Yao offers — “develop novel evaluation setups for real-world utility, then solve them with the recipe or augment the recipe with new components” — generalizes the runtime “fix the eval first” rule to the field-level “fix the eval frame first”.
Evaluation as the RSI bottleneck (Weng, Jul 2026)
2026-07-07-harness-engineering-self-improvement confirms evaluation’s bottleneck status from a third angle — self-improving systems. Three additions to this page’s frame:
- Weak and fuzzy evaluators top the challenge list. Weng’s seven future challenges for harness-driven recursive-self-improvement open with exactly this: many real-world tasks lack fast, precise verifiers, and the whole evolutionary-search family only works “when evaluation is fast and fitness quantifiable” (GPU kernels, algorithm contests) — it struggles in slow/ambiguous domains. Every grader weakness on this page becomes load-bearing once an optimization loop, not just a dashboard, consumes the score.
- The evaluator must sit outside the loop it grades. When the harness itself is being optimized (self-improving-harness), an evaluator inside the loop is just another surface for reward-hacking: “the evaluator and permission control should likely sit outside the loop that evolves harness, with held-out tests, trace audits, and human review at decision points that matter.” This is the self-improvement analogue of “fix the eval before fixing the Agent” — the eval must be unreachable by the thing being fixed.
- Time-horizon evals put a number on the long-task gap. RE-Bench: agents score ~4× better than human experts at the 2-hour budget, but humans exceed agents at 8h+ (long-running-agents). This is the first benchmark family in the wiki that measures the sequential/duration axis Yao complained was missing — partial progress on the i.i.d. critique above, though within-task duration is still not cross-task memory accumulation.