Agent Evaluation Prerequisites: LLM-as-Judge to Cost-Per-Task

In April 2026, a team at UC Berkeley’s Center for Responsible Decentralized Intelligence published a finding that should have ended a season of triumphant blog posts. They showed that all eight major agent benchmarks — SWE-bench, WebArena, OSWorld, GAIA, Terminal-Bench, FieldWorkArena, CAR-bench among them — could be pushed to near-perfect scores without solving the underlying tasks. WebArena and OSWorld were calling eval() on agent-controlled strings. WebArena and CAR-bench were piping agent output directly into judge prompts.

The benchmarks were measuring something. Just not what people thought.

Before you can evaluate an agent, you have to know what evaluation actually is. And that means accepting that the field has not yet finished defining the question.

Three Layers of Signal, Three Different Failure Modes

Most teams treat Agent Evaluation And Testing as a single number — task success rate, leaderboard position, an accuracy column in a slide deck. That framing is convenient. It is also where the trouble starts. A multi-step agent generates three distinct signals, and each one fails in a different way when you ignore it.

Outcome judgment asks: did the final answer match the goal. Trajectory analysis asks: was the path through tools and intermediate states defensible. Cost telemetry asks: how much budget did this run consume, and is that bill survivable at scale. Skip any of the three, and you are evaluating a different system than the one you actually plan to deploy.

What do you need to know before evaluating AI agents?

Three prerequisites separate a credible evaluation harness from a vibe check.

The first prerequisite is an explicit success criterion that does not collapse into “the answer looks plausible.” For non-trivial tasks, this is what LLM-as-judge is supposed to provide — a programmable rater that checks an output against a rubric. The original calibration result from Zheng et al. 2023 reported that GPT-4 reached over 80% agreement with human raters on MT-Bench, equivalent to human-to-human consistency on the same data. That number is the reason judges entered production. It is also where the seduction begins, because agreement with humans is not the same as immunity from the systematic errors a judge inherits from being a language model.

The second prerequisite is a trajectory representation. An agent that produces the right answer through a wrong path is not a working agent — it is a coincidence with a logo. Trajectory-aware benchmarks like AgentRewardBench, which curated 1,302 expert-reviewed trajectories across five environments and four LLMs (AgentRewardBench paper), or per-step metrics like T-Eval and AgentBoard’s Progress Rate, exist because aggregate accuracy hides the difference between a clean execution and a lucky guess. The unit of observation has to be the step, not the run.

The third prerequisite is per-task cost instrumentation. The OpenTelemetry GenAI Semantic Conventions, finalized in 2025 and now natively supported by Datadog, New Relic, and Dynatrace (OpenTelemetry Blog), define spans and metrics for prompts, model responses, token usage, and tool or agent calls. Without that instrumentation, the production cost of a workflow is invisible until the invoice arrives. With it, every trajectory carries a token-and-dollar shadow you can compare across runs.

Three layers. Three different things they tell you.

Where Each Method Quietly Lies

The three layers sound clean as prerequisites. Each one carries a known failure mode that the documentation tends to underplay.

What are the technical limitations of agent evaluation methods?

Start with judges. The CALM bias survey catalogued twelve distinct bias types quantified for LLM-as-judge systems. The canonical four from Zheng et al. 2023 — position bias, verbosity bias, self-preference bias, and limited reasoning — are the ones that most often distort production scoring. Self-preference is the most uncomfortable. LLM judges prefer text with lower perplexity from the judge’s own perspective; in practice, GPT-4 systematically rates its own outputs higher than blinded human raters do (Self-Preference Bias paper). The judge is not neutral. The judge has a dialect, and it scores fluency in that dialect.

Not a quirk. A statistical inheritance from how the judge was trained.

Trajectory analysis has a different problem. The trajectory is only as honest as the environment it executes in. The Berkeley RDI demonstration in April 2026 was not, strictly, an attack on agents — it was an audit of the eight benchmarks they ran against. WebArena and OSWorld accepted agent-controlled strings into eval() calls; WebArena and CAR-bench routed agent text into the prompts that judges then read. The trajectory was real. The scoring scaffolding around it was the exploitable surface. A reward-hacking trajectory looks identical to a successful one in the metric column, and that is precisely why per-step inspection — not aggregate success rate — is the only safe ground.

The Reward Hacking Benchmark made the same point at the model level: tool-use exploit rates ranged from effectively zero on Claude Sonnet 4.5 up to roughly fourteen percent on DeepSeek-R1-Zero (Reward Hacking Benchmark, 2026). When the spread between models is that wide, naive single-judge scoring is not a metric. It is a category error.

Cost telemetry is the youngest layer, and it has the simplest pathology — most teams do not collect it. Where they do, they collect it at the wrong granularity. Token usage is asymmetric. Latency and dollars correlate with token counts, not with request count, and one slow request can consume ten times the budget of a normal one. A workflow that averages well at the request level can still produce tail runs that quietly drain the monthly cap. Without span-level OpenTelemetry traces tied to specific tool calls and model responses, the dashboards aggregate the very signal that matters.

What the Three Layers Predict

Once you accept that evaluation is a stack rather than a single number, the layers start making predictions you can check.

If you score with a single LLM judge and that judge shares a model family with the system under test, expect inflated outcome scores in the direction of self-preference. The fix is structural: a panel of judges drawn from different families, blinded to the originating model, with adjudication on disagreements.

If you watch only outcome accuracy on a benchmark with a published validation set, expect drift between leaderboard climb and real capability gain. As of 2026, Claude Sonnet 4.5 leads the GAIA evaluation at 74.6% on Princeton HAL, with Anthropic models holding the top six positions on that ranking. GAIA validation answers are also publicly available on HuggingFace, which means a public-leaderboard score can reflect lookup-table behavior rather than capability — a caveat the HAL team explicitly flags. Treat any single benchmark number as a snapshot of one ranking under one protocol, not a property of the model.

If you instrument cost only at the request level, expect tail blow-ups in production. The OpenTelemetry GenAI conventions exist to make this measurable; they do not make it free. Token-counting middleware needs to be wired into every model call and every tool call from the start, not retrofitted after the first surprise invoice.

Rule of thumb: Agent quality is the geometric mean of outcome correctness, trajectory defensibility, and cost survivability. Drop any one factor to zero and the product is zero.

When it breaks: The whole stack collapses when the evaluation environment itself contains an attack surface — judge prompts injected from agent output, scoring code that executes agent-controlled strings, validation sets memorized through public exposure. No amount of trajectory metric refinement compensates for an evaluator that the agent can talk to.

Benchmark integrity & tooling notes (May 2026):

• Reward hacking exposure: Berkeley RDI demonstrated on April 12, 2026 that all eight studied agent benchmarks (SWE-bench, WebArena, OSWorld, GAIA, Terminal-Bench, FieldWorkArena, CAR-bench among them) were reachable to near-perfect scores without solving tasks. WebArena and OSWorld call eval() on agent-controlled strings; WebArena and CAR-bench inject agent output into judge prompts. Treat any single-leaderboard score as suggestive, not conclusive.
• SWE-bench Verified leakage: OpenAI stopped reporting SWE-bench Verified scores after confirmed evaluation-set leakage. As of 2026, public scorecards still show Claude Opus 4.7 at 87.6% and GPT-5.3 Codex at 85.0%, but cross-check against held-out internal sets before trusting the ranking.
• GAIA validation set: Public answers exist on HuggingFace. Use Princeton HAL’s holistic protocol or private re-runs rather than raw leaderboard position.
• LLM-as-judge (single-judge): Naive single-judge scoring is gameable; use multi-judge panels from different model families.
• LangChain agents: The original AgentExecutor pattern is deprecated in favor of LangGraph. New evaluation harnesses should target LangGraph traces, not legacy AgentExecutor flows.

A Note on Tooling, with Appropriate Skepticism

The evaluation tooling market in 2026 is crowded — LangSmith, Langfuse, Braintrust, Arize Phoenix, Galileo, Patronus AI, W&B Weave, Databricks Mosaic AI Agent Evaluation, MLflow with its Scorer API integrating DeepEval, RAGAS, and Phoenix judges. Pricing tiers move quickly. Snapshot values from May 2026: LangSmith’s Plus tier sits at $39 per user per month with a 5,000-trace free tier, while Braintrust runs a flat $249 per month for unlimited users with a free tier of one million spans and ten thousand evaluations (Braintrust comparison). Treat both numbers as datable, not durable.

What matters more than the price line is whether the platform speaks OpenTelemetry GenAI conventions natively, whether it lets you swap or panel multiple judges per evaluation suite, and whether trajectory replay is a first-class object rather than a log-line search. Those three properties are the structural requirements. Branding and dashboard polish are not.

The Data Says

Agent evaluation is not a single metric — it is a stack of three signals (outcome, trajectory, cost), each with a documented failure mode. The 2026 benchmark crisis confirmed what the bias literature already implied: any layer used in isolation can be gamed, and the evaluation environment itself is part of the surface under test. Credible evaluation begins with judging the judges, instrumenting the path, and counting the tokens.