Introducing Arbiter-Bench: How well can agents judge agent trajectories?

We measured how frontier models agentically grade rollouts.

Today we are sharing first results from the first version of Arbiter-Bench, a benchmark of how agent judges (models that open the files, run commands and investigate before ruling) grade real agent runs: 71 audited judge failures drawn from six public agent benchmarks and 8 judge models.

Introduction

A customer-service agent verifies a customer, updates her shipping address, and quotes the correct price for the cheapest T-shirt in stock. The customer jokes around; the agent jokes back that the shirts' quality "hopefully lasts longer than most of my punchlines". The database is correct and the benchmark marks the run a pass. One frontier model, asked to grade that run, failed it in every trial, at default and at high reasoning effort: as it is an "unsupported product claim".

This is a small example of a subjective judging error. Model-graded rewards now decide agent behaviour, and the thing being graded is no longer a short answer. It is a multi-step run in a terminal desktop. Almost nobody checks the graders on that kind of work, whether deterministic or LLM-based.

We wanted to know how often that verdict is wrong, and what wrong looks like (especially the ambiguous cases).

01 · SetupWe graded the graders on real agent runs, with the benchmark's own tests as ground truth

Every item is a real agent trajectory from a public benchmark (Terminal-Bench, OSWorld, SWE-bench, tau2-bench, ITBench, DTap). Ground truth is the benchmark's own executed test, except in a few audited cases where that test was wrong. We tried writing runs designed to fool a judge, and frontier judges saw through nearly all of them, so every failure here is mined from real agent work.

The judges are frontier models in their own vendor's agent harness (Claude Code, Codex, Gemini CLI, Grok Build; the open-weight models run in OpenCode and the Kimi CLI through OpenRouter) at default settings. Each one gets a sandbox with the task, the trajectory, and the files the run left behind. It can open files and run commands, and must write a pass or fail verdict. The network is closed except for the model's own API. Every judge grades every item three times. Two of the open-weight models (GLM-5.3, DeepSeek V4 Pro) are text-only, so they skip the 7 screenshot items.

An item enters the hard set if at least one frontier judge gets it wrong in a majority of its trials, and the ground-truth label survives an audit: reviewers from at least three model families review the item blind, and a human rules on every split vote. Items with a disputed label are archived and never scored.

02 · Overall accuracyThe results

We drew 622 runs with a pre-registered, balanced sampling design and scored the three judges we ran on all of them with inclusion-probability weights. All three landed in the 90-98%, so we filtered down a "hard set".

Most agent runs clearly work or clearly fail, so average accuracy is high. The hard set matters if you use a judge as a reward signal, because optimisation finds exactly the runs the grader gets wrong.

Cost does not track accuracy. Per scored judgment, at list price as each harness reports it, Astra costs $0.72 against $1.18 for Opus and $1.37 for Fable at the same accuracy, and answers in under a minute. It is not cheaper per token (it is the most expensive per token read); it reads about a third as much of the run before deciding. DeepSeek V4 Pro costs six cents a judgment and lands about 25 points lower.

03 · Error directionJudges fail in opposite directions, and more reasoning fixes only one

One accuracy number hides that judges fail in opposite directions. Astra fails 91 percent of the runs that genuinely fail (true negative rate) but passes only 51 percent of the runs that genuinely pass (true positive rate). Gemini is the mirror image: an 83 percent true positive rate against a 25 percent true negative rate. Fable is even.

More reasoning fixes one direction only. At high reasoning effort Astra roughly doubles its output and raises its true negative rate sharply (71 to 89 percent on the 25 items run at both settings), while its true positive rate does not move at all (27 percent at both). Strictness is a disposition, not a compute budget.

04 · Blind spotsA judge fail is often unique to the model

Take every run a judge got wrong and hand it to the other judges. Claude Fable 5.1 scores 36 percent on its own misses and 90 on GPT-6 Astra's. Astra scores 38 on its own and 93 on Fable's. Grok 4.6, which selected none of the items, still only reaches 57 overall. Claude Opus 5, which also selected nothing, scores 88 on Astra's and Gemini's misses but only 44 on Fable's: blind spots run in families. These are often subjective differences: Astra, for example, is stricter than the ground truth.

judge scored	on Fable's misses	on Astra's misses	on Gemini's misses	balanced
Claude Fable 5.1	35.7	90.0	90.9	72.2
GPT-6 Astra	92.9	38.3	99.2	76.8
Gemini 3.1 Pro	26.2	72.2	23.1	40.5
Claude Opus 5 (never selected)	43.8	87.5	87.9	73.0
GLM-5.3 (never selected)	41.0	80.3	63.5	61.6
Grok 4.6 (never selected)	33.3	82.5	55.3	57.1
DeepSeek V4 Pro (never selected)	26.2	72.2	50.0	49.5
Kimi K3 (never selected)	17.9	71.4	31.8	40.4

The top three (Astra 76.8, Opus 73.0, Fable 72.2) are a statistical tie; their intervals overlap almost entirely.

So a second judge from a different model family is a cheap check, and the runs where two judges disagree are the ones worth a human look.

The hard set is built from judges' misses, so whichever judge screened more items looks worse on it. Our headline number weights each screener's misses equally. Own-miss scores are still biased a little low, because the trials that selected an item also score it.

05 · MechanismsFailure mechanisms 

We classified the mechanisms for judging failures. The mechanism distribution is model-specific rather than case-specific: on nearly three quarters of the runs that two or more judges failed (39 of 54), they failed for different reasons.

The judges' own logs make the difference concrete. Gemini, passing a run on the strength of the agent's self-report: "Output from over 62 agent-authored test cases demonstrates successful preservation of valid HTML." Astra, failing a run whose every stated requirement it had just confirmed: "Passing equal-shape self-tests does not satisfy the unrestricted microbatch-list interface."

Judges also re-scope the task in both directions. One passes a failing run because "the original request does not separately require" the thing that went wrong. Another fails a passing run for a requirement that appears nowhere in the instruction.

06 · Benchmark integrityOnly a third of apparent judge errors survived a label audit

Selecting on disagreement between a frontier judge and a benchmark's ground truth was the heuristic I used to find wrong labels. Of 232 runs where a judge disagreed with the ground truth, 71 survived a blind audit by reviewers from at least three model families plus a human pass on every split vote. About a quarter were confirmed label or task defects, and another third had labels that reasonable reviewers disputed. When a test is wrong, the judge that disagrees with it is right.

This matches what others have found by auditing benchmarks directly. Epoch AI estimated that about 10 percent of OSWorld tasks have serious errors, and an OpenAI audit found that many SWE-bench Verified test suites reject correct fixes. Selecting on judge disagreement concentrates those tasks: a few percent of bad labels in the source becomes the majority of the candidates.

It also changed how we audit. Our first pass used a single strong model as the adjudicator, and it accepted 99 candidates as genuine judge errors. Three model families reviewing blind, with the original verifier output in front of them, kept about a third.

07 · PitfallsMost of what went wrong looked like a result

We list these because the failures of the measurement are part of the finding, and because runs we had to throw away are flagged and kept, not hidden.

1. Judges with a network look up the answer. 28 of 654 early trials fetched the hidden tests. Judges now run on an allow-list; one vendor's server-side search cannot be blocked, only detected.
2. One upload without screenshots invalidated 279 trials (138 of them on hard-set items). Every judge graded computer-use runs blind; one wrote "no screenshot is viewable" and carried on.
3. A judge is a model plus a harness. One CLI injects today's date, which influenced a date-dependent case.
4. A harness can blind a model that can see. One CLI adapter never declared image support, so a vision model graded 27 screenshot runs without opening a single image, and never said so. One config key fixed it.
5. An output cap looks like a timeout. One harness caps output at 32k tokens and counts reasoning against it, so a long-thinking judge lost a quarter of its trials with no verdict.

None of these showed up as an error message. Each one produced plausible numbers, and we caught them by reading transcripts.

08 · LimitationsThe set is small and Terminal-Bench-heavy, and one known bias remains

The hard set is small (71 runs) and 52 of them are Terminal-Bench. False-pass items (the judge passed a failing run) outnumber false-fail items (the judge failed a passing run) three to one, and false-fail labels are contested far more often.

Three more caveats. The top three judges are statistically tied; read groups, not ranks. Their earliest trials ran before we closed the network and were rerun closed, so every scored trial here is closed-network. Judges that helped select items are scored partly on their own misses while the five that selected nothing are not, so compare within those groups. And Claude Fable 5.1 is not only a judge here: it also served as one of the blind label reviewers and wrote much of the pipeline, under human direction.

Half of the set (36 items, the labels we are most sure of) is public with every judge's logs; the other 35 are held out to keep the benchmark usable, and carry a canary string.

This is version one. We plan to grow the set beyond Terminal-Bench and improve the way we evaluate judges. There were certain limitations, and a human ruled on 40 of the 71 labels (every split vote); the rest rest on the benchmark's executed test plus the blind model review, and were not individually human-audited.

We think alignment with human preference is the more important measure. For instance, if an agent tasked with playing a music video successfully played it but at zero volume, should it be failed? Partial rewards for judging outcomes is another consideration as well as minimal pairs where we require the judge to grade both a correct and incorrect rollout per task. 

Interesting work is coming soon, stay tuned!