What agent benchmarks measure, and what they miss

Agent benchmarks measure one thing directly: whether an agent's output passes a fixed check on a fixed set of tasks. SWE-bench runs a repository's tests on the agent's patch. GAIA compares a short answer with a reference. WebArena, OSWorld and tau-bench check answers or the final state of a website, a desktop or a database [1, 2, 3, 4, 5]. A score reflects capability only when the check is correct and the tasks are new to the model.

Audits keep finding cases where one condition or both fail. A 2025 checklist study of ten agentic benchmarks found outcome-validity flaws in seven and task-validity flaws in seven [7]. On tau-bench's airline tasks, an agent that returned empty responses scored 38%, above a function-calling agent built on a widely used hosted model [7]. In February 2026, the model developer that built SWE-bench Verified with the benchmark's authors stopped reporting its scores, citing flawed tests and contamination [25, 26].

What the main agent benchmarks measure

Sources: [1, 2, 3, 4, 5].

GAIA, WebArena and OSWorld also published human baselines of 92%, 78.24% and over 72.36% [2, 3, 4]. Major model providers cited SWE-bench Verified, GAIA, WebArena, OSWorld and tau-bench in launch posts between January 2024 and March 2025 [7]. Agent evaluation metrics defines the measures built on these scores, such as pass^k.

Are agent benchmarks reliable? Task and outcome validity

The Agentic Benchmark Checklist (ABC) calls an evaluation rigorous when two conditions hold [7]. Task validity: a task is solvable if and only if the agent has the target capability. Outcome validity: a passing result means the task was done. All ten benchmarks ABC assessed had gaps in reporting [7]. Errors run in both directions.

• tau-bench. 38% of airline tasks are intentionally impossible, and success means leaving the database unchanged. An agent that does nothing passes them. An agent that outputs all the data scores 40%, because 2% of airline tasks are also graded by substring matching [7].
• WebArena. Substring matching accepts answers padded with extra text, and the judge accepts an empty reply on tasks labelled "N/A". ABC puts the overestimate at 1.4 to 5.2% [7].
• OSWorld. Website changes broke 13 of 46 tasks in its Chrome section and understated one open-source agent's score there by 28 points [7].
• WebArena-Lite. A 2026 human review of all 165 tasks recovered 5.45 to 8.49 points of success that the automatic grader had missed [19].

SWE-bench Verified contamination and weak tests

SWE-bench Verified was released in August 2024 as a human-screened subset of the original. 93 developers screened 1,699 tasks, 68.3% were filtered out for underspecified problems, unfair tests or other issues, and 500 remained [25].

• Tests that pass wrong patches. UTBoost found 26 of the 500 tasks had insufficient tests. After its corrections, rankings changed for 11 of 45 leaderboard entries (24.4%), and the leading entry fell from 55% to 53.6% [10]. For three state-of-the-art tools, a second study estimated that weak patch validation inflated reported resolution rates by 6.4 percentage points [11].
• Remembered answers. Given only the issue text, models named the buggy file 60% to 76% of the time on Verified, against under 53% on popular repositories outside SWE-bench [13].
• The co-creator's audit. The developer that built Verified with the SWE-bench authors [25] audited 138 problems that one reasoning model failed to solve consistently over 64 runs. At least 59.4% had tests that reject functionally correct submissions. Every frontier model tested could reproduce the reference fix or verbatim problem details for some tasks. The top score rose from 74.9% to 80.9% in the six months before February 2026 [26].

The developer concluded that gains "increasingly reflect how much the model was exposed to the benchmark at training time" and recommended SWE-Bench Pro [26].

Fresh tasks and successor benchmarks

Fresh-task benchmarks draw on recently created issues, and SWE-rebench flags issues older than a model's release [14, 15]. Each row below runs one setup both ways.

SWE-rebench's authors say some models' Verified scores "might be inflated due to contamination issues" [14]. The third row is a second kind of leak: agents can read fixes from Git history or the network during the run, and blocking both cut 21.48 points [17]. SWE-Bench Pro holds out 858 problems to test for overfitting, and the models tested on its public set stayed below 45% in the paper's November 2025 version [16]. By October 2026 its maintainers had repaired or retired 480 of the 731 public tasks [18]. On the repaired 642-task set, which is not comparable with the 2025 results, frontier agents resolve up to 99.4% [18].

OSWorld vs WebArena: graders and drift

WebArena runs four self-hosted websites in containers, does not model website drift and has no held-out test set [3, 9]. The top agent on its leaderboard in mid 2024 scored 35.8% and hard-coded policies for specific WebArena tasks [9]. WebArena-Verified reviewed every task and replaced the model judge and substring matching with deterministic scoring [27].

OSWorld runs real applications, some on live websites [4, 7]. Its maintainers released OSWorld-Verified in July 2025 after fixing more than 300 reported issues [28]. In the verified results, viewed 24 October 2026, the top entry, an agentic framework, scored 90.19% at a 100-step limit in July 2026, above the original 72.36% human baseline [29, 4]. One model released in 2025 scored 9.1%, 17.17% and 23% at 15, 50 and 100 steps [29]. OSWorld 2.0, released in June 2026, uses 108 workflows that take people a median of about 1.6 hours, and the top model completed 20.6% of them [20].

tau-bench explained: scoring with a simulated user

tau-bench introduced pass^k, the chance that all k trials of a task succeed. In June 2024 the leading function-calling agent passed over 60% of retail tasks on one trial, and its pass^8 fell below 25% [5]. The maintainers' 2025 follow-up found that the simulated user, itself a language model, caused task-critical errors in 13% of airline conversations [6]. In February 2026 they audited and fixed more than 50 airline and retail tasks [30]. On taubench.com, viewed 24 October 2026, the top tau2-bench text entry scored 87.9% on pass^1, against 55.2% for the top entry on the newer banking domain [30].

Saturation, single runs and cost

A 2026 study of 60 language model benchmarks found nearly half saturated, with rates rising with benchmark age [23]. HAL ran 21,730 agent rollouts across 9 models and 9 benchmarks for about $40,000. Higher reasoning effort gave equal or lower accuracy in 21 of 36 combinations, and two web agents differed ninefold in cost for a two-point gap in accuracy [21]. HAL has since declared CORE-Bench solved and paused adding new models [31]. SWE-rebench's authors name reporting pass@N as pass@1, and reporting only the best run, as practices that overstate capability [14]. The black-box evaluation protocol covers repeated trials.

Which benchmark for LLM agents: questions to ask

Match the benchmark to the work: SWE-bench and its successors for code, WebArena for websites, OSWorld for desktop applications, tau-bench for policy-bound conversations and GAIA for questions that need search and tools [1, 2, 3, 4, 5]. Then check how far the score carries. In a 2025 METR study, an early-2025 agent had an average success rate of 38% on 18 real issues, measured by the maintainers' tests, and none of the 15 pull requests reviewed by hand was mergeable as-is [35].

For any reported score, check:

• Which benchmark version, and how many tasks?
• Which scaffold, step limit and number of runs, and is the figure pass@1 or the best of several [14]?
• Were the tasks created after the model's training cutoff [14]?
• What do a trivial agent and a baseline that uses no model score? ABC asks benchmarks to report both [7].
• What did each success cost [21]?

Rillor's agent evaluation and simulation work tests the agents an organization plans to deploy on its own tasks, with benchmark suites of fixed tasks and scoring code rerun on each new model version so results stay comparable over time.

Read the full report

The full report adds each benchmark's audit record, the CORE-Bench case, time horizons and implications by reader, with all 35 sources.