How reliable are LLM agents when the same task comes back? In the benchmark studies published from 2024 to 2026, an agent that solves a task once often fails it on a later try. On τ-bench in June 2024, the model that led the study solved about 61% of retail tasks on one try and under 25% on all eight of eight tries [3]. In Thinkingbox, a 2026 preprint of 507 business workflows run 20 times each, the proprietary model that led on single-run success fell from 66.50% to 47.53% when all 20 tries had to pass [10].
The studies measure this in three ways: success on every one of k tries, a consistency score per task, and the spread of scores between runs. Each figure below stays with its benchmark, run count and date. None is a production failure rate, and figures from different benchmarks are never combined.
How reliable are LLM agents? pass@k and pass^k
pass@1 is the share of tasks an agent solves on one try. pass@k counts a task as solved if any one of k tries succeeds, a measure that began in code generation, where one model solved 28.8% of problems with one sample and 70.2% with 100 [2]. τ-bench introduced the reverse for agents: pass^k, "the chance that all k i.i.d. task trials are successful, averaged across tasks" [3]. At k = 1 the two are the same number.
The arithmetic shows why the gap grows. At 75% success per trial, three independent trials all pass about 42% of the time [28]. The example comes from one model provider's January 2026 evaluation guide, which says the measure "especially matters for customer-facing agents where users expect reliable behavior every time" [28]. At 90% per trial, ten trials all pass about 35% of the time (computed for this report). How each measure is computed is set out in agent evaluation metrics and what each one tells you.
pass^k on τ-bench and τ²-bench
τ-bench pairs a tool-calling agent with a simulated user and scores each trial by comparing the database state at the end of the conversation with an annotated goal state [3]. Its authors wrote in 2024 that "with increasing k, the chance of consistently solving a task drops rapidly" [3].
The fall holds on current entries. Two submissions to the τ²-bench leaderboard dated 2 March 2026, both on version 1.0.1 with four trials per task, report pass^1 to pass^4 by domain [6, 7]:
| Entry and domain | pass^1 | pass^4 | Fall (derived) |
|---|---|---|---|
| Open-weight model, retail | 84.43% | 59.65% | 24.78 points |
| Open-weight model, airline | 81.5% | 68.0% | 13.5 points |
| Open-weight model, telecom | 97.81% | 92.11% | 5.70 points |
| Proprietary model, retail | 75.88% | 47.37% | 28.51 points |
| Proprietary model, airline | 80.5% | 66.0% | 14.5 points |
| Proprietary model, telecom | 91.01% | 74.56% | 16.45 points |
The two entries ran under different settings, and the proprietary one counted infrastructure errors as failures (3 in airline, 20 in retail and 2 in telecom), so each is read against itself [6, 7].
Twenty trials per task: Thinkingbox
Thinkingbox scores 507 workflows in retail, hospitality, auto insurance, neobank internal IT and consulting IT and HR support with executable checks that reject wrong, missing or extra effects, and runs 18 models 20 times per task [10]. One open-weight model reached 93.89% when any of 20 tries could count, 57.37% on one try and 17.60% when all 20 had to pass [10]. A model fifth on single-run success (58.31%) was second on all-20 success (46.89%), so the two rankings differ [10].
Many failures looked like successes. Of 79,853 failed trials across 12 models, 80.88% ended cleanly and had called a tool that changes state [10]. "A response-only or tool-call-only view would therefore make many incorrect executions appear complete", the authors write [10].
Capability rises faster than consistency
A Princeton study accepted at ICML 2026 ran 15 models released from early 2024 to mid 2026 five times per task on GAIA and a verified subset of τ-bench airline [11]. "Despite 24 months of model development, overall reliability only shows small improvements over time", its authors found, and outcome consistency, whether an agent succeeds or fails the same way on repeated attempts, "remains low across all models" [11]. The final version of its dashboard adds that newer models jumped in accuracy while their reliability did not rise at nearly the same rate [12]. In the dashboard table, the agent with 16.7% accuracy scored 0.80 on outcome consistency, above agents with 50% to 71% accuracy, because the score rewards steady failure as well as steady success [12].
The study set temperature to zero, except for reasoning models, so that the variation left came from sources other than sampling, such as floating-point arithmetic and batch size [11]. Those causes are covered in LLM nondeterminism: why the same input gets different answers.
One run is a noisy score
On SWE-bench Verified, a 2026 workshop paper ran three models on two scaffolds, ten runs per configuration, 60,000 trajectories in all [14]. Depending on which run was picked, single-run scores moved by 2.2 to 6.0 percentage points, with standard deviations above 1.5 points even at temperature 0 [14]. Its power analysis puts the runs needed to detect a 2% improvement at p < 0.05 with 80% power at about 9 per agent, and 36 for a 1% improvement [14]. For one open-weight coding setup, pass@1 was 63.5% and pass^5 49.1% [14].
Rankings move too. Resampling one run per question from 8,000 runs of four models, 29.3% of single-run evaluations ranked the models differently from the multi-run result [15]. In a September 2026 preprint of coding agents on one machine-learning task, the median pairing's run-to-run standard deviation, 0.0107 AUC, exceeded the 0.0095 spread between the six pairings' means, and three-run comparisons put the weaker pairing ahead 28 to 44 percent of the time [17].
Where repeated runs part ways
Runs diverge early. On SWE-bench Verified, trajectories diverged within the first 1% of tokens at the median [14]. In a 2026 preprint of 1,140 tool-calling traces, agents picked the same tools in the same order more consistently than they passed the same arguments, with mean scores of 0.87 and 0.69, and 60% of divergence began in the first two steps [16]. Agreement between runs carried information: on HotpotQA questions, tasks with at most two distinct paths reached 82% to 87% accuracy, against 41% to 65% for tasks with four or more [15].
Longer tasks and broken tasks
Longer tasks pass less reliably. METR's time horizon is the task length, in human expert time, at which an agent is predicted to succeed with a given reliability [22]. METR found models' 80% time horizons 4 to 6 times shorter than their 50% horizons [21]. On tasks taking a human expert 90 minutes to 3 hours, an agent with a 50% horizon of about 2 hours 17 minutes passed about a third every time, failed a third every time and was mixed on the rest [22].
Task errors move repeated-run scores as well. After the benchmark's maintainers fixed 27 airline and 26 retail tasks in February 2026, most of the fixes taken from τ-Bench Verified, airline pass^1 rose by 14.0 to 20.0 points depending on the model, and one model's pass^4 went from 50.0% to 72.0% [8]. How audits find such flaws is covered in what agent benchmarks measure, and what they miss.
What production teams do
Measuring Agents in Production, accepted at ICML 2026, drew on 20 case studies and a survey of 86 practitioners, and found that 68% of the systems surveyed execute at most 10 steps before human intervention and 74% depend primarily on human evaluation [27]. Its authors found that "Reliability (consistent correct behavior over time) remains the top development challenge", which teams address through systems-level design such as read-only modes and verification in a sandbox before production [27].
What to ask before an agent goes live
The sources point to a short list [3, 10, 11, 12, 14, 27]:
- pass^k beside pass@1, with k and the number of runs stated
- results per task, so flaky tasks show
- success judged on the final state of the environment
- run counts chosen by power analysis
- logs for every run, and reruns at regular intervals and after a model changes
Rillor tests each agentic system it builds on questions with known answers drawn from the client's own material, plus out-of-scope and adversarial questions, and logs every run so it can be replayed.
Read the full report
The full report adds each benchmark's setup and settings, the consistency scores in detail, the power analyses, the task-length studies and the limits of every figure.