Agent evaluation metrics turn trial runs into numbers, and each number answers a different question. Task success, pass@k and pass^k check the final state once and across repeated trials. Tool-call, refusal and escalation measures check the path and the decisions along it. Cost per success prices the result, and time horizon restates success rates as a length of task in human time.
At 75% success per trial, an agent scores about 98% on pass@3 and about 42% on pass^3 (computed here for independent trials, from [3, 37]). Which number matters depends on whether one success is enough or every run must succeed.
The main agent evaluation metrics
| Metric | Question it answers | How it is computed | Sources |
|---|---|---|---|
| Task success (pass^1) | Did the trial end in the goal state? | Final state against an annotated goal state | [3, 37] |
| pass@k | Does at least one of k trials succeed? | 1 - C(n-c, k) / C(n, k) | [3, 4] |
| pass^k | Do all k trials succeed? | C(c, k) / C(n, k) | [3] |
| Outcome consistency | Same result on every run? | (2p - 1)² | [7, 27] |
| Tool-call accuracy | Are the calls right? | Calls matched to a reference, or state checked each turn | [23, 24] |
| Trajectory match | Did the agent take the expected path? | Tool calls against a reference trajectory | [12, 20] |
| Refusal and escalation | Did it stop or hand off when it should? | Cases with the correct action set in advance | [13, 15, 24] |
| Cost per success | What does one success cost? | Cost per attempt divided by success rate | [9] |
| 50% time horizon | What task length does it complete half the time? | Logistic fit of success against human time | [10] |
pass@k, pass^k and outcome consistency are computed per task and averaged over tasks. For one task, n is the number of trials, c the number that succeeded and p the success rate. C(a, b) counts the ways to choose b items from a.
Task success: check the final state
A January 2026 guide from one model provider separates what an agent says from what it did. A flight-booking agent can report that the flight is booked. The outcome is whether a reservation exists in the database [37]. tau-bench, a 2024 benchmark of customer-service agents, scores an episode 1 only if the final database matches the goal state and the agent's replies contain the information the user needed [3]. Simulated auctions fix the efficient outcome in advance with induced values, as the auction measurement note describes.
Success checks can be wrong. A 2025 checklist study found that a trivial agent returning empty responses scored 38% on tau-bench's airline tasks, because impossible tasks counted as successes [14]. In the guide's example, one frontier model's CORE-Bench score rose from 42% to 95% after fixes to rigid grading, ambiguous task specifications and stochastic tasks that could not be reproduced exactly, plus a less constrained scaffold [37].
pass@k vs pass^k
pass@k comes from code generation: the chance that at least one of k samples passes the tests [4]. tau-bench introduced pass^k, the chance that all k trials succeed [3]. At k = 1 the two are equal. As k grows, pass@k rises toward 100% and pass^k falls toward zero [37]. The guide recommends pass@k where one success is enough and pass^k where consistency is essential [37].
With 6 successes in 8 trials, pass^3 is C(6, 3) / C(8, 3) = 20/56, about 36%, below the 42% that 0.75³ suggests (computed here from [3]). The 2021 HumanEval paper showed that the shortcut 1 - (1 - p)^k is a biased estimate of pass@k [4].
In June 2024, a function-calling agent on a widely used hosted model had a pass^1 of about 61% on tau-bench retail, and its pass^8 was below 25% [3]. On the tau-bench leaderboard, the top banking entry run over four trials, evaluated on 3 August 2026, scored 55.2% on pass^1 and 35.1% on pass^4 [29, 30].
Consistency across runs, rephrasings and versions
A 2026 reliability study proposes outcome consistency, (2p - 1)². It is 1 when an agent always succeeds or always fails on a task and 0 at a 50/50 split [7, 27]. An agent at 75% success scores 0.25 (computed here). The study also measures prompt robustness on paraphrased instructions, and sensitivity to paraphrasing varied substantially across 15 models [7]. Versions matter too. In 2023, the March and June versions of one hosted model scored 84% and 51% at telling prime from composite numbers [16]. The black-box evaluation protocol covers repeated trials and error bars.
Tool-call accuracy and trajectory measures
Benchmarks score tool use in different ways.
- AST matching. The Berkeley Function Calling Leaderboard (BFCL) checks the function name and every required parameter for an exact match, is strict on types and fails the answer on any mismatch [23].
- State and response checks. BFCL's multi-turn tasks compare the backend state and the execution path with ground truth after every turn. An entry passes only if both hold in all turns [24].
- Trajectory match. One cloud provider's documentation defines exact match, in-order match, any-order match, and precision and recall of actions against a reference trajectory [20].
- Milestones and minefields. ToolSandbox scores the critical steps a valid path must include and sets the score to 0 if the agent triggers a forbidden event [12].
Outcome vs trajectory evaluation
Path checks see what state checks miss: some calls, such as a zip-code lookup, leave no trace in the system state [24]. Strict paths penalize valid alternatives, so the guide advises grading what the agent produced over the path it took [37]. In the 2025 HAL study, agents on two benchmarks almost never finished a run without a tool-call failure, even when they succeeded [6].
Refusal and escalation as scored outcomes
Refusal and escalation become measurable once the correct action is set in advance. tau-bench's policies tell the agent to transfer the user to a human agent if and only if the request falls outside its actions [3]. BFCL's multi-turn set has 200 cases with missing parameters, where the model should ask for clarification [24]. XSTest pairs 250 safe prompts a model should answer with 200 unsafe prompts it should refuse, so over-refusal counts as an error [15]. AgentAtlas, a May 2026 proposal, scores six control decisions: act, ask, refuse, stop, confirm and recover [13].
Cost per success and latency
Accuracy alone can be raised by retrying, so the authors of "AI Agents That Matter" argue that evaluations must control for cost [8]. Cost-of-pass divides the expected cost of one attempt by the success rate [9]. In a hypothetical case computed here, $0.50 per attempt at 40% success costs $1.25 per success, and $2.00 at 90% costs about $2.22.
Across 21,730 HAL rollouts, reported in October 2025, more reasoning effort gave equal or lower accuracy in 21 of 36 model, agent and benchmark combinations. Two agents on one web benchmark cost $171 and $1,577 for a two-point difference in accuracy [6]. BFCL lists an estimated cost per full benchmark run in US dollars and latency in seconds [21]. NIST's January 2026 draft recommends reporting cost alongside performance when systems are not uniformly cost controlled [2].
Time horizon
METR's 50% time horizon is the task length, in human expert time, at which an agent is predicted to succeed half the time, from a logistic fit of success against human completion time [10, 33]. On METR's Time Horizon 1.1 suite it was about 60 minutes for a model released in February 2025 and about 12 hours for one released in February 2026, with bounds of about 5 to 61 hours [34]. That model's 80% horizon was about 70 minutes [34]. METR treats measurements above 16 hours as unreliable, and its suite is mostly software, machine learning and cybersecurity tasks [33]. METR last updated the page on 8 September 2026 [33]. The paper calls its projection of a one-month horizon between mid-2028 and mid-2031 a naive extrapolation [10].
What outcome-only scores hide
In an August 2025 METR study, an early-2025 agent had an average success rate of 38% on the maintainers' tests across 18 real repository issues, and none of the 15 pull requests reviewed by hand was mergeable as-is [36]. HAL's log review found that web benchmarks score an agent that abstains the same as one that leaks a user's credit card details [6].
How to measure LLM agent performance and report it
NIST's generative AI profile asks evaluators to avoid extrapolating from narrow, non-systematic and anecdotal assessments [1]. NIST's January 2026 draft on automated benchmark evaluations asks reports to state the benchmark version, the number of test items, the exact model version and the cost controls [2]. The Agentic Benchmark Checklist adds confidence intervals and the score of a do-nothing agent [14].
Rillor's agent evaluation and simulation work scores four of these measures before an agent goes live: task success on cases with known answers, refusal and escalation, tool-use errors, and consistency across repeated runs, rephrasings and model versions.
Read the full report
The full report adds a worked table of pass@k and pass^k, the grader types, dated leaderboard and time-horizon figures, and all 38 sources.