Agent escalation to a human is a decision with two ways to go wrong: the agent proceeds when it should hand off, or hands off when it should proceed. A test of escalation therefore needs cases where handing off is right and cases where it is wrong, a score for each error, and checks on the handoff itself. Published guides say when an agent should escalate. The research below shows how to test whether it does.
Escalation is one of several measures in agent evaluation metrics, and it can be scored only where the correct action is written down before the run. This note sets out how to build and score that test.
Agent escalation to a human: the published triggers
One model provider's guide (April 2025) names two triggers for human intervention [1]. The first is exceeding failure thresholds, such as failing to understand a customer's intent after multiple attempts. The second is high-risk actions that are "sensitive, irreversible, or have high stakes", such as canceling orders, authorizing large refunds or making payments [1]. A second provider adds pauses for human feedback at checkpoints or blockers, along with stopping conditions such as a maximum number of iterations to keep control [2]. A governance framework from Singapore's Infocomm Media Development Authority (IMDA), version 1.5 of May 2026, adds outlier behavior, such as an agent reaching a system outside its work scope, and limits that users set, such as purchases above an amount [7].
One agent platform's analytics sorts escalations by cause: a business rule set by the builder, a threshold such as three failures at a task, or a user's request for a person [4]. Its escalation rate is the share of engaged sessions that handed off [5]. That number shows how often the agent escalated. It cannot show whether each handoff was needed or whether a needed one was missed.
The missed kind has a documented cost. In 2023 the Consumer Financial Protection Bureau described "doom loops" in consumer-finance chatbots with no offramp to a person, and warned that "only specific words or syntax may trigger the recognition of a dispute" [6]. A test case for that failure states the same dispute without the trigger words.
Two errors, one decision
Research on abstention, where a model declines to answer, scores the decision with a confusion table [9]. Read "abstain" as "hand off" and each test case lands in one of five cells: proceeded and right, proceeded and wrong, handed off when it could have proceeded, proceeded when it should have handed off, and handed off correctly [9]. Unneeded handoffs and missed handoffs become two separate numbers.
Each error alone has a trivial winner. A model that refuses every prompt "would be perfectly harmless but also completely useless", as XSTest's authors put it [11]. In ToolEmu, an agent that seeks confirmation before every action, even harmless ones, can reach a perfect safety score while failing the task, so its authors score helpfulness alongside safety [13]. One provider's evaluation guide states the rule plainly: "One-sided evals create one-sided optimization" [3].
The two errors tend to move together. Across 32 models in OR-Bench, an over-refusal benchmark, rejection rates on toxic prompts and on safe prompts had a rank correlation of 0.89, and the authors conclude that "most models simply trade over-refusal for safety" [12].
Build paired cases
A paired case is the same request in two forms, one the agent should handle and one it should not. XSTest pairs 250 safe prompts with 200 unsafe ones [11]. Ambig-SWE gives each of 500 software issues a fully specified form and an underspecified form [16].
Four kinds of request belong in an escalation set:
- Missing or unclear information. NoisyToolBench finds that language models "tend to arbitrarily generate the missed argument" [15]. BFCL V3 has 200 cases where the right move is to ask for a missing parameter [14].
- No correct answer. AbstentionBench covers unknown answers, underspecification, false premises, subjective questions and outdated information. Across 20 models abstention was unsolved, larger models helped little, and two reasoning models showed an average 24% drop in abstention against their non-reasoning counterparts [10].
- Policy limits. tau-bench, a customer-service benchmark, tells its agent to transfer the user "if and only if the request cannot be handled within the scope of your actions" and to deny requests against policy [18]. Its successor's telecom policy asks the agent to try to resolve the issue before transferring [19]. Resolve, deny and transfer are different correct actions.
- Sensitive actions. Irreversible or high-impact steps that need a person's confirmation [1, 7, 34].
Write the correct action into every case before the run. AgentAtlas (May 2026) names six states for that: act, ask, refuse, stop, confirm and recover [26]. Write forbidden actions in too. ToolSandbox scores a whole trajectory zero when the agent does what the case says it must not, such as calling a tool on a timestamp it invented [17].
Score both errors side by side
Ambig-SWE shows why a single rate misleads. It reports unneeded questions and missed questions for each model under three prompts [16]. One open coding model never asked: 0.00 unneeded and 1.00 missed under every prompt [16]. One proprietary model moved from 0.00 unneeded and 0.81 missed under a neutral prompt to 0.36 and 0.10 under strong encouragement to ask [16]. A single share of questions asked would hide both results.
Check the action and the message as well as the end state. On tau-bench's airline tasks, 38% of tasks are intentionally impossible and success means leaving the environment unchanged, so an agent that did nothing scored 38%, above a model-based agent at 35% [20]. What agent benchmarks measure covers that benchmark's other scoring flaws. Where a model judge decides whether the agent declined, check the judge too. AbstentionBench's judge was 88% accurate on a hand-labeled sample [10].
Confidence thresholds need the same two numbers. In KnowNo, a robot planner from 2023, a lower target error raised task success and required more human help [24]. In one 2024 study, models stating their own confidence tended to be overconfident, and every method tested struggled on tasks that need professional knowledge [23].
Test the handoff itself
A handoff can fire and still fail. One agent platform counts a session as escalated "whether the conversation transfers to a live agent or not" [4]. Follow each case to a person or a queue.
Read the note. In a retail task in tau-bench's appendix, the agent broke the one-exchange rule, hit an error on its second exchange and transferred the case with a note that blamed the system and left out its own first exchange [18]. IMDA's framework asks that approval requests be short and clear and include data such as the risk of the action [7].
Test decline, cancel and reject. The Model Context Protocol (MCP) gives a request for information three outcomes: accept, decline and cancel [28]. One agent framework's documentation warns that under a conditional pause, if the condition changes on resume, "the tool runs and the reviewer's answer is ignored, even a reject" [29]. The same framework reruns the code before a pause when the run resumes, and a paused run waits indefinitely for an answer [30]. IMDA asks for denying action by default when no approver is reachable [7].
Prompts move both errors, so rerun
A prompt that fixes one error can create the other. In XSTest's 2023 tests, adding a guardrail system prompt to one open model cut its unsafe responses and raised its full refusal of safe prompts from 0.8% to 9.6% [11]. AbstentionBench found that a carefully written system prompt raised abstention without fixing the models' inability to reason about uncertainty [10]. Treat every prompt or model change as a new version and rerun both halves of the set, several times per case, for the reasons set out in LLM nondeterminism. Compare versions on paired differences per case [35].
A passing result proves less than a failing one. XSTest's authors say their suite "has negative predictive power" [11]. An escalation test shows how the agent handled the cases in it, and it is no safety guarantee. For buyers, questions to ask before you deploy a vendor's agent turns these points into requests for evidence.
Rillor's agent tests before go-live run on an organization's own tasks with black-box logging and replayable runs, and score refusal and escalation: whether the agent declines or hands off to a person when it should, and proceeds when it should.
Read the full report
The full report adds the confusion-table formulas, the published transfer rules, the confidence and deferral research, the oversight texts that name the behavior and a paired test-set design with a source behind each element.