The questions to ask an agent vendor about testing come down to seven: which model and version the agent runs on, which test cases it passed, how many times each case ran, when it stops and asks a person, how it held up against prompt injection, what its logs record and who did the testing. Each comes below with what a good answer contains and the public text behind it. This explainer is research, never legal advice.
The evidence is rarely public. The 2025 AI Agent Index documented 30 deployed agents from public information and developer correspondence, as of 31 December 2025 [1, 3]. Of the 30, 25 disclosed no internal safety results and 23 had no third-party testing information [2, 3]. Of the 13 agents with frontier levels of autonomy, 4 disclosed any agentic safety evaluations [1]. For most agents in the index, a buyer who wants test evidence has to ask for it.
Questions to ask an agent vendor about testing
| Question | A good answer contains | Sources |
|---|---|---|
| Which model and version, and what happens when it changes? | A pinned version in the logs, notice before changes, regression results on the new version | [20, 26, 29] |
| Which test set, with what known answers? | Cases like your tasks, answers fixed in advance, a setting close to production | [6, 19, 30, 32] |
| How many runs per case? | Several trials per case, a result that counts every trial, error bars | [7, 17, 26] |
| When does it escalate, refuse or ask for approval? | Written handoff rules, cases where stopping is correct, approval before high-impact actions | [10, 12, 17] |
| How was it tested for prompt injection? | Direct and indirect attacks, adaptive attacks, several attempts each, results per task | [8, 9, 18] |
| What do the logs record? | Model version, tool calls with arguments and results, approvals, a stated retention period | [12, 28, 29] |
| Who did the testing? | Testers apart from the developers, results you can read, a contract right to evaluate | [3, 5, 6] |
Which model and version, and what happens when it changes
An agent is a model with prompts, tools and settings around it, and any of them can change after you sign. In 2023, the March and June versions of one widely used hosted model scored 84% and 51% at telling prime from composite numbers [20]. The Guidelines for secure AI system development, published in November 2023 by UK and US government agencies with international partners, ask providers to treat major updates like new versions and to help users evaluate model changes, for example with preview access and versioned APIs [29].
A good answer names the model family and the pinned version, says how much notice comes before a change and shows a regression suite rerun on each new version. A January 2026 guide from one model provider describes regression evals that "should have a nearly 100% pass rate", with automated evals run on each agent change and model upgrade [26]. NIST AI 600-1, which is voluntary, suggests that vendor contracts consider "GAI system changes over time (e.g., fine-tuning, drift, decay)" [5].
Which test set, with what known answers
Ask what the test cases were, who fixed their answers and how close they are to your work. The NIST AI RMF asks that test sets and metrics be documented and that performance be shown "for conditions similar to deployment setting(s)" [6]. Singapore's Model AI Governance Framework for Agentic AI recommends testing whole workflows in an environment that "mirrors production as closely as possible" [30].
A headline figure can come from the wrong data. The FTC alleged that a content detector marketed as "98 percent" accurate scored 53 percent on general-purpose content in testing, and that its model had been trained or fine-tuned to classify academic content [32]. A 2025 study found task and grading flaws in agent benchmarks that can misestimate performance by up to 100% in relative terms [19]. What agent benchmarks measure explains why a public score says little about your own tasks.
A good answer lists cases built from tasks like yours, answers fixed before the runs, cases kept out of the agent's development and the grading rule for each.
How many runs per case
An agent can give different results on the same input. At 75% success per trial, it passes all three of three trials about 42% of the time [26]. On tau-bench, a 2024 customer-service benchmark, "state-of-the-art function calling agents" succeeded on under 50% of tasks, and in its retail domain the best agent's chance of passing all eight of eight trials of a task, averaged over tasks, was under 25% [17]. NIST's January 2026 draft on benchmark evaluations says that repeating each item "can reduce uncertainty of evaluation results" and asks for statistics with estimated uncertainties [7].
A good answer gives the number of trials per case, error bars and a result that counts every trial, such as pass^k, the chance that all k trials of a task succeed, averaged across tasks [17]. Agent evaluation metrics sets out how pass^k and consistency scores are computed.
When the agent escalates, refuses or asks for approval
Escalation can be tested only once the right action is written down. tau-bench's airline policy tells the agent to transfer the user to a human agent "if and only if the request cannot be handled within the scope of your actions" [17]. OWASP's entry on excessive agency recommends that a person approve high-impact actions before they are taken and that downstream systems enforce authorization [10]. The 2025 Index found that users configure enterprise agents at autonomy Level 1 to 2, while deployed agents often run at Level 3 to 5, triggered by events without human involvement [1].
A good answer shows written rules for handoff and approval, and results on cases where stopping or asking is correct and on cases where proceeding is. For high-risk systems, Article 14 of the EU AI Act asks that overseers can override or reverse the output and interrupt the system through a "stop" button or a similar procedure [12].
How the agent was tested for prompt injection
OWASP describes direct injection through a user's own prompt and indirect injection through outside content such as websites or files, and says it is unclear whether fool-proof prevention exists [9]. In NIST's 2025 agent hijacking evaluations, attempting each attack 25 times raised the average attack success rate from 57% to 80%, and new attacks developed with red teamers reached 81% success on held-out tasks, against 11% for the top baseline attack [8]. A 2025 study bypassed 12 recent defenses with attack success above 90% for most, though most of them had first reported near-zero rates [18].
A good answer names every channel the agent reads, the attack set and its date, adaptive attacks written against the agent's own defenses, several attempts per attack and results per task [8, 18]. Prompt injection testing covers the method in detail.
What the logs record
OpenTelemetry's conventions for recording tool calls require the tool's name, while a call's arguments and results are opt-in [28], so a log that follows the defaults can show which tool ran without what went in or came back. The 2023 guidelines ask providers to give customers high-quality audit logs at no extra charge [29]. In a 2025 study, log inspection found agents searching for the benchmark instead of solving the task, and misusing credit cards in flight-booking tasks [24].
For high-risk systems, the EU AI Act requires that systems allow automatic logging and has deployers keep the logs under their control for at least six months [12]. MCP access control covers logging and permissions at the tool layer.
Who did the testing
The 2025 Index documented third-party testing for 3 of its 30 agents [3]. NIST AI 600-1 suggests verifying that people running structured human feedback exercises are not directly involved in developing the same model, writing contract clauses that let an organization evaluate third-party processes and evaluating capability claims "using empirically validated methods" [5]. The FTC alleged that one legal service did not test whether it performed at the level of a human lawyer, and the final order the FTC announced in February 2025 requires $193,000 in monetary relief [31]. In 2024, a British Columbia tribunal held an airline responsible for what its website chatbot told a customer [33].
A good answer names who tested, how they were kept apart from the developers, which version they tested and where you can read the results, with a contract clause that lets you test too. Rillor's agent evaluation work tests the agents an organization plans to deploy on its own tasks, with black-box logging and replayable runs.
What the EU AI Act asks of the deployer
For high-risk systems, Article 26 asks deployers to use the system under its instructions for use, assign oversight to people with the necessary competence, training and authority, monitor its operation and keep its logs [12]. Article 13 asks that the instructions state the level of accuracy the system was tested and validated against [12]. Ask for those instructions before you sign. Under the consolidated text after the 2026 Digital Omnibus on AI, these provisions apply from 2 December 2027 for high-risk systems under Annex III and from 2 August 2028 for those under Annex I [13, 15].
Read the full report
The full report adds the evidence behind each question, the 2025 AI Agent Index figures, the deployer duties in the EU AI Act, a mapping of the questions to NIST, OWASP and EU AI Act items, and all 33 sources.