SI evaluation
We test how agents behave before you trust them with real decisions. Tests run on your own tasks or in simulated markets, and every run can be replayed.
Why evaluate agents
Agents now act for the people and organizations that deploy them.
LLM agents increasingly answer, decide, negotiate and buy on someone's behalf. What they do depends on the model version, the prompt and everything they read along the way.
That behavior can be measured. We test agents on cases where the right outcome is known in advance, record every decision, and compare their choices with the answers agreed during scoping, with economic theory and with published human results.
Behavior is measured before an agent is trusted with a decision, and again whenever its model changes.
Agent evaluation vs LLM evaluation
LLM evaluation scores the text a model writes for a prompt. Agent evaluation also scores what the agent does with tools across steps: which tools it calls, the arguments it passes, how it handles each result and when it stops or hands off to a person.
Agent tests before go-live
We test the agents an organization plans to deploy on its own tasks, with black-box logging and replayable runs.
Task success
Whether the agent completes the task as specified, scored on cases with known answers.
Refusal and escalation
Whether the agent declines or hands off to a person when it should, and proceeds when it should.
Tool-use errors
Calls to the wrong tool, malformed arguments, ignored errors and actions taken on stale results.
Consistency
Consistency: the same decision across repeated runs, rephrasings and model versions.
Attacks on an agent and its tool access are tested in an agentic security engagement.
Agentic securityPublic benchmarks and your own tasks
Public benchmarks compare model families. Tests built from your own tasks show how your agent handles your work.
A public benchmark scores many models on the same shared tasks, so its results compare across model families and versions. Its tasks leave out your tools, your data and the decisions your agent is trusted with.
We build test cases from your own tasks, with the answers agreed during scoping, and run them on the pinned model versions your agent uses. The cases are rerun on each new model version, so results stay comparable over time.
What we need from you
Send us the tasks your agent handles, the model versions it runs on and the decisions it is trusted with.
What you receive
Findings come with the runs behind them.
Benchmark suites
Fixed sets of tasks and markets with scoring code, rerun on each new model version so results stay comparable over time.
Bias classifiers
Models that detect anchoring, herding and other biases from run logs alone.
Evaluation reports
Findings for one agent or a set of model families, with the hypotheses, the measures and the run logs behind them.
The method in full:
Measuring LLM agents in simulated auctionsBlack-box by design
Agents are measured only through what they receive and what they return. Any model family can be tested the same way.
- Inputs and outputs onlyPrompts, documents and tool results go in. Answers, actions and stated reasons come out.
- Pinned versionsEach agent runs on a named model family and a pinned version.
- Frozen snapshotsTasks, rules and prompts are frozen per run, so a rerun sees the same conditions.
- Replayable runsEvery run can be replayed from its log, step by step.
- Re-baselined on version changeWhen a model version changes, its baseline runs are repeated before any result is compared.
- Hypotheses firstWhat we expect to find is written down before the runs start.
The evaluation loop
Every study runs the same cycle, and every pass leaves a record that can be replayed.
LLM agents in simulated markets
We run superintelligence (SI) agents in simulated markets and auctions, measure how they bid and keep every run for replay.
Common questions
What is agent evaluation?
Testing how an LLM agent behaves before it is trusted with real decisions. We give it cases where the right outcome is known in advance, record every decision and score what it does. Every run can be replayed from its log.
How is agent evaluation different from LLM evaluation?
An LLM evaluation scores the text a model writes. An agent evaluation also scores what the agent does across steps: the tools it calls, the arguments it passes, how it handles each result and when it hands off to a person.
Which measures do you report?
Task success on cases with known answers, refusal and escalation, tool-use errors and consistency across repeated runs, rephrasings and model versions. Each finding comes with the hypotheses, the measures and the run logs behind it.
Can you test an agent built on any model family?
Yes. Agents are measured only through what they receive and what they return, so any model family can be tested the same way. Each agent runs on a named model family and a pinned version, and its baselines are rerun when that version changes.
Research
The newest articles from our Insights library, each with a full research report and its sources.
- Testing when an agent should hand off to a personHow to test agent escalation to a human as one decision with two errors, using paired cases, a confusion table, checks on the handoff itself and repeated runs.SI evaluation7 min readPDF report
- Questions to ask before you deploy a vendor's agentQuestions to ask an agent vendor about testing: model versions, test sets, repeated runs, escalation, prompt injection, logs and who tested, with a public source behind each.SI evaluation7 min readPDF report
- Validating an LLM judge against human labelsHow to check an LLM-as-a-judge against human labels: chance-corrected agreement, probes for position, length and self-preference bias, re-validation when the judge changes, and corrected final estimates.SI evaluation7 min readPDF report
Notices
SI outputs. Superintelligence (SI) outputs can be wrong. Important decisions should include human review.