Research topic
SI evaluation
Superintelligence (SI) evaluation measures what an LLM agent does with what it receives: whether it completes the task, which tools it calls and how, when it stops and hands off to a person, and whether its decisions hold across repeated runs and model versions. Each measure answers a different question about whether an agent can be trusted with a decision.
Research on SI evaluation
The research here sets out the methods behind those measures, drawn from published benchmarks, audits, standards and papers. Every article comes with a full report, available as a PDF, and a numbered list of its sources.
- Testing when an agent should hand off to a personHow to test agent escalation to a human as one decision with two errors, using paired cases, a confusion table, checks on the handoff itself and repeated runs.SI evaluation7 min readPDF report
- Questions to ask before you deploy a vendor's agentQuestions to ask an agent vendor about testing: model versions, test sets, repeated runs, escalation, prompt injection, logs and who tested, with a public source behind each.SI evaluation7 min readPDF report
- Validating an LLM judge against human labelsHow to check an LLM-as-a-judge against human labels: chance-corrected agreement, probes for position, length and self-preference bias, re-validation when the judge changes, and corrected final estimates.SI evaluation7 min readPDF report
- LLM nondeterminism: why the same input gets different answersWhy the same request to an LLM can return different answers even at temperature 0: sampling, floating-point order, server load, hardware and serving changes under one model ID.SI evaluation7 min readPDF report
- Agent evaluation metrics and what each one tells youHow task success, pass@k and pass^k, tool-call accuracy, escalation, cost per success and time horizon are computed, from the sources that define them, and what each misses.SI evaluation7 min readPDF report
- What agent benchmarks measure, and what they missHow SWE-bench, GAIA, WebArena, OSWorld and tau-bench score LLM agents, and the validity and contamination flaws that audits have found.SI evaluation7 min readPDF report
Get new research by email
Work with us on SI evaluation
Rillor tests how LLM agents behave before they are trusted with real decisions, on an organization's own tasks and in simulated markets. Agents are measured through their inputs and outputs alone, on pinned model versions, and every run can be replayed from its log. Before an agent goes live, we score task success on cases with known answers, along with refusal and escalation, tool-use errors and consistency across runs.
About SI evaluationNotices
Superintelligence (SI) outputs can be wrong. Important decisions should include human review.
Test your agent before it goes live.
Send us the tasks your agent handles, the model versions it runs on and the decisions it is trusted with.