An agent training dataset and an agent evaluation set are built from the same unit: a task with known answers that a program can check. Each task states what the agent is told, the state it starts in, the tools it may use, the end state or answer that counts as success, and the check that compares them [1, 4, 32, 35]. This note covers how to write those tasks, verify each answer and keep the evaluation split apart from anything an agent trains on.
Sets copied from public benchmarks carry their flaws with them. Zhu and colleagues' Agentic Benchmark Checklist (ABC) found outcome-validity flaws in seven of ten popular agentic benchmarks and task-validity issues in seven [11]. Kapoor and colleagues found 7 of 17 agent benchmarks with no held-out set at all [13]. Those audits are covered in what agent benchmarks measure, and what they miss. This note is about building a set of your own that avoids them.
Agent training datasets and evaluation sets share one format
The five benchmarks below each store a known answer and a check with every task.
| Benchmark | Known answer | Check |
|---|---|---|
| SWE-bench (2023) | Tests that fail before the fix and pass after it | Apply the patch and run the tests [1] |
| tau-bench (2024) | The annotated database end state | Compare the final database with the goal [4] |
| WebArena (2023) | A reference answer or a target site state | String rules, a model judge or a state query [6] |
| OSWorld (2024) | The final state of files and windows | A per-task evaluation script [8] |
| GAIA (2023) | A short string, number or list | Quasi exact match after normalization [10] |
Training sets reuse the record. Each of SWE-Gym's 2,438 training tasks pairs a Python codebase with an executable environment, unit tests and a task in plain language [14]. Inspect, an open evaluation framework, stores a sample as an input, a target, files for the sandbox and a setup script [32]. The answer key goes by several names in the sources: the ground truth in GAIA and the gold patch for SWE-bench Verified [3, 10].
Keep the answer out of the agent's reach. In tau-bench the task annotation "is not visible to the agent and is used only for user simulation and evaluation" [4]. NIST's initial public draft on automated benchmark evaluations (January 2026) warns that agents can cheat by finding solution files "unintentionally left in the testing environment" [31].
Where the tasks come from
Start from real work. An engineering guide from a model developer (January 2026) recommends "20-50 simple tasks drawn from real failures", taken from the manual checks run during development and, once in production, from the bug tracker and support queue [34]. SWE-bench's tasks are real GitHub issues and the pull requests that resolved them [1].
When tasks are mined from real work, expect to discard most candidates. SWE-bench kept 2,294 tasks from about 90,000 pull requests [1]. Each kept task has at least one test that moves from fail to pass and runs without installation or runtime errors [1]. Generated tasks can carry their own proof. tau2-bench composes tasks from scenarios defined by initialization, solution and assertion functions, and checks that the final state satisfies every assertion once the solution functions have run, and not before [5].
Verify each known answer
Give every task a second reader. GAIA counted a question as validated only when two new annotators, answering on their own, reached the original answer [10]. 68% passed as written, and the rest were corrected or removed [10]. WebArena annotated each reference answer twice and had a third annotator settle disagreements [6].
Screening an existing set finds more. For SWE-bench Verified (2024), 93 developers annotated 1,699 SWE-bench samples, three times each, and 68.3% were filtered out for underspecified issues, unfair tests or other problems [2]. SWE-bench Verified is 500 of the samples that passed [2]. An audit in February 2026 still found material issues in 59.4% of 138 Verified problems that a frontier model failed to solve consistently [3].
Use agent runs to test the tasks. tau-bench's authors ran each retail task more than 40 times and polished the instructions until they were certain no ambiguities existed [4]. The engineering guide notes that a 0% pass rate across many trials "is most often a signal of a broken task", and suggests a reference solution for each task that passes all graders [34].
Rillor's agent evaluation work tests agents on cases where the right outcome is known in advance, records every decision and compares their choices with the answers agreed during scoping.
Include tasks where the agent should stop
The engineering guide warns that "One-sided evals create one-sided optimization" [34]. OSWorld includes 30 infeasible tasks and scores an agent as right when it accurately predicts that such a task fails [8]. WebArena labels its unachievable tasks "N/A" [6], following SQuAD 2.0, which added over 50,000 unanswerable questions written to look like answerable ones [24].
The check matters here as well. On tau-bench's airline tasks, an agent that returned only empty responses scored 38%, because empty replies counted as success on the intentionally impossible tasks [11]. Pair each stop case with a look-alike where the agent should go ahead, as XSTest pairs 250 safe prompts with 200 unsafe contrasts [25].
Choose the check before writing the tasks
Prefer a program that reads the end state or runs tests. tau-bench's authors write that with one possible database outcome, "subjective and noisy human judgments can be replaced by simple and objective database state comparisons" [4]. OSWorld chose execution-based scripts because evaluation that assumes a single solution for each task "wrongfully penalizes alternative correct solutions" [8]. ABC found that WebArena's string matching and its unvalidated model judge overestimate agent performance by 1.4% to 5.2% [11]. WebArena-Verified reviewed all 812 tasks and replaced the model judge and substring matching with deterministic scoring [7].
Where a model grades, measure the grader first. NIST's draft lists comparing a model judge's grading with human grading, and computing interrater agreement, as an emerging practice [31]. How the resulting scores are computed, from task success to pass^k, is set out in agent evaluation metrics and what each one tells you.
Keep the evaluation split out of training
Separate the sources. SWE-Gym's tasks come from 11 repositories "separate from those used in SWE-Bench to avoid contamination" [14], and SWE-smith's 128 repositories leave out all 12 SWE-bench test repositories [15].
Hold the answers back and mark the files. GAIA released its questions and kept the answers to 300 of its 466 private [10]. BIG-bench put a canary string in every task file so that training corpora can filter the tasks out [23].
Check the training data against the set. Tülu 3 removed training sets whose prompts overlapped its unseen evaluations on 8-gram matching [21]. Paraphrase and translation slip past n-gram checks [22], so dated tasks add a second guard: LiveCodeBench scores newer models only on problems released after their cutoff date [17]. Fixing the held-out split in a written specification is covered in how to specify a training dataset before anyone builds it.
How many tasks an agent evaluation set needs
No source reviewed sets a size. NIST's draft points to statistical power analysis [31]. Small sets carry wide intervals: a 90% pass rate runs from 69.9% to 97.2% on 20 tasks and from 87.1% to 92.3% on 500 (95% Wilson interval, derived) [28]. Tasks that share a template or repository are clusters, and clustered standard errors can be over three times the naive ones [27]. WebArena's 812 tasks come from 241 templates [6].
Version the set as answers drift
Answers and environments change. Only 44.4% of ToolBench's API calls succeeded when StableToolBench's authors tested them in 2024 [20]. By July 2025 OSWorld's maintainers had fixed more than 300 reported issues, among them changed websites and tasks with dates that had passed [9]. Date the source behind each answer, as GAIA advises [10], log every fix and retire tasks the agents have outgrown: a suite at 100% "tracks regressions but provides no signal for improvement" [34].
An evaluation set's written requirements are its tasks, the source behind each answer, the checks, the split and the refresh cadence. Rillor's custom datasets service scopes and builds training and evaluation data that does not exist yet and keeps it current, and each dataset comes with its schema, records with resolved identities, source provenance, lineage, version history, quality notes and stated limits.
Read the full report
The full report adds each benchmark's task record and construction effort, the verification and decontamination rules in detail, an interval table, a checklist for building an evaluation set, the steps for a training set, and the complete source list.