Applied Scientist, Agent Evaluation
You measure how LLM agents behave before they are trusted with real decisions. You design the metrics and test sets, compare runs across model families and versions, and find regressions when a model changes. You write findings a client can check against the run logs behind them.
Duties
- Write down hypotheses, conditions and measures before any run starts.
- Build test sets from a client's own tasks, with cases whose right outcome is known in advance.
- Define metrics for task success, refusals, tool errors and consistency across repeated trials.
- Compare agents across model families and pinned versions with the statistics each comparison needs.
- Rerun baselines when a model version changes, and report any regression with the runs that show it.
- Check test sets for leaks, ambiguous cases and scoring that rewards the wrong behavior.
- Write evaluation reports with the hypotheses, the measures and the run logs behind each finding.
What you bring
- 3+ years in applied statistics, data science or machine learning evaluation.
- A graduate degree in statistics, computer science or another quantitative field, or equivalent experience.
- Experience with experimental design, hypothesis tests and confidence intervals.
- Python for analysis.
- Experience evaluating LLMs or LLM agents.
Nice to have
- Experience measuring agreement between human raters.
- Experience with Bayesian methods or mixed-effects models.
- Published work on evaluation methods or benchmarks.
Pay and benefits
$106,000 to $173,000 a year
Full-time W-2 employment with health insurance, a retirement plan and paid leave.
No application deadline. We accept applications for this role on an ongoing basis.
Location
Remote, United States
Apply
We review applications as each team grows and contact you when there is a match.
Rillor is an equal opportunity employer. We consider qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, veteran status or any other status protected by law.
If you need an accommodation to apply, email careers@rillor.com.