Machine Learning Engineer, LLM Agent Benchmarks

• Build benchmark harnesses that run agents through task suites with their tools, inputs and known answers.
• Write scoring code for each task, and test it against cases whose scores are known.
• Keep test sets sealed, with every access logged, so no model or prompt is tuned on them.
• Run contamination checks that look for test items in training data and on the public web.
• Freeze tasks, rules and prompts for each run and pin model versions, so a rerun sees the same conditions.
• Rerun benchmark suites on each new model version and keep results comparable over time.
• Log every run so it can be replayed step by step.
• Review tasks for leaks, shortcuts and ambiguous scoring.

• 3+ years building machine learning systems in Python.
• Experience evaluating LLMs or LLM agents.
• Experience building test harnesses or data pipelines that run at scale.
• Experience designing experiments and reading their statistics.
• Clear technical writing.

• Experience with PyTorch or JAX.
• Experience running large batches of jobs in containers.
• Published work on benchmarks or evaluation.

$124,000 to $201,000 a year

Full-time W-2 employment with health insurance, a retirement plan and paid leave.

No application deadline. We accept applications for this role on an ongoing basis.

Remote, United States