Annotation and Data Quality Lead

• Write labeling guidelines with worked examples and edge cases for each client's schema, and agree them with the client before labeling starts.
• Define the label types each schema sets, such as classes, entities, spans, relationships and ratings.
• Run calibration rounds on a pilot set and revise the guidelines wherever labels disagree.
• Measure inter-annotator agreement by label type and guideline version, and act when it drops.
• Draw audit samples, set the review rules and measure accuracy on each reviewed sample.
• Keep every label traceable to how it was set and which version of the guidelines it followed.
• Trace systematic errors to their cause in the guidelines, the tools or the source data, and fix them there.
• Report the measured accuracy and the limits found with each delivery.
• Work with data engineers on schemas, labeling tools and delivery formats.

• 3+ years leading data labeling, annotation or data quality work for machine learning.
• Experience writing annotation guidelines, taxonomies or coding manuals.
• Working knowledge of sampling and agreement statistics.
• Python or SQL for quality analysis.
• Clear writing for clients and engineers.

• Labeling tools such as Label Studio, Prodigy or CVAT.
• Experience rating LLM outputs or building preference data.
• Experience labeling text, documents or tables.
• A background in linguistics, statistics or survey research.

$106,000 to $173,000 a year

Full-time W-2 employment with health insurance, a retirement plan and paid leave.

No application deadline. We accept applications for this role on an ongoing basis.

Remote, United States