Research topic
Labeling and annotation
Labels tell a model what each record means: the class it belongs to, the entities and spans it holds, how they relate and how a response rates. Labeled training data is only as useful as the schema behind it and how evenly that schema is applied. Written guidelines and worked examples set how each label is chosen, and agreement between annotators on the same records shows how consistently the guidelines are followed.
Research on labeling and annotation
The research here covers the label types text and documents take, how annotation guidelines are written, how agreement between annotators is measured, what published studies find when language models are compared with people as annotators, and what a preference data record holds and how it is checked. Every figure carries its source and its date.
- Language models as annotators: agreement with people in published studiesWhat published studies find when an LLM is used as an annotator: agreement with crowd workers and experts, task by task, and how to check model labels.Labeling and annotation7 min readPDF report
- Data labeling vs data annotation: label types for text and documentsHow standards and tools use the two words, and the label types for text and documents: classes, entities and spans, relationships, ratings, page regions and tables, with their storage formats.Labeling and annotation7 min readPDF report
- Writing annotation guidelines: definitions, worked examples and edge casesHow to write annotation guidelines from published practice: label definitions with inclusion and exclusion rules, worked examples, edge cases, pilot rounds and a versioned change log.Labeling and annotation7 min readPDF report
- Measuring label quality with inter-annotator agreementHow percent agreement, Cohen's kappa, Fleiss' kappa and Krippendorff's alpha fit each annotation design, what inter-annotator agreement misses, and the label error rates published test sets carry.Labeling and annotation7 min readPDF report
Get new research by email
Work with us on labeling and annotation
Rillor delivers training data labeled to your schema, then reviewed. Labels cover classes, entities, spans, relationships and ratings, as your schema sets them. Written guidelines and worked examples are agreed with you before labeling starts. Each label records how it was set and which version of the guidelines it followed, and accuracy is measured on a reviewed sample and reported with the delivery.
About labeling and annotationNotices
Source data is provided within the rights of each source.
Bring us records to label.
Send a sample of the records, the labels your schema sets and the accuracy you need.