The question of data labeling vs data annotation has a short answer in the standards and documentation reviewed here. Many guides and tools use the two words for the same work [6, 12]. Where a source separates them, annotation is the wider act of attaching information to data, and a label is the value a model learns to predict [3, 4]. The choice that changes a project is the label type: a class on a whole record, a span inside it, a link between two parts, a rating on a response or a region on a page.
This explainer sets out those label types for text and documents, from the standard, tool documentation or dataset paper that defines each, and how each is stored.
Data labeling vs data annotation: how the sources use the words
The W3C Web Annotation Data Model, a 2017 Recommendation, treats an annotation as a set of connected resources, typically a body and a target, where the body is related to the target [1]. Classifying is one of its 13 listed motivations, next to tagging, identifying, assessing and commenting [1]. In corpus linguistics, Geoffrey Leech describes annotation as "adding interpretative linguistic information to a corpus", with word-class tags, or labels, as one common type [5].
ISO/IEC 22989:2022, as quoted by itSMF Switzerland, defines data annotation as attaching descriptive information to data "without any change to that data", in the form of metadata, labels and anchors [3]. It defines a label as the "target variable assigned to a sample" [3]. Google's machine learning glossary calls a label the "answer" or "result" portion of an example, and it has no entry for annotation [4].
Vendors and tools often merge the two. IBM's explainer opens with "Data labeling, or data annotation" [6]. Label Studio's export schema describes its annotation results as those "of the labeling or annotation task" [12]. A SageMaker Ground Truth labeling job writes its entity spans under an "annotations" object [10]. AWS's own explainer draws the line elsewhere: the separate responses for one item are "called 'annotations'", and they are consolidated "into a single label" [7].
The EU AI Act lists "annotation, labelling, cleaning, updating, enrichment and aggregation" as separate data-preparation operations in Article 10, and Article 3 defines neither word [2]. No source reviewed fixes one industry-wide difference. Read each glossary for its own sense, and write the label types into the project's schema.
Labels on the whole record: classes and multi-label sets
A class label puts the whole item in one category. Azure Machine Learning's text labeling separates three task shapes: multi-class applies "only a single label to each piece of text", multi-label applies "one or more labels", and named entity recognition labels words inside the text [11].
Published sets show both shapes. GoEmotions labels about 58,000 English comments for "one or more of 27 emotion(s) or Neutral" [26]. SNLI labels 570,152 sentence pairs as entailment, contradiction or neutral, and its validated pairs with no three-vote majority, about 2%, carry the placeholder "-" [25]. Token-level classes work the same way: Universal Dependencies defines 17 part-of-speech tags [16].
Labels inside the record: entities and spans
An entity label marks a stretch of text and gives it a type. SageMaker's guide describes named entity recognition as locating noun phrases and categorizing each with a label such as person or organization [9]. CoNLL-2003 used four types: persons, organizations, locations and miscellaneous names [18]. An entity counts as correct "only if it is an exact match" [18].
Span rules differ on overlap. CoNLL-2003 assumes entities are "non-recursive and non-overlapping", and where one entity sat inside another, usually only the top-level entity was annotated [18]. Finkel and Manning report that roughly 17% of entities in the GENIA corpus are embedded in another entity, and nearly half in the AnCora corpus [19]. spaCy's entity recognizer accepts only non-overlapping spans and throws an error otherwise [15]. Its span categorizer labels "potentially overlapping spans", each with zero or more labels [14].
Spans can also be long and sparse. CUAD labels 13,101 clauses in 510 contracts across 41 categories, and on average only about 0.25% of a contract is highlighted for each label [35]. How models score on those clause spans is covered in what benchmarks show about LLM contract review accuracy.
Labels between parts: relationships, coreference and events
A relation label links two marked parts and names the link. In the brat standoff format, a binary relation has an ID, a type and its arguments, and unlike an event it marks no trigger text [13]. An event has a type, a trigger (the text stating the event) and arguments [13]. Equivalence relations group annotations that refer to the same real-world entity, which is one way to mark coreference [13].
A relation schema needs a label for no link. TACRED has 119,474 examples and 42 relation labels, and 78.7% of its examples are negative, labeled no relation [22]. Relations also cross sentence boundaries. In DocRED, at least 40.7% of the relational facts can only be extracted from multiple sentences, and its annotation marks entities and coreference before relations [23]. The CoNLL-2012 shared task linked coreferring mentions into entity and event chains [21].
Labels on a response: ratings and preferences
A rating puts a score on one response. SummEval rated summaries on a Likert scale from 1 to 5 for coherence, consistency, fluency and relevance, which gave 12,800 summary-level annotations [27].
A preference compares responses. In the instruction-following study by Ouyang and colleagues, responses were ranked in sets of 4 to 9 per prompt, a ranking of K responses produced K choose 2 pairwise comparisons, and ties were dropped [28]. Each response was also rated on a 1 to 7 scale, with binary flags such as one for hallucination [28]. Hugging Face's TRL library stores a preference as a prompt with a chosen and a rejected completion, and an unpaired preference as one completion with a true or false label [29].
Document label types: page regions, form fields and tables
Documents add position. DocLayNet labels 80,863 pages with bounding boxes in 11 classes, from Caption to Title [31]. Its authors excluded scanned documents wherever possible, because rotated or skewed pages do not allow annotation with rectangular boxes [31]. Predicted boxes are scored against ground-truth boxes with intersection over union (IoU) [4].
FUNSD shows a form label in full. Each entity in its 199 scanned forms has an identifier, a label (question, answer, header or other), a bounding box, its words and directed links, such as from a question to its answer [30]. Tables are labeled as structure: PubTabNet pairs 568k table images with HTML and scores recognition with TEDS, a tree-edit-distance similarity [33]. Labels can also come from a document's own markup. PubLayNet matched the XML and content of over 1 million PDF articles to annotate over 360 thousand page images [32].
Scoring an extraction tool against labeled answer keys is the subject of how to test document extraction accuracy before you buy.
How the label types are stored
Three storage patterns recur.
- Standoff offsets. brat keeps annotations in a separate .ann file and never modifies the text. Offsets start at 0 and exclude the character at the end offset [13]. The W3C Text Position Selector counts the same way [1]. SageMaker's output gives each entity a startOffset and an endOffset in Unicode positions [10].
- Token columns. CoNLL-2003 puts one word per line with four fields: the word, its part-of-speech tag, its chunk tag and its entity tag [18]. Its IOB tags come from Ramshaw and Marcus (1995), where I marks a word inside a chunk, O a word outside, and B the first word of a chunk that directly follows another [17, 18]. CoNLL-U uses 10 tab-separated fields per word [38].
- JSON records. Label Studio exports JSON, CoNLL2003, COCO and YOLO, and its documentation converts a CoNLL2003 export to spaCy's format with spacy convert [12]. FUNSD stores each link as an
[id_from, id_to]pair [30].
Offsets count characters and IOB tags count tokens [13, 18], so the schema states which convention a delivery uses.
What to settle before labeling starts
Klie, Eckart de Castilho and Gurevych separate the two documents a labeling task needs. The annotation scheme "defines the annotation labels", and the guidelines "describe how to decide when to apply which label" [8]. The label type sets much of what the scheme has to say:
- the unit: a whole record, a token, a span, a pair of texts, a response or a page region
- one label or several per unit, and whether spans may overlap or nest
- a label for nothing found, such as no relation, and a rule for ties
- the offset convention and the export format
The label type also sets the quality measure. Span labeling tasks can be scored with Krippendorff's unitized alpha family, because the units are chosen before they are labeled [8]. The coefficients for each design are in measuring label quality with inter-annotator agreement.
Rillor's data labeling and annotation services deliver training data labeled to your schema, then reviewed, with classes, entities, spans, relationships and ratings as your schema sets them.
Read the full report
The full report covers each label type with its dataset figures, the storage formats side by side, how the label type changes the quality measure, and all 38 sources.