Annotation guidelines are the written rules that tell anyone applying a label set how to decide which label fits each item. A 2024 audit of 591 dataset papers draws the line plainly: the annotation scheme defines the labels, and the guidelines describe how to decide when to apply each one [1]. The same audit calls properly written guidelines essential for consistency and reproducibility, for example when a dataset is re-annotated or extended [1]. In many projects they are revised several times in a pilot study before the main labeling starts [1].
Published datasets often say little about them. Of 74 machine learning papers that used new human labels for Twitter data, 35 (47.30%) gave no information about the instructions annotators received (2020) [26]. In the 2024 audit, only 22% of the papers with human annotators mentioned a pilot study [1].
What annotation guidelines contain
Published guidelines share a common set of parts: the label set, a definition for each label, rules for choosing between labels, worked examples, edge cases and a record of changes [1, 6, 7, 13, 22]. Their length follows the task. By page count (derived), the Penn Treebank bracketing guidelines run to 318 pages [19], the ACE entity guidelines to 72 [8], the TimeML guidelines to 71 [20] and a search engine's published rater guidelines to 182 [13]. In crowdsourced projects, guidelines are usually very short and often fit into the annotation screen [1]. One cloud labeling service's documentation splits short instructions on the task page from full instructions that show edge cases, and says the short ones should take less time to understand than the task takes to complete [14].
Fewer labels help. A meta-analysis of 96 annotation studies found that agreement fell as the number of categories rose, and its authors advise schemes "as sparse as possible" where reliability has to be high [5].
Definitions with inclusion and exclusion criteria
Social scientists call annotation guidelines a codebook [7]. In one qualitative research study, the codebook gives each code a label, a definition, a description, qualifications or exclusions, and examples from the raw data [6]. A machine-readable format proposed for political science codebooks has the same parts: the label, a one-sentence definition, clarifications of what fits and what does not, and positive and negative examples [7].
Exclusions carry much of the work. The MUC-7 named entity rules state that titles such as "Mr." and role names such as "President" are not part of a person name, and that generic names such as "the police" are not tagged [9]. Where an item fits two labels, a precedence rule settles it: the ACE guidelines assign the most specific organization type possible, then list the exceptions [8]. Where context decides, the rule shows both readings, as MUC-7 does with "member of the 91st Congress" and "commencement of the 91st Congress" [9].
Worked examples: positive, negative and near misses
A search engine's published rater guidelines call their examples "very important" for showing how each concept applies [13]. Negative examples can say where an item belongs instead. In the political science format, each negative example ends with the reason it belongs to the other class [7]. The ACE guidelines tag the names of fictional characters, and list "Morris the cat" among the names that yield no person mention, since animal names are not tagged [8].
Examples also bias data that annotators write. In 14 benchmarks where annotators wrote the examples, about 73% of instruction examples shared a few clear patterns, and those patterns appeared in about 61% of the collected instances [11]. Models scored about 7% (base) and 3% (large) higher on the examples that carried those patterns [11]. The authors recommend a large and varied pool of examples, a different random sample for each annotator, and checks for word patterns during collection [11].
Edge cases and "when in doubt" rules
The 2024 audit advises picking pilot data with as many corner cases and difficult instances as possible, so that significant changes come before the main labeling, where they could force costly re-annotation [1]. It also prefers adding edge cases to the guidelines over filtering hard items out [1].
Some rules settle doubt directly. For page quality ratings, the rater guidelines tell raters unsure between two ratings to use the lower one, and raters torn between three to choose the middle [13]. The same guidelines define flags for special cases, such as "Did Not Load" for pages with no content created by the website [13]. No guideline covers everything. Hovy and Lavid note that instructions for every conceivable case would make annotation unnecessary, since they could be encoded as a program, and that fully prespecifying any nontrivial phenomenon is impossible [2]. When two similar categories keep low agreement, the audit names merging them into one label as one possible fix [1].
Prescriptive or descriptive annotation guidelines
Röttger and colleagues separate two aims [16]. The prescriptive paradigm discourages annotator subjectivity, so a model can be trained to apply one belief consistently. The descriptive paradigm encourages it, so the data can survey and model different beliefs [16]. In their experiment, three groups of 20 annotators applied a binary label to the same 200 social media posts [16]. Detailed prescriptive guidelines produced 93.72% agreement (Fleiss' kappa 0.78), a short definition 72.50% (0.15) and the descriptive prompt 73.90% (0.20) [16]. They advise dataset creators to aim explicitly for one paradigm or the other [16].
Higher agreement can also hide a cost. Aroyo and Welty found that increasingly precise guidelines eliminate disagreement without increasing quality, by forcing annotators into choices they may not think valid [17].
Pilot, measure and revise
Hovy and Lavid describe the cycle: write the manual, annotate a fragment of the corpus, measure agreement, and return to the definitions and the manual when agreement falls short [2]. OntoNotes annotated a 50-sentence sample and checked its agreement at once, and scores below 90% led to a revision and clarification of the sense groupings [3]. The project reports at least a three-fold increase in annotator productivity [3]. In a medical corpus, agreement was computed at the end of each day of the training period, disagreements were settled in consensus sessions and the guidelines gained clarified rules and examples [4].
Agreement tests how evenly the guidelines are applied, and the coefficients are set out in measuring label quality with inter-annotator agreement. Training matters as well. In the meta-analysis, trained annotators reached 81% average agreement, against 70% for untrained ones [5].
Version the guidelines and record each change
Universal Dependencies marks its notable guideline changes as amendments, which negate an earlier policy, or clarifications, which resolve an ambiguity without negating one, with a third type for changes to its validator [22]. From December 2021 to June 2026 it listed 18 notable changes, 7 of them amendments and 10 clarifications (derived) [22]. The rater guidelines carry a change log of updated definitions and added or removed examples [13]. Hovy and Lavid advise that later changes never supersede earlier decisions [2], and the audit warns that an update may mean discarding or reviewing labels already made [1]. One cloud labeling service locks the instructions once a job starts, so a change means cloning the job [14]. A test set scored against labels, as in testing document extraction accuracy before you buy, depends on the guideline version its answer key followed.
Annotation guidelines as model prompts
The same text can instruct a model. Pangakis and colleagues replicated 27 annotation tasks with a commercial chat model and found a median F1 of 0.707, ranging from 0.059 to 0.969 [24]. Their workflow gives people and the model the same codebook, as the model's prompt, refined on one human-labeled subset and tested on the rest [24]. In an ablation, a model trained to follow annotation guidelines scored a mean zero-shot F1 of 55.3, and 42.3 with the guidelines and every other component removed [12]. Open-weight models of 7 to 12 billion parameters had limits following political science codebooks zero-shot, and were sensitive to the order in which categories were described [7]. A prompt is one more version to record, as dictionaries, FinBERT or language models for financial sentiment shows for sentiment labels.
What to publish with the dataset
One dataset hub's dataset card template asks for a description of the annotation process, including the guidelines provided and agreement statistics [27]. The audit suggests publishing the raw annotations beside the adjudicated labels [1]. The guidelines belong in the dataset's specification from the start, as how to specify a training dataset before anyone builds it sets out.
Rillor's data labeling and annotation services work from written guidelines and worked examples agreed with you before labeling starts, and each label records which version of the guidelines it followed.
Read the full report
The full report adds a guideline template and an example entry built from published rules, a checklist to run before labeling starts, the experiments on instructions, examples and training in detail, the model studies and all 30 sources.