Inter-annotator agreement measures how often people who label the same items independently give them the same label, beyond the agreement chance alone would produce [1]. It shows whether a labeling process is reproducible. It does not show whether the labels are right, since two annotators can share the same mistake [1, 2]. A label quality report needs both parts: an agreement coefficient that fits the annotation design, and an error rate measured on a reviewed sample [2].
Published datasets often leave these out. An audit of 591 dataset papers found that 52% of those with human annotators reported any agreement measure and 18% estimated an error rate, and it found common errors in how agreement was used and how error rates were computed [2]. A study of 10 widely used test sets estimated at least 3.3% label errors on average, and at least 6% in the ImageNet validation set [5].
What inter-annotator agreement measures
Artstein and Poesio call data reliable when coders can be shown to agree on the categories they assign, to the extent the study's purpose requires [1]. When coders agree consistently, the authors infer that the coders share a similar understanding of the annotation guidelines [1]. Agreement is evidence about the guidelines and how evenly people apply them [1].
Percent agreement is the starting point, and it is biased. With two equally likely categories, coders choosing at random agree on half the items [1]. When 95% of items belong to one class, chance agreement reaches 90.5%, so an observed 90% is worse than chance [1]. Chance-corrected coefficients subtract expected agreement. Cohen's kappa is (p_o − p_e) / (1 − p_e), where p_e comes from each annotator's own label distribution [10]. In one published example, 94% raw agreement became a kappa of 0.85 [12].
Cohen's kappa vs Krippendorff's alpha
The coefficients differ in how they estimate chance. Scott's pi pools both coders' label distributions and Cohen's kappa uses each coder's own [1]. Krippendorff's alpha compares observed with expected disagreement, and it handles any number of annotators, any level of measurement and missing labels [8]. On one table with 88% raw agreement, five coefficients range from about 0.7995 to 0.82, all close to the 0.8 the survey suggests where a threshold is needed [1]. On that table pi falls just under 0.8 and kappa just over it [1].
Kappa has two known paradoxes. For the same raw agreement, it rises when the two coders' label distributions differ more [1], and it can fall far below raw agreement when one category dominates [9, 13]. The survey's test for such data is whether coders agree on the rare categories [1].
The audit gives a plain rule. Alpha "can be used in almost all circumstances" and is recommended. Cohen's kappa can be added for two annotators who label every item, and Fleiss' kappa for more annotators only if they are randomly assigned to items. Percent agreement should rarely be used and never alone [2].
| Annotation design | Coefficient | Basis |
|---|---|---|
| Two annotators label every item, unordered labels | Cohen's kappa or alpha | [2, 9] |
| Same number of annotators per item, randomly assigned | Fleiss' kappa or alpha | [2, 11] |
| Varying annotators per item or missing labels | Krippendorff's alpha | [2, 8] |
| Ordered labels such as high, medium and low | Weighted kappa or ordinal alpha | [8, 9] |
| Scores on a numeric scale | Intraclass correlation (ICC) or interval alpha | [2, 8, 15] |
| Entities and spans in text | Pairwise F1 or unitized alpha | [2, 3, 16] |
The metric has to match the label type. In Krippendorff's own example, the same data give alpha of 0.743 when read as nominal labels and 0.849 as interval scores [8].
Spans, ratings and preferences
Studies that mark phrases usually lack a well-defined count of negative cases, which kappa needs [3]. Pairwise F1 equals positive specific agreement, and kappa approaches it as negatives grow numerous [3]. On one archaeology test document, kappa was 0.82 on all tokens and 0.67 on annotated tokens, against an F1 of 0.95 [16].
For scores on a scale, correlation shows only whether two raters' scores move together. Two raters with opposite biases can correlate highly and still disagree [2]. The ICC or interval alpha measures agreement itself [2, 8]. For choices between two model responses, chance is high. On MT-Bench, random judges agree 50% of the time when ties are left out, and expert humans agreed with each other on 81% of first-turn votes without ties [4].
What agreement does not tell you
The audit states that high agreement "does not automatically guarantee high-quality labels" [2]. Passonneau and Carpenter call standard agreement measures neither necessary nor sufficient for a high-quality corpus [20]. Reidsma and Carletta show that patterned disagreement can make a model's performance figures look better than they are, so even a value of 0.8 does not guarantee that good-looking performance is real [21].
Thresholds are conventions. Landis and Koch's ranges call 0.61 to 0.80 substantial and 0.81 to 1.00 almost perfect [2, 9]. Content analysis researchers generally treat values above 0.8 as good reliability and 0.67 to 0.8 as enough for tentative conclusions [1]. The audit recommends against borrowing thresholds from the literature [2].
A figure also needs its sample. In the audit, 51% of the agreement values computed on a subset came from 200 items or fewer, and the authors found no confidence interval for any agreement value [2]. To estimate an agreement near 0.8 within ±0.05 at 95% confidence, the audit puts the minimum near 500 items [2]. Annotators must also work independently, since discussion or consensus by majority vote makes the data unusable for measuring reproducibility [1].
Self-consistency is a separate measure. In one 2023 study, two runs of a commercial chat model agreed on about 97% of labels on average at a temperature of 0.2, against about 79% for trained annotators [22]. The model's figure measures run-to-run consistency, and the study measured accuracy against the trained annotators [22].
Label errors in published test sets
Northcutt, Athalye and Mueller showed each flagged label to five independent workers [5]. Error shares ranged from 0.15% in MNIST to 10.12% in QuickDraw [5]. On average 51% of the flagged candidates were confirmed [5], so a tool such as confident learning produces candidates for review [5, 24]. On corrected ImageNet labels, ResNet-18 outperforms the larger ResNet-50 if the share of mislabeled test examples rises by just 6% [5]. Audits of agent benchmarks find flawed tests too, as covered in what agent benchmarks measure, and what they miss. A pilot that scores a tool against labeled documents should check its answer key first, as testing document extraction accuracy sets out.
Small samples mislead on error rates as well. One dataset's authors estimated 6.7% errors from 300 instances, and a later full inspection found 23.9% [2]. The audit recommends inspecting at least 500 instances or the whole dataset, whichever is smaller [2].
Crowd, expert and model annotators
In a 2008 study, an average of 4 non-expert labels matched a single expert's agreement on affect recognition [27]. In 2023, a commercial chat model's zero-shot accuracy beat crowd workers' on most tasks across four sets of tweets and news articles, by about 25 percentage points on average [22]. A second 2023 study of 27 tasks found nine where a language model's precision or recall fell below 0.5, and its authors conclude that model labels must be validated against human labels [7]. The alternative annotator test compares a model with at least three human annotators on 50 to 100 examples [28].
When disagreement is data
Some disagreement is real. With 100 annotations per example, models scored near-perfect accuracy where people agreed and barely beat a random guess where they did not [6]. Plank argues that conventional practice assumes a ground truth and neglects genuine variation in human labels [29]. Financial sentiment scores also rise with annotator agreement, as dictionaries, FinBERT or language models for financial sentiment describes.
What a label quality note should report
Drawing on the survey and the audit [1, 2], a label quality note states:
- the number of annotators per item, how they were assigned and whether they worked independently from fixed written guidelines
- the coefficient and why it fits the design
- the sample size, with a confidence interval
- an agreement table per label, so an overall figure does not hide disagreement on rare categories
- the error rate measured on a reviewed sample, with its interval
- how labels were combined, and whether the unaggregated labels are kept [29]
The provenance side of that record is the subject of datasets that state their limits. Rillor's data labeling and annotation services label training data to your schema from written guidelines and worked examples agreed before labeling starts, and report the accuracy measured on a reviewed sample with the delivery.
Read the full report
The full report adds each coefficient's formula and paradoxes, guidance for each annotation design, the error table for all 10 test sets, error detection methods, the disagreement studies and a seven-point reporting list, with all 33 sources.