Using an LLM as annotator means asking a language model to label text that crowd workers, trained coders or experts would otherwise label. Published studies since 2021 find that the results depend on the task. In a 2023 study of tweets and news articles, a commercial chat model given instructions and no examples was about 25 percentage points more accurate than crowd workers on average [3]. The same year, a replication of 27 social science annotation tasks found nine where model labels had precision or recall below 0.5 [7], and a 2025 study of expert datasets reported "a notable gap between current LLMs and human experts" [12].
The full report gathers 34 papers and keeps each figure with its task, model family and year. It pools nothing, because the studies measure agreement in different ways. Some score accuracy against labels set by trained annotators or experts [3, 12]. Some measure chance-corrected agreement with annotators, such as kappa [6, 15]. Some report how often a model repeats its own labels [3, 8]. The coefficients themselves are explained in measuring label quality with inter-annotator agreement.
How an LLM as annotator compares with crowd workers
The 2023 study by Gilardi, Alizadeh and Kubli covered 6,183 tweets and news articles labeled for relevance, stance, topics and frames, with the authors' trained annotators as the reference [3]. On relevance alone, the model's accuracy ranged from 59% to 83% across the four datasets [3]. Its cost per annotation was under $0.003, about thirty times cheaper than the crowd platform [3].
A second 2023 study gave a later commercial model 500 tweets by US politicians during the 2020 election and asked for each author's party, a task with a known answer [4]. The model outperformed every individual human classifier and the combined crowd, and its bias did not differ significantly from the experts', while the crowd workers were significantly more biased toward guessing Democrat [4].
One method can win on one task and lose on the next. A 2024 method that has the model explain labeled examples before it annotates reached 75.60 accuracy on query-keyword relevance against 71.5 for crowd annotators, and 69.17 on word sense against a human figure of 80.0 [11].
Earlier studies priced model labels as training data. In 2021, labels from a commercial model cost 50% to 96% less than human labels for the same downstream performance [1]. In a 2023 sentiment test, 6,000 model labels trained a better classifier than 3,000 human labels for about 10% of the cost [2].
Agreement changes by task and label
Relabeling five social computing datasets, a commercial chat model's accuracy ranged from 0.571 on hate speech to 0.649 on sentiment [5]. Inside the hate speech task, its precision was 0.353 on tweets containing anti-Asian hate speech and 0.791 on tweets containing counterarguments to it [5]. A 2024 evaluation of 13 models on 25 English benchmarks found fair to strong agreement with humans on 12 of 17 classification tasks, and unusably low performance on some [6].
Across 27 replicated tasks, median accuracy was 0.850 and median F1 0.707, but F1 ran from 0.059 to 0.969 [7]. On two binary tweet tasks, a supervised classifier generally beat both open-source models and a commercial chat model, whose recall fell as low as 0.1 in one setting [9]. How model labels compare with dictionaries and fine-tuned encoders on financial text is the subject of dictionaries, FinBERT or language models for financial sentiment.
Expert, span and long-text work
Expert labels show a clear gap. Across five finance, law and biomedicine datasets with expert consensus labels, four instruction-tuned models averaged 64.2% to 68.7% accuracy, chain-of-thought prompting brought only marginal or negative gains, and reasoning models were not significantly better in most settings [12].
Span annotation, where the annotator marks the exact words that carry a label, is mixed. On propaganda techniques, the top model's agreement with annotators reached γ = 0.16, against γ = 0.31 between experts in an earlier study [13]. On errors in text generated from data, three models mostly reached or surpassed human agreement, and annotating 1,000 outputs cost about $500 with crowd workers against $3.60 with a reasoning model [13].
Long, specialized texts are harder. Seven models coding 121 ritual features across 567 ethnographic excerpts reached mean F1 of 0.12 to 0.41 [14]. In a Mandarin study of spoken narratives, the best-performing model reached κ = 0.794 with trained raters against 0.872 between people and cut annotation time by 65% [15].
Whose agreement: subjective labels
On subjective tasks, "agreement with people" depends on which people. Most of nine models tested on politeness and offensiveness matched White participants' labels more closely than Asian or Black participants', and prompting the model to answer as a member of a group did not consistently help [17]. In interviews with Rohingya refugees, a model's coding errors were not random with respect to who was interviewed [18]. When a commercial chat model extracted stereotypes from community survey responses, over thirty percent of its predicates for one benchmark expressed a bias that was not in the response [16]. Plank argues that annotation projects neglect genuine variation between annotators, and that it affects data, modeling and evaluation [19].
Consistent is not the same as correct
A model that repeats its labels can still be wrong. On average, two runs of a commercial chat model agreed on 91% of items at temperature 1 and 97% at temperature 0.2, against 79% for two trained annotators [3]. The authors of the 13-model evaluation argue that ensembling does not fix such errors, because models agree with themselves even when inaccurate [6]. Wording moves labels too: across 45 pairs of instructions for one news task, Krippendorff's alpha between the label sets ran from 0.24 to 0.7 [8]. A hosted model can also change under the same name: outside annotation, one fell from 84% to 51% accuracy on a math task between its March and June 2023 versions [20].
Agreement with people is not the last word either. In a 2026 preprint, model codes for educator messages overlapped less with human codes than human codes did with each other, mean Jaccard 0.30 against 0.52, yet a blind expert preferred the two at indistinguishable rates [26]. A 2026 paper on model judges showed protocol choices alone moving one reported accuracy from 0.551 to 0.899 without a single verdict changing [27].
Using model labels without biasing the result
Model labels used as data can bias the answer. Treating them as true labels in statistical analyses produced substantial bias and invalid confidence intervals even at 80% to 90% accuracy [22]. Replicating 37 annotation tasks with 18 models, one 2025 study found incorrect conclusions on about 31% of hypotheses with state-of-the-art models and on half with smaller ones [21].
The correction is a smaller set of human labels. Design-based supervised learning and prediction-powered inference combine model labels with human labels and keep confidence intervals valid [22, 23]. Confidence-Driven Inference picks which items people label, and in each of three settings (politeness, stance and media bias) it cut the human annotations needed by over 25% [24]. Prediction-powered inference also corrects a model judge's pass rate, as validating an LLM judge against human labels explains.
For training data the picture is closer. Across 14 social science tasks, classifiers fine-tuned on model labels scored a median F1 only 0.039 below those fine-tuned on human labels [33]. Two set-ups trained on model labels had significantly better recall, and the highest precision came from a classifier trained on human labels [33].
People who review model suggestions
Showing people a model's suggestion changes what they label. In a pre-registered 2025 experiment with 410 annotators and over 7,000 annotations, suggestions did not make them faster but raised their self-reported confidence [30]. Crowd labels matching the model's suggestion rose from 40% without help to 81% to 87% with it, and the model's measured F1 against those labels rose by about 0.32 [30]. Separately, in a 2023 summarization task on a paid crowd platform, an estimated 33% to 46% of crowd workers used a language model [29]. The authors call generalization to other tasks unclear [29].
Checking an LLM annotator before use
The studies point to a short set of checks:
- Have people label a validation sample from your own task, under the same guidelines [7, 25].
- Measure chance-corrected agreement and accuracy, with precision and recall for each label [5, 6, 7].
- On subjective labels, compare agreement across annotator groups [17].
- Record the model, its version, the prompt, the settings and the date, and repeat runs and prompt wordings [8, 20, 21].
- For estimates, keep human labels and apply a correction [22, 23, 24].
- When people review model suggestions, keep an unassisted reference set [30].
Rillor's data labeling and annotation service delivers training data labeled to your schema, then reviewed, with accuracy measured on a reviewed sample and reported with the delivery. Testing model output against a source applies beyond labels, as how to check LLM citations against their sources shows for cited answers.
Read the full report
The full report adds each study's tasks, datasets and measures, the tables behind these figures, the statistics of correcting estimates made with model labels, and where the evidence stops.