An LLM-as-a-judge setup has one language model grade the outputs of another, and its scores mean something only after the judge has been checked against human labels on the same task. Validating an LLM judge means labeling a sample of outputs by hand, measuring agreement with a chance-corrected statistic, probing for order, length and self-preference bias, and checking again whenever the judge changes. A small set of human labels can then correct the judge's bias in the final estimate.
The published record shows why the check matters. In 2023 tests on MT-Bench and Chatbot Arena, strong judges matched human preferences in more than 80% of cases, as often as two people matched each other [1]. A 2024 study of 13 judges found that even the top judges stayed well behind agreement between people on short factual answers, and that judges with over 90% agreement could still give scores more than 10 points away from human scores [5]. A 2024 study across 20 annotated datasets, accepted at ACL 2025, concludes that judges should be validated against human judgments before use [2].
How LLM-as-a-judge scores go wrong
Agreement depends on the task and on who labels. In the 20-dataset study, all 11 judges correlated more with non-expert labelers than with experts on graded labels, and agreed more when grading human-written text than model-generated text [2]. On JudgeBench (2024), whose labels reflect objective correctness on knowledge, reasoning, math and coding, many strong judges scored only slightly better than random guessing [19].
Judges also carry measurable biases. The figures below come from 2023 and 2024 judges, and a July 2026 preprint found that stronger judges reduce position and length bias without removing it [4].
- Order. With one commercial judge in 2023, swapping the two answers changed the verdict on 66 of 80 queries [3].
- Length. In a 2023 test across 23 answers, two of three judges preferred an answer padded with a rephrased copy of its own list 91.3% of the time [1].
- Self-preference. In 2024, three models told their own summaries apart from other sources with over 50% accuracy and disproportionately favored them [6].
- Leniency. In the 2024 study of 13 judges, most judges tended to mark answers correct when their criteria did not fully match the instructions [5].
- Appended phrases. A 2024 study found that a universal phrase of 5 tokens, added to an answer, pushed judges toward maximum scores [24].
Support labels show the same gap. In the TREC 2024 retrieval-augmented generation (RAG) track, a commercial judge's support label matched the human assessor's on 56% of assessments made from scratch [20]. How to check LLM citations against their sources works through that kind of check claim by claim.
A January 2026 guide from one model provider advises deterministic graders where possible, model graders where necessary and human graders for additional validation, and it lists model graders as needing calibration with human graders [14]. Agent evaluation metrics and what each one tells you sets out where each kind of grader fits in scoring task success.
Build a calibration set from your own task
Sample outputs from the task the judge grades and from the models it grades, and label them by hand. Expect the rubric to move. In a 2024 study of nine practitioners, people refined their grading criteria as they graded and went back to change earlier grades [27]. Pilot the rubric on a first batch, revise it, then freeze it for validation. The January 2026 guide treats a task as well specified when two domain experts would independently reach the same verdict, and suggests grading each rubric dimension with its own judge [14].
Include the cases where judges fail: close pairs, where position bias grows with a small quality gap [21], outputs from the judge's own model family [6] and questions the judge cannot answer itself. In a 2025 finance study, judges without a correct reference showed high agreement with experts only on questions they could answer themselves, and expert-written references largely mitigated the problem [26].
How many labels does that take? The alternative annotator test (2025) compares a judge with at least three human annotators on 50 to 100 items, and its t-test needs at least 30 items, with a Wilcoxon signed-rank test below that [8]. Its authors recommend expert annotators whenever possible [8].
Measure agreement beyond percent agreement
Percent agreement counts matches and ignores chance [25]. Suppose people pass 180 of 200 outputs. A judge that passes every output agrees 90% of the time, and Cohen's kappa, which corrects for chance agreement, is 0 (illustrative, derived) [25]. In the 2024 study of 13 judges, an 8-billion-parameter open-weight judge agreed with humans on well over 80% of answers with a Scott's pi of 59 out of 100 [5]. Judges with Scott's pi above 80 gave scores within 5 points of human scores [5]. The authors recommend reporting both percent agreement and Scott's pi, paired with qualitative analysis [5]. How kappa, Fleiss' kappa and Krippendorff's alpha suit different labeling designs is covered in measuring label quality with inter-annotator agreement.
No source opened for this note sets a kappa threshold for LLM judges. One review written for health research reads any kappa below 0.60 as inadequate [25].
Put an interval on every rate. With 85 agreements in 100 items, the 95% Wilson interval runs from 76.7% to 90.7% (derived) [15]. With 425 in 500 it runs from 81.6% to 87.9% (derived) [15]. For pass and fail labels, one cloud provider's judge-evaluation guide reports the confusion matrix and balanced accuracy [28]. Count false passes apart from false fails, since a lenient judge raises pass rates [5].
Probe order, length and self-preference bias
Each probe reuses calibration items with one change. For order, run each pairwise comparison twice with the answers swapped, and count a win only when the same answer is preferred in both orders, a tie otherwise [1]. The flip rate is the measure. In the July 2026 preprint, flip rates ran from 0.320 for a 1.7-billion-parameter open-weight judge to between 0.117 and 0.147 for four releases of a commercial family, and the last of those releases still changed 14.7% of verdicts [4]. One cloud provider's judge settings include an option that flips the order on half of all calls [18].
For length, pad an answer with restated content and see whether the judge prefers the padded version [1]. A length-controlled version of AlpacaEval (2024) predicts the judge's preference at zero length difference, which raised its Spearman correlation with Chatbot Arena from 0.94 to 0.98 [22]. For self-preference, compare the judge's scores with human labels on outputs from its own family [6]. A 2024 study found that judges scored low-perplexity text higher than human evaluators did, whoever wrote it, which suggests that self-preference comes from familiarity [7].
Re-validate the LLM judge whenever it changes
In the July 2026 preprint, scores moved when only the judge changed and the candidate answers stayed fixed, and the authors conclude that judge upgrades are not interchangeable [4]. A served model can change under the same name too. Between March and June 2023, one commercial model's accuracy at telling prime from composite numbers fell from 84% to 51% [9]. Rerun the calibration set when the judge model or its release changes, after prompt edits, and when the judge moves to a new task [2, 4, 27].
Sampling one judge several times steadies its verdict [18] and adds little accuracy when its errors correlate [4]. In the July 2026 preprint, majority votes of 1, 3 and 5 samples from a 1.7-billion-parameter judge reached LLMBar accuracies of 0.463, 0.475 and 0.482 [4]. A 2024 study found that a panel of smaller judges from different families beat a single large judge [17], while the 2026 preprint found that mixed panels also fell short of what independent errors predict [4].
Correct the final estimate with human labels
A judge's errors can bias its pass rate. Prediction-powered inference (2023) measures the judge's error on a labeled sample and corrects the judge's average over all outputs, with valid confidence intervals [10]. Suppose a judge passes 80% of 2,000 unlabeled outputs. On 200 other outputs that people also labeled, the judge passes 82% and people pass 74%. The corrected pass rate is 80% minus 8 points, or 72% (illustrative, derived) [10].
The saving in labels has limits. Experiments published in 2024 raised the effective human-labeled sample size by up to 50% with one commercial judge [30]. A 2024 analysis shows that when the judge is no more accurate than the model it grades, no correction cuts the labels needed by more than half [11]. Selective evaluation (2024) takes another route: it trusts the judge only when its confidence is high enough and escalates the rest, and on one hard subset it guaranteed over 80% human agreement while judging almost 80% of items [32].
Rillor's agent evaluation work scores task success on cases with known answers, and each finding comes with the hypotheses, the measures and the run logs behind it.
Read the full report
The full report adds the agreement figures by task, each bias measurement with its date, worked kappa and interval examples, a seven-step validation procedure with a probe table, the correction methods and their limits, and all 32 sources.