Contract review accuracy for language models depends on the task a test sets. Benchmarks built on the same public set of 510 commercial contracts produce scores from 44.0% to 92.9% [1, 3]. The higher figure is our mean of LegalBench's balanced accuracy on yes-or-no questions about single clauses. The lower one is the precision a 2021 model reached when it had to find 80% of the relevant clauses in whole contracts.
Across eight benchmarks published from 2021 to 2026, models score high when they see one short clause and answer a fixed question. Scores drop when the model must search a long document for every relevant passage, follow an exception stated pages from its rule or catch drafting errors in a finished contract [1, 4, 13, 15].
Comparisons with lawyers rest on small studies. In VLAIR, a February 2025 industry study of four LLM-based legal tools, the top tool scored 75.1 on data extraction against 71.1 for a lawyer control group, and the lawyers led on redlining [21].
Contract review benchmarks, side by side
Each benchmark fixes what the model sees, what it must return and how the answer is scored.
| Benchmark | What the model sees | What it must do | Best result | Metric |
|---|---|---|---|---|
| CUAD, 2021 | A whole contract | Highlight text for 41 clause categories | 44.0% at 80% recall | Precision |
| LegalBench CUAD tasks, 2023 | One clause (task means of 42 to 100 words) | Say whether it belongs to a category | 92.9% | Balanced accuracy, chance 50% |
| ContractNLI, 2021 | A whole NDA | Label 17 statements and cite evidence | 0.892 | Accuracy |
| MAUD, 2023 | Deal-point text a lawyer already extracted | Answer deal-point questions | 57.8% (random 16.8%) | AUPR |
| ContractEval, 2025 | A whole contract | Return the clause text or "no related clause" | 0.644 | F1 |
| CLAUSE, 2026 | An original and an altered contract | Say whether a planted discrepancy exists | 62.8% to 65.4% | F1, CUAD categories |
| ContractScrub, 2026 | A whole contract of 10 to 15 pages | Extract defined terms and find drafting errors in eight categories | 0.750 | Macro recall |
| VLAIR extraction, 2025 | One to seven documents | Extract text verbatim with its location | 75.1 (lawyers 71.1) | Mean question score, graded by a model |
Sources: [1, 3, 4, 5, 9, 13, 15, 21]. The LegalBench figure is our computation [3].
Yes-or-no clause questions score high
LegalBench, a 2023 benchmark from Stanford researchers and collaborators, turned 38 of CUAD's 41 clause categories into balanced yes-or-no tasks [3]. Each item is one clause shown alone, and half the items come from other categories [3]. Averaged over the 38 tasks, the three commercial models tested in mid 2023 scored 92.9%, 89.2% and 88.1% balanced accuracy, by our computation [3]. Five open models of about 13 billion parameters averaged 50.5% to 81.4% [3].
The live LegalBench leaderboard showed a top score of 88.56% on 13 October 2026, with the first three models within 0.26 points [23]. That score spans LegalBench's six reasoning categories, contract clause interpretation among them [3, 23]. The LegalBench authors note that its CUAD, ContractNLI and MAUD tasks were public before the models were trained, so some may have leaked into pretraining data [3].
Clause extraction accuracy on whole contracts
CUAD, published at NeurIPS in 2021, asks a model to highlight the text relevant to each of 41 labels in contracts that run from a few pages to over one hundred [1]. The text for any one label averages about 0.25% of a contract [1]. The best model reached 44.0% precision at 80% recall and 17.8% at 90% recall [1].
ContractEval gave full CUAD test contracts to 19 model configurations, built on 12 distinct models, in 2025 [13]. The best F1 was 0.644 [13]. The top model approached or passed 0.9 F1 on governing law, while both it and the open model compared with it scored near zero on uncapped liability, joint IP ownership and notice period to terminate renewal [13]. The rate of answering "no related clause" when a clause existed ran from 0.000 to 0.306 [13].
Absent clauses make up most of the test. Of the 4,182 questions in CUAD's published test file, 2,938 have no annotated answer, about 70% [17]. An independent re-evaluation of one public model in September 2026 found that it returned text for 45.4% of those questions, while recovering 82.2% of reference spans at 8.2% precision [17]. A score that counts only found clauses leaves that review workload out.
Exceptions and merger agreements
ContractNLI gives a model a whole non-disclosure agreement and 17 statements, such as "Some obligations of Agreement may survive termination of Agreement" [4]. The best 2021 model reached 0.892 accuracy against 0.674 for a majority-vote baseline, and 0.405 F1 on contradictions [4]. Contracts often add an exception to a rule, sometimes pages later [4]. For distant exceptions, accuracy on the less common label fell to 0.00 [4].
MAUD asks 92 questions from the American Bar Association's 2021 deal points study about 152 merger agreements [5]. Lawyers had already extracted the relevant text, so the test covers interpretation of a located passage. The best model reached 57.8% AUPR against 16.8% for random answers [5]. On LegalBench's 34 MAUD tasks, the model that led on its CUAD tasks averaged 47.8% balanced accuracy [3]. Finding the passage is a separate step: a 2024 retrieval benchmark found a basic pipeline reached 25.6% recall on merger-agreement questions with 64 retrieved chunks, against 75.7% on CUAD [18].
Contract scrubbing: the final review
ContractScrub, an August 2026 workshop paper by researchers at a legal-technology company and Imperial College London, tests the last review before signing [15]. Nine lawyers with at least eight years of practice annotated 44 CUAD contracts, logging existing errors and inserting new ones [15]. The best macro recall was 0.750 [15]. F1 scores in the paper's Table 2 run from 0.316 to 0.655 [15].
Errors with explicit word signals were caught far more often than errors that need context. Mean recall was .835 for defined terms, against .427 for incorrect capitalization in context [15]. Recall on wrong section references fell as the distance between the reference and the section it points to grew, most sharply past ten thousand characters [15]. The authors judge that full automation likely needs more than the 75% recall reached so far [15].
CLAUSE, in the Findings of EACL 2026, planted model-generated discrepancies in over 7,500 contracts from CUAD and ContractNLI [9]. Shown both versions of each contract, the top model reached 62.8% to 65.4% F1 on CUAD categories, through recall above 90% and precision below 50% [9].
Contract review tools vs lawyers
Three public studies compare review tools with lawyers on contract work. Each is small or run by an interested party.
| Study | Material | Lawyers | Top tool result | Run by |
|---|---|---|---|---|
| VLAIR extraction, 2025 | 30 questions, 29 documents | 71.1 | 75.1 | Evaluation company with customer ties to participants [21] |
| VLAIR hardest question | 40 fields, three credit agreements | 39.2% | 61.7% | Same study [21] |
| Vendor study, 2024 | 10 procurement contracts | F 0.860, junior lawyers | F 0.871 | Vendor staff, preprint [20] |
| NDA study, 2018 | 5 NDAs, 20 lawyers | 85% | 94% | The vendor tested [25, 26] |
The 2025 study graded answers with a language model and gave failing scores a second review [21]. Its lawyers did not know they were in a study, and led on redlining at 79.7 [21].
Reading a contract review accuracy claim
A scoring rule alone can move a headline number far. On a legal agent benchmark, the leading model passed 27.08% of tasks on 5 November 2026, while 96.30% of its individual grading criteria passed [24]. A task counts only when all of its criteria pass [24].
The benchmarks point to five questions for any claim:
- What did the model see: one clause, a whole contract or text a lawyer had already found [1, 3, 5]?
- What counts as correct, and over which denominator [17]?
- How do absent clauses count, when they make up about 70% of CUAD's test questions [17]?
- What are the scores by clause type, where averages can hide near-zero categories [13]?
- Were the test documents like yours? NIST's AI Risk Management Framework pairs accuracy measurements with realistic test sets that represent expected use [11].
Pinned versions, repeated trials and a test fixed before it runs are set out in a black-box protocol for evaluating LLM agents.
In 2025 the FTC finalized an order barring an online legal service from advertising that it performs like a real lawyer without sufficient evidence [12]. A preregistered study of legal research tools, a different task, found their providers' claims overstated, with hallucination rates of 17% to 33% [10].
Rillor's document and regulation intelligence service uses LLMs to extract parties, defined terms, obligations, conditions and deadlines from contracts as fields, each linked to the exact passage it came from. A person reviews every record before it is used, the method set out in from rule text to a checkable model.
Read the full report
The full report adds each benchmark's design, review status and availability, the inconsistencies in the sources, and a test a buyer can run on their own contracts.