To test document extraction accuracy before you buy, run each candidate tool, whether an intelligent document processing product or an LLM, on a labeled sample of your own documents and score every field with a rule written for its data type. Report results by field and by record, with unseen layouts kept apart. Then check that the tool's confidence scores predict its errors, because those scores decide which fields a person reviews.
A single accuracy figure hides the choices behind it. In the DocVQA benchmark (2020), the same human answers score 0.981 on one published rule and 94.36% on another [1]. One model's output on business cards scores 57.8 on a strict field-level F1 and 84.4 on a document-level tree similarity (2022) [10]. One cloud provider's document-processing service reports precision, recall and F1 and no accuracy metric, because optional and repeated fields make accuracy "less meaningful" [12].
Build the test set from your own documents
The VRDU authors argue that public benchmarks miss problems met in practice, such as unseen templates and complex schemas [6]. Public test documents may also sit in a model's training data, which inflates its score on them [33]. Use your own documents.
Hold out layouts. DocILE's test set is 25% layouts unseen in training, 25% seen up to three times and 50% seen more often [9]. Its text-only baseline scored F1 of 0.524 on unseen layouts and 0.742 on well-seen ones (2023) [9]. Include scanned and born-digital copies, and documents where a field is absent, since a correct empty answer is part of the test [4, 23].
Check the answer key too. A study of ten benchmark test sets estimated at least 3.3% label errors on average (2021) [32]. Review each tool-label disagreement before counting it as an error.
Field-level scoring rules: exact match, normalization and ANLS
Exact match is the strict baseline [8]. ANLS gives partial credit: one minus the normalized edit distance, set to zero once that distance reaches 0.5 [2]. DocVQA made it the primary metric because "no OCR is perfect" [1].
Partial credit misfires on numbers and dates. Scored as strings, 0.2 against 0.199999999 earns 0.0, and one date written two ways earns 0.58 [3]. VRDU converts prices to numbers and parses dates before comparing them [6]. In one Kleister dataset (2021), about 5% of income and spending values need their scale, such as thousands or millions, read from context [7]. Published rule sets disagree: VRDU treats "4" and "4.0" as equal [6], and one provider's guide states that 1 and 1.00 do not match [12]. Write yours down.
| Field type | Example rule | Basis |
|---|---|---|
| Identifiers and addresses | Exact match after normalizing case, spaces and punctuation | [6, 8, 12] |
| Amounts | Parse to a number, then compare value, currency and scale | [6, 7] |
| Dates | Parse, then compare as dates | [6, 7] |
| Names and free text | Normalized match, or ANLS where OCR noise is expected | [1, 2, 5] |
| Fields absent from the document | Any returned value counts as an error | [3, 4] |
Document extraction accuracy by field and by document
Field-level F1 fails a field when a single character is missed [10]. A record is usable only when every required field is right, so the complete-record rate sits below the field rate. If each of 20 fields is right 98% of the time and errors are independent, a record is fully right 66.8% of the time (0.98^20, an illustration from the binomial formula) [27]. Errors share causes such as unreadable source material and OCR noise [7, 19], so they cluster in some records. Measure the record rate directly.
Averages hide the spread between fields. On Kleister's NDA set, five baseline models scored F1 of 45.73 to 60.33 on the contract term and 92.87 to 96.50 on jurisdiction (2021) [7]. Report each field on its own, then the share of complete records.
Measure recall where a missed field costs most
When the job is to find every clause or line item, report precision and recall. CUAD counts a predicted span as a match when its word overlap with the label, measured by Jaccard similarity, is at least 0.5 [11]. Its authors report precision at 80% and 90% recall and state that "recall is more important than precision" for their task [11]. At about 30% precision, a lawyer reads about 2 irrelevant clauses for every relevant one [11]. Contract results are in what benchmarks show about LLM contract review accuracy.
By default, one provider sets the confidence threshold that maximizes F1, and a higher threshold raises precision and lowers recall [12]. A buyer who fears omissions sets the threshold from a recall target instead. Average precision has its own trap: adding low-scoring predictions can never lower it [9].
Table extraction accuracy: score the structure with TEDS
TEDS, introduced with the PubTabNet dataset, compares tables as HTML trees, so misaligned and empty cells cost edits [13]. A wrong cell span costs a full edit, both in PubTabNet's definition and in the ICDAR 2021 competition that reused it [13, 14]. A table with 90% of its first-row cells shifted scored 34.9% on TEDS and 80.3% on an older adjacency F1 (2019) [13].
Report simple and complex tables apart. In the ICDAR 2021 table competition, each of the nine teams in the results table scored 2.6 to 4.8 points lower on complex tables (derived) [14].
Check page parsing by document type
Text-based extractors read each page through OCR or a PDF parser first [7, 23]. On Kleister's mix of scanned and born-digital files, changing the OCR engine moved one model's F1 by 8.3 points (2021, derived) [7]. OmniDocBench (paper version of March 2025) found pipeline tools strong on academic papers and financial reports, vision-language models stronger on slides and handwritten notes, and a clear drop for every model on complex layouts [16]. The benchmark grew from 981 pages to 1,651 by version 1.7 in April 2026 and changed its scoring formula on the way, so a score needs its version [16, 17].
Test whether the LLM extraction confidence score predicts errors
A calibrated score means that of 100 fields scored 0.8, about 80 are right [20]. One provider's documentation says a field confidence of 0.95 means the prediction is likely correct "19 out of 20 times" [23]. A pilot can test that by grouping fields by confidence and comparing stated with observed accuracy [20].
Published evidence on confidence scores for LLM extraction is thin. One single-author study by an author at Perfios Software Solutions, an extended workshop paper, tested 267 invoice fields with one hosted model (June 2026) [19]. The model failed on 26.6% of the fields processed. Its token probabilities, its stated confidence and agreement across five samples separated right from wrong fields with ROC AUC of 0.705, 0.692 and 0.744 [19]. A trained classifier reached 0.928 by combining token probabilities with a second, differently prompted extraction call and with OCR, image-quality and layout features [19]. OCR features alone, including whether the value appears in the page text, reached 0.896, and at 80% coverage the full classifier's accepted fields were 99.1% right [19]. That is about 2 errors in about 214 fields, and an exact 95% interval still allows an error rate up to 3.3% (derived) [19, 29]. Separate work on reasoning and knowledge tasks finds that language models stating their own confidence tend to be overconfident, mostly in the 80% to 100% range [22]. The value-in-text check applies the idea behind checking LLM citations against their sources.
One provider's human-review service starts review on a confidence band, a missing key you specify or a random sample [24]. The random sample measures errors the confidence score misses.
Size the proof of concept and compare tools
Report every error rate with an interval. With NIST's exact binomial method, zero errors in 100 fields still allows a true error rate up to 3.62% (95%, derived) [29]. Telling a 10% defect rate from 20% takes about 102 items in NIST's example [30].
Run both tools on the same documents and compare them where they disagree. In a worked example of McNemar's test, two models scored 99.7% and 99.6% on 10,000 cases. With 11 against 1 disagreements the gap was significant (p = 0.0039), and with 25 against 15 it was not (p = 0.1138) [31].
Keep sampling after go-live
A hosted model can change between versions: in one study, a task's accuracy fell from 84% to 51% between the March and June 2023 versions of the same service [34]. Re-run the test set whenever the vendor changes its model or prompt, as in a black-box protocol for evaluating LLM agents.
Between changes, sample the output. Acceptance sampling decides whether a batch is likely acceptable and does not estimate its quality [25]. Under NIST's example plan of 52 items with up to 3 defects allowed, a batch with 10% defects still passes 22.3% of the time [27].
In Rillor's document and regulation intelligence service, a person reviews every record before it is used. The LLM proposes each field beside the passage it came from, the reviewer accepts, corrects or rejects it, and corrections go into the test set for the next extraction run, so a repeated error is caught before review.
Read the full report
The full report adds each metric's formula and origin, the routing and sampling arithmetic, a ten-step pilot checklist with notes for engineers and review teams, and the complete source list.