Entity resolution decides which records, within one source or across many, describe the same person, company or product when the sources share no reliable key. The U.S. Census Bureau used it to deduplicate 2020 Census responses and to link survey respondents to the people the census counted [3]. The task also goes by record linkage, deduplication, object identification and fuzzy matching [1, 8].
The work runs in stages. Fields are aligned and standardized. Blocking limits which pairs of records get compared. Each remaining pair is scored, often with the Fellegi-Sunter model, and linked pairs are grouped into clusters, one per entity [5]. Pairs that score between two thresholds go to a person for review [1].
Common accuracy measures can rank the same results in opposite order [7], so a match rate means little without its truth set, threshold and measure.
Entity resolution vs record linkage
Binette and Steorts, writing in Science Advances in 2022, separate the terms by scope [5].
| Term | What it covers |
|---|---|
| Deduplication | Removing duplicate records within one database |
| Record linkage | Merging databases and removing duplicates across them |
| Entity resolution | Removing duplicates across and within databases at the same time |
| Canonicalization | Building one representative record per resolved entity |
In practice the words overlap, and studies of language models call the pairwise step entity matching [3, 10].
Why exact joins miss matches
Ordinary typing errors defeat exact comparison. In a major census application for measuring undercount, more than 25% of true matches would not have been found by character-by-character matching [1].
Businesses are harder. Winkler's Census Bureau overview shows one gasoline station listed under its location name, its owner's name and home address, and an abbreviated corporate name in care of its accountant [1]. In some business lists, Winkler estimates that possibly 40% to 60% of true matching pairs have names and addresses that both differ significantly [1]. Names change too. The SEC record for CIK 0001326801 reads "Meta Platforms, Inc." and lists the former name "Facebook Inc" from May 2005 to October 2021 [25, 26].
Blocking: cutting the number of comparisons
Comparing every record with every other grows with the square of the file. Deduplicating one million records means about 500 billion pairs [28]. Blocking compares only pairs that agree on a key, such as a surname or a date of birth [1]. The cost is recall. The slightest difference in two duplicates' blocking keys puts them in different blocks [6], and a true match that no rule brings together is never compared [28].
Several passes with different keys help. The Census Bureau's 2024 test ran two of its three tools on the 2020 Post-Enumeration Survey's four passes, from phone number to a broad geographic and demographic key [3]. Winkler reports that ten passes over a 300-million-record census kept 99.5% of true matches in about one trillion of the roughly 100 quadrillion possible pairs [1].
The Fellegi-Sunter model scores each pair
The Census Bureau has used the Fellegi-Sunter framework since the 1980s [3]. For each pair, the model compares how likely the observed field agreements are among true matches with how likely they are among non-matches. That ratio, or its logarithm, is the match weight [1].
The documentation of Splink, an open-source package from the UK Ministry of Justice, calls these the m and u probabilities. The m probability largely measures data quality, and the u probability measures coincidence: with ages spread evenly over about 30 years, two unrelated records share a date of birth about 0.01% of the time, and a sex about half the time. Fields are treated as independent, so their weights add up [29].
Agreement on a rare value counts for more. Winkler's example is the surname Martinez, relatively rare in Minneapolis and common in Los Angeles [1]. Splink adds a term-frequency adjustment to each comparison [30]. String comparators such as Jaro's give partial credit for insertions, deletions and transpositions [1].
The clerical review band
The decision rule has three outcomes. A pair above the upper threshold is a match, a pair below the lower threshold is a non-match, and a pair in between is held for clerical review. The thresholds come from error bounds on false matches and false non-matches set in advance [1]. The Census Bureau's Person Identification Validation System sets its cutoffs conservatively, because its identifier links one person across many files [4].
The Post-Enumeration Survey's blocking scheme reduces falsely discovered links to keep that review workload manageable [3]. Reviewed pairs also serve as training data: Winkler's semi-supervised method combines less than 0.5% of pairs, labeled from the review region, with unlabeled data [1].
From matched pairs to entities
Pairwise decisions can contradict each other. If one record links to a second and the second to a third, all three belong to one entity, even if the first and third scored as a non-match. Clustering settles this. One early method takes the connected components of the graph of links, also called transitive closure [5, 9]. A 2009 study on synthetic data found that it produced poor duplicate groups, even against other algorithms of equal speed [9]. In a worked example in the Splink documentation, raising the threshold from 0.95 to 0.99 and then 0.999 splits one cluster into two and then three [32].
Company name matching and registry identifiers
A shared identifier helps where both sources carry it. The Legal Entity Identifier is a 20-character code under ISO 17442, and each LEI represents only one legal entity [18]. Its records carry a primary legal name, other names and transliterated names [19]. On 7 November 2026 the Global LEI Index held 3,487,659 records: 2,001,906 issued, 1,217,492 lapsed and 6,518 marked as duplicate registrations [20]. A lapsed LEI is still valid and means the entity is behind on renewal [21].
The LEI does not identify people acting in a private capacity [22]. Legal forms also vary: the ISO 20275 list holds more than 3,600 of them across more than 200 jurisdictions, each in its local language [24]. For awards reported across several federal systems, see where federal contract data comes from.
Entity matching with large language models
Peeters, Steiner and Bizer (EDBT 2025) tested three hosted models from one provider and three open-weight models on six public product and bibliographic benchmarks of about 1,250 test pairs each [10]. Without task-specific training data, the strongest hosted model reached 89% F1 or more on five of the six, beating fine-tuned language-model matchers on three benchmarks and trailing on three. On products unseen in training, fine-tuned matchers lost 22 to 61 F1 points, and every model tested beat the best transferred matcher by at least 8 points. No single prompt worked best across models and datasets [10].
A 2023 re-evaluation, published at ICDE 2024, found only 4 of 13 established benchmark datasets challenging [12]. Cost grows with the number of pairs, and batching questions into one prompt cut cost four to seven times in one study [11]. A May 2026 study across three models and ten datasets found the models' own explanations of match decisions often unstable and weakly faithful [14].
How to evaluate entity resolution
Evaluation needs a truth set, usually pairs labeled by people, and such sets rarely exceed several thousand samples [8]. Metrics that count correctly rejected pairs flatter large databases, because non-matching pairs grow with the square of the records [8]. Plain accuracy has the same flaw [34].
Measures can disagree. Menestrina, Whang and Garcia-Molina (VLDB 2010) found one pair of measures ranking 52 of 100 result pairs in opposite order on synthetic data, and conflicts on real data too [7]. Barnes recommends reporting pairwise F1, a cluster measure and generalized merge distance [8]. A buyer can ask for the truth set behind any accuracy figure, its threshold, and one pairwise and one cluster measure [7, 8].
The Census Bureau's 2024 test shows how far tools differ on the same 1,167,705 records and truth set of 3,941 clerically reviewed links, each at its own F1-optimal cutoff [3].
| Tool | Seconds | Precision | Recall | F1 |
|---|---|---|---|---|
| BigMatch | 38 | 0.8733 | 0.7161 | 0.7869 |
| fastLink | 44,161 | 0.9636 | 0.4568 | 0.6198 |
| Splink | 40 | 0.7325 | 0.6336 | 0.6795 |
The authors note possible bias toward BigMatch, the Census Bureau's own tool, because software that incorporates it did the first round of deduplication in the 2020 Post-Enumeration Survey [3].
Keep each match decision on the record
Linkage errors become analysis errors. A missed link splits one person into several, and a false link merges two people into one [17]. The GUILD guidance asks data linkers to share record-level indicators, such as the pass in which a pair linked and its match score, with the threshold and cleaning rules, so analysts can adjust for linkage error [16]. The LEI index keeps each duplicate registration, marked DUPLICATE [19, 20]. Those decisions belong next to the provenance of each record.
Rillor scopes, builds and maintains custom datasets to a client's requirement, including the entity types that must resolve and join, and reconciles identities without erasing the source record. Each comes with its schema, records with resolved identities, source provenance and lineage.
Read the full report
The full report details the decision rule, blocking measures and clustering evidence, sets the language-model results against benchmark difficulty, and lists all 35 sources.