Near duplicate detection finds documents in a training set that repeat the same text with small changes, such as a timestamp, an advert or a name swapped into a template [3, 18]. Exact hashing catches identical copies. Near duplicate detection scores how much two documents overlap and flags the pairs above a threshold, with MinHash for word overlap, SimHash for weighted term overlap or embeddings for text that says the same thing in other words [3, 5, 18]. Each method has its own definition of similarity, and each threshold decides what is removed.
Duplicates cost more than disk space. Lee and colleagues found duplicates in all four common language datasets they studied, and text from all four validation sets inside the training data [6]. A sequence present 10 times in training data is generated about 1,000 times more often on average than one present once [7]. In a 2022 study of repeated data, repeating 0.1% of the data 100 times degraded an 800-million-parameter model to the performance of a model half its size [9].
Near duplicate detection, defined
Broder defined two measures in 1997 [1]. A document becomes a set of shingles, the runs of w consecutive words it contains. Resemblance is the overlap of two shingle sets divided by their union, the Jaccard similarity, and captures "roughly the same". Containment captures "roughly contained" [1].
Shingle size changes the score. In Broder's example, "a rose is a rose is a rose" and "a rose is a flower which is a rose" resemble each other 60%, 50% or 42.85% at shingle sizes 1, 2 and 3 [1]. Larger shingles respond more to reordering, and "the change in one token affects w shingles" [1]. Resemblance is also not transitive: each version of a paper may resemble the next while "version 100 is probably quite far from version 1" [1]. A chain of matches can join documents that are not alike.
Exact duplicates first
Exact methods need no similarity score. C4 discarded all but one of any three-sentence span that occurred more than once [13]. Dolma's web pipeline removed 53.2% of documents by exact URL, then 14.9% of the rest as identical documents, then 18.7% of paragraphs [15]. The paragraph stage runs last because "paragraph removal risks disrupting content analysis" [15].
Exact substring removal goes further. Lee and colleagues built a suffix array and removed any shared run of at least 50 tokens [6]. In C4, 7.18% of training tokens sat in such runs [6]. The suffix array for the 350GB C4 took 1.5TB of space and under 12 hours to build on one 96-core machine [6].
MinHash LSH deduplication
MinHash applies random hash functions to a document's shingles and keeps the minimum of each, and two documents' minimums collide with a probability set by their Jaccard similarity [5, 28]. Locality-sensitive hashing (LSH) splits the signature into b bands of r values. Two documents become candidates when they agree on every value of at least one band [26]. A pair with similarity s is caught with probability 1 - (1 - s^r)^b, an S-curve that rises from near 0 to near 1 around the threshold [26].
FineWeb used 5-word shingles and 112 hashes in 14 bands of 8, aiming at 75% similarity [10]. Pairs at 0.7, 0.75, 0.8 and 0.85 are caught 56%, 77%, 92% and 98.8% of the time [10]. RefinedWeb used 9,000 hashes in 450 bands of 20, which catches pairs at 0.75 and 0.8 with probability 76% and 99.4% [12]. More hashes give a steeper curve and cost more compute and storage [10].
| Corpus | Signature | Rule | Reported result |
|---|---|---|---|
| The Pile, two web subsets | 10 hashes | Jaccard 0.5 | Duplicate rates of 28% and 26% [14] |
| C4, Lee and colleagues | 5-grams, 9,000 hashes | Jaccard 0.8, then edit similarity 0.8 | 3.04% of training examples had a near duplicate [6] |
| FineWeb | 5-grams, 112 hashes | 75% similarity, each snapshot apart | 20 trillion tokens kept [10] |
| The Stack | 256 permutations | Jaccard above 0.85 | 38.6% of files removed [16] |
Matches then become a graph, and each connected component becomes a cluster with one copy kept [6, 12]. If A matches B and B matches C, A, B and C form one cluster [12]. The Stack trimmed its clusters so that every original file stays similar to at least one file kept [16]. Entity resolution faces the same all-pairs cost and clustering choice, as how entity resolution matches records shows.
MinHash vs SimHash
SimHash comes from Charikar's random-hyperplane method. Each document gets a short bit string, and the Hamming distance between two strings estimates the angle between their weighted term vectors, a cosine measure [2, 3]. For a web search engine's crawler, Manku and colleagues found 64-bit fingerprints within 3 bits of each other reasonable for 8 billion pages, with precision and recall both near 0.75 [3].
MinHash's collisions follow resemblance, and SimHash's follow cosine similarity [5]. On 1.6 billion distinct web pages, Henzinger found precision of 0.50 for Charikar's random projection and 0.38 for shingling, both low on pairs from the same site [4]. In retrieval experiments, Shrivastava and Li found that MinHash "virtually always outperforms SimHash when the data are binary" [5]. None of the sources reviewed compares the two on one training corpus with model results.
Semantic deduplication with embeddings
Embeddings catch pairs that share meaning without sharing words. SemDeDup embeds each item with a pre-trained model, clusters the embeddings with k-means and treats pairs above a cosine threshold within a cluster as duplicates [18]. In the LAION-440M image set, 30% of images had a semantic duplicate at a cosine distance of 0.00095 and 50% at 0.03 [18]. Removing 37% of the set caused no performance drop, and removing 50% a drop under 0.5% [18]. On C4 text at 4% pruning, SemDeDup and Lee's MinHash method performed comparably [18].
About a third of LAION-2B's images, roughly 700 million, are duplicates, found with compressed image embeddings [20]. In a study of image generation models, training images were extracted "most often when they have been duplicated at least k = 100 times" [21].
Choosing a threshold and testing the detector
A threshold is a trade between two errors. Manku and colleagues sampled equal numbers of pairs at each Hamming distance from 1 to 10 and labeled each pair a true positive, a false positive or unknown [3]. Lee and colleagues plotted the similarity of all colliding pairs, reading few pairs near the threshold as a sign that most near duplicates above it were caught [6]. The labeled-sample method in testing document extraction accuracy applies here too: precision and recall on pairs a person has judged.
Boilerplate is a common false positive. Henzinger counted two shopping pages with shared boilerplate and different products as a wrong pair [4]. The 2020 paper on a 175-billion-parameter model ignored 13-grams that matched more than 10 training documents, mostly "common cultural phrases, legal boilerplate, or similar content" [11].
Recall is harder to measure. Henzinger could not determine it across 1.6 billion pages [4]. Small samples also hide duplication: with every document repeated 100 times in 20 trillion tokens, a 1-billion-token sample shows almost every document as unique [10].
When more deduplication stopped helping
FineWeb first ran MinHash globally across all 96 of its crawl snapshots, deduplicating each snapshot against the newer ones, and the result "showed little improvement over a model trained on the non-deduplicated data" [10]. Run on each snapshot apart, it matched RefinedWeb's performance [10]. Its authors suggest the gain comes from removing clusters of hundreds of thousands of copies, and that deduplicating clusters of fewer than about 100 "can harm performance" [10]. RefinedWeb found that MinHash alone did not match the zero-shot performance of exact substring removal, and that combining the two gave no further gain [12].
Check training data against evaluation sets
In Lee and colleagues' study, 4.6% of C4's validation examples and 14.4% of RealNews' had an approximate duplicate in training, which "could cause evaluation metrics to be unfairly inflated" [6]. The 2020 paper removed each 13-gram shared with a benchmark and a 200-character window around it [11]. Dolma's Bloom filter can be seeded with test examples to flag them during mixing [15]. String checks miss paraphrased test items. With embedding search and a model judging each pair, Yang and colleagues found 8% to 18% of the HumanEval benchmark overlapping with RedPajama-Data-1T and a public code corpus [24]. Contamination is one reason benchmark scores mislead, as what agent benchmarks measure, and what they miss shows. A specification can set the allowed near-duplicate rate against the held-out split as an acceptance test, as in how to specify a training dataset.
What to record when a duplicate is removed
Each removal needs a record a reviewer can check. FineWeb and RefinedWeb kept one randomly chosen document per cluster [10, 12], and SemDeDup kept the item with the lowest cosine similarity to its cluster's centroid [18]. One open-source curation toolkit identifies duplicates first and removes them in a separate step [29]. Datasheets for Datasets asks of every dataset whether there are "any errors, sources of noise, or redundancies" and asks for a description [25]. Keeping the cluster, the copy kept, the rule and the threshold for each removal gives that description its evidence.
Rillor's training data cleaning deduplicates, filters and de-identifies data to the client's rules, with exact and near duplicates found and removed.
Read the full report
The full report adds the published settings for each method, computed S-curves for the MinHash settings, the effects measured in training runs, compute costs, a checklist and all 31 sources.