Pretraining data filtering decides which crawled pages a language model learns from, and most pages are dropped. Nearly 90% of the documents originally in Common Crawl were removed on the way to RefinedWeb [3], and DCLM-Baseline keeps 1.4% of the documents in its source pool [5]. The filters come in two kinds: hand-written heuristics that test length, symbols, punctuation and repetition, and trained classifiers that score each document against reference text. Published ablations measured what each kind did to the benchmark scores of models trained on the result, and the conditions go with every number.
"Quality" in these papers has no ground truth. One study counts as quality or toxic any document that triggers the classifier [10], and another argues that quality filtering "necessarily implies a value judgment" [9]. The filter ablations reviewed here trained models of up to 8 billion parameters [5, 18].
Where pretraining data filtering sits in the pipeline
Dataset papers run similar stages. RefinedWeb filters URLs against a blocklist of 4.6 million domains, extracts text, identifies the language, applies repetition and quality rules, then removes duplicates [3]. DCLM-Baseline applies an English filter, page length and repetition rules and other filters, then Bloom filter deduplication and a fastText classifier [5].
Text extraction already changes results. In DCLM's tests at 1 billion parameters, the same heuristic pipeline scored 24.1 and 24.5 on its core benchmark set with the resiliparse and trafilatura extractors, against 20.7 on Common Crawl's pre-extracted WET files [5].
Deduplication follows the rule-based stages in both pipelines [3, 5], and near-duplicate detection for training data covers how it works. Gopher's pipeline also removed documents that overlapped benchmark test sets [2], a guard against the contamination discussed in what agent benchmarks measure.
Heuristic filters: C4, the Gopher rules and FineWeb
C4 kept only lines ending in terminal punctuation and lines of at least 5 words, dropped pages with fewer than 3 sentences, and removed pages containing "lorem ipsum", a curly bracket or any word from a list of obscene words [1]. An unfiltered variant, 6.1TB against C4's 745GB, scored lower on every task its authors tested, for example GLUE 81.46 against 83.28 and SuperGLUE 68.04 against 71.36 [1].
The Gopher quality filter, written for the MassiveWeb part of MassiveText, removes documents outside 50 to 100,000 words, with a mean word length outside 3 to 10 characters, with fewer than 80% of words containing a letter, or with fewer than two of eight common English stop words [2]. Its repetition rules remove documents in which duplicate lines make up more than 30% of lines, with similar limits for paragraphs and repeated n-grams [2]. In tests on 1.4-billion-parameter models, adding these filters "dramatically improves downstream performance across the board" [2].
FineWeb tested C4's larger rules one at a time. The terminal punctuation rule gave the biggest single gain but removed about 30% of tokens, so FineWeb kept the other C4 rules it tested, for curly brackets, word length, lorem ipsum, Javascript and policy notices, and dropped that one [4]. It then compared over 50 statistics between two versions of one crawl that FineWeb treated as higher and lower quality, set thresholds where the lower-quality version was denser, and kept three new rules [4]. Together they removed about 22% of tokens and raised the aggregate score by about 1% in 28-billion-token ablations [4]. Dolma combined all the Gopher rules with C4's punctuation rule, a mix that outscored C4's full rule set and the Gopher rules alone in its tests on 1.2-billion-parameter models [11].
Language identification and perplexity filtering
Language identification makes the first large cut. After URL filtering, extraction and language identification, RefinedWeb kept 48% of documents, "mostly filtered out by language identification" [3]. Several pipelines use a fastText model that recognizes 176 languages [3, 7, 33], with different thresholds: 0.5 in CCNet [7], 0.65 in RefinedWeb and FineWeb [3, 4] and 0.3 in Nemotron-CC [18]. FineWeb2 found that no single threshold suits every language. Swahili did well around 0.3, while Arabic and Russian did better above 0.8 [20].
Perplexity filtering scores a document by how predictable a reference language model finds it. CCNet split the pages of 48 languages into head, middle and tail thirds by the perplexity of models trained on Wikipedia, then kept all three, because valid text far from Wikipedia, such as blog comments and specialized forums, landed in the tail [7]. Later studies disagree on which end to keep. One kept the quarter of its data with the lowest perplexity [23], another found that the middle of the range beat the lowest-perplexity subset [16], and a third kept high-perplexity text from the Pile and medium-perplexity text from Dolma [17].
Classifier filters: from the GPT-3 paper to FineWeb-Edu and DCLM
The GPT-3 paper trained a logistic regression classifier with curated text as positives and unfiltered Common Crawl as negatives, then kept documents through a random draw weighted toward high scores, so some low-scoring documents stayed in [6]. The Pile used the same selection rule with a fastText classifier [24]. DSIR resamples documents to match a target distribution, and with Wikipedia and books as the target it improved GLUE by 2 to 2.5% over random selection and heuristic filtering [15].
Newer classifiers learn from labels that a language model writes. FineWeb-Edu had an instruction model of the Llama 3 family rate 460,000 pages for educational value on a scale of 0 to 5, trained a classifier on the ratings and kept pages scoring 3 or more [4]. DCLM compared selection methods at 1 billion parameters [5]. A fastText classifier trained on the OH-2.5 and ELI5 reference sets scored 30.2 on its core benchmarks, against 29.0 for perplexity filtering, 28.6 for prompting a model to rate documents and 27.5 for its heuristic baseline [5]. QuRating had a model of the GPT-3.5 family compare pairs of documents on four qualities [14], and Nemotron-CC takes the maximum of three classifiers' scores and sorts documents into five quality levels [18].
What the ablations measured
| Dataset or study | Filter | Result as reported |
|---|---|---|
| FineWeb-Edu [4] | Educational classifier, threshold 3 | MMLU 33% to 37% and ARC 46% to 57% against FineWeb, 1.71B model, 350B tokens |
| DCLM-Baseline [5] | fastText classifier, top 10% kept | 64% five-shot MMLU, 7B model, 2.6T tokens |
| Nemotron-CC [18] | Three-classifier ensemble, plus synthetic rephrasing | High-quality subset (1.1T tokens, 0.5T of them synthetic) 5.6 MMLU points above DCLM, 8B model, 1T tokens |
| RefinedWeb [3] | Gopher rules and line corrections | 1.6 points zero-shot, 3.5 with deduplication added |
| Perplexity pruning [17] | 125M reference model | Up to 2.04 points average for a 3B model |
Gains were not uniform. RefinedWeb's filtering lowered one OSCAR version's score by 0.4 points while deduplication helped every dataset tested [3], and a perplexity filter on C4 helped only at 4.2 billion parameters [23].
What filters remove that teams wanted to keep
C4's word blocklist removed 42% of documents in African American English and 32% in Hispanic-aligned English, against 6.2% in White-aligned English [8]. A replication of the GPT-3 filter was more likely to rate newspapers from larger schools in wealthier, more educated and urban ZIP codes as high quality [9]. Across 28 models of 1.5 billion parameters, quality filtering raised both benchmark scores and toxic generation, and toxicity filtering cut toxic generation at a cost in generalization [10]. Nemotron-CC found that heuristic filters removed 18.1% of the tokens the FineWeb-Edu classifier rated high quality, and applies them only to the splits its own classifiers rate lower quality [18].
How much to filter
FineWeb-Edu's threshold of 3 removed 92% of its source data, and a stricter threshold raised knowledge and reasoning scores while lowering HellaSwag and PIQA [28]. DCLM found that keeping its top 10% beat keeping 15% or 20% [5]. Nemotron-CC argues that removing 90% of data limits long-horizon training, which reached 15 trillion tokens for the Llama 3.1 family, and its full 6.3-trillion-token dataset matched DCLM on MMLU with four times more unique real tokens [18]. Repetition is the other lever: up to 4 epochs of repeated data changed loss negligibly against unique data [23].
Testing a filter before trusting it
FineWeb trained two 1.71-billion-parameter models per dataset version, identical apart from their data, on about 28 billion tokens, and scored them on benchmarks chosen for low variance between runs [4]. DataDecide found that rankings from 150-million-parameter models predicted which corpus produced the better 1-billion-parameter model in about 80% of comparisons [22]. Agreement with human raters was a weak guide in DCLM: a filter at about 82% ROC-AUC against human labels produced weaker training data than fastText filters at about 73% [5].
The datatrove library returns a named reason for each failed rule and counts dropped documents per reason [31, 32]. Another curation toolkit's documentation advises analyzing score distributions before applying filters, and notes that its default logs report aggregate counts without each record's rejection reason [34]. For record matching, how entity resolution matches records describes keeping each match decision on the record.
Rillor's training data cleaning and filtering filters out records outside a client's scope, by source, date, language or type, to the rules the client sets, and the results of its checks ship with every version of the cleaned data.
Read the full report
The full report adds the thresholds each paper publishes, the ablation tables with model sizes and token budgets, the bias audits and where the sources disagree.