Data preparation for LLM training is the work between a raw source and the files a training run reads: extracting the text, identifying its language, filtering it, removing duplicates, handling personal data, keeping evaluation sets out and setting the mix. Most of a web crawl does not survive it. The stages behind RefinedWeb removed nearly 90% of the documents originally in Common Crawl [2], and DCLM-Baseline kept 1.4% of the documents in its original pool [6]. Every removal is a choice, so what a buyer needs from a supplier is the record of each one: the rule, the share removed, a sample of what went and the version it produced.
The examples below come from open corpora and open model families. Their shares describe English-heavy web crawls [1, 2, 5], and their thresholds were tested on small models [1, 11]. Each is one paper's setting, and none is a standard.
Data preparation for LLM training, stage by stage
| Stage | A published example | What the buyer should receive |
|---|---|---|
| Source and rights record | WARC files hold the raw crawl, WET files only its text [29] | Sources, capture dates, terms and the raw copy |
| Text extraction | FineWeb extracted WARC files with trafilatura [1] | Extractor, version and sample pages |
| Language identification | A language score of 0.65 or more [1, 2] | Classifier, threshold, share removed |
| Rule-based quality filters | Word count, word length and repetition rules [3] | Each rule and the share it removed |
| Model-based quality filters | DCLM kept the top 10% by classifier score [6] | Classifier, reference data, cut-off, samples |
| Deduplication | URL, document and paragraph stages [5] | Level, settings, share removed per level |
| Personal data | Three identifier types masked [5] | Types handled, method, share affected |
| Decontamination | 8-gram matching against evaluations [13] | A report against the buyer's benchmarks |
| Mixing and repetition | A mix of general, math, code and multilingual text [7] | The mix and how it was chosen |
| Packaging | The datasheet's preprocessing questions [12] | Files, datasheet, removal log, version |
Start from the source record
Preparation enforces a scope that exists first: which sources are in, what each record holds and how the delivered set is accepted. That document is the subject of how to specify a training dataset, and every filter below applies part of it.
Keep the raw copy. Common Crawl's WARC format is "the raw data from the crawl", while WET files "only contain extracted plaintext" [29]. Datasheets for Datasets asks whether the raw data was saved and whether the processing software is available [12]. Terms also move after capture. A 2024 audit of 14,000 web domains found that restrictions added in one year left about 5% or more of all C4 tokens fully restricted, and that terms of service now restrict 45% of C4 [17]. A capture date on each record shows which terms applied.
Text extraction from HTML and PDFs
The FineWeb authors found that WET files "retained too much boilerplate and menu text", and text extracted from WARC files with trafilatura trained a better model in their tests [1]. In DCLM, resiliparse and trafilatura scored at least 2.5 points above WET text, and resiliparse ran 8 times faster [6]. One open-weight model family's technical report removed markdown markers, which it found harmful for a model trained mostly on web data [7].
PDFs add layout and OCR. The olmOCR authors put the cost of their 2025 setup at 176 US dollars per million PDF pages, against over 6,240 dollars through a proprietary vision model's API [23]. Before relying on an extractor, test its accuracy on your own documents.
Language and quality filters
Language identification is often the first large cut. fastText's published models recognize 176 languages [22]. RefinedWeb and FineWeb kept documents with a language score of 0.65 or more [1, 2]. Dolma kept pages with an English score of 0.5 or more, which removed 61.7% of its data by bytes [5]. C4 required a probability of at least 0.99 from langdetect [4].
Rule-based filters are short and written down. The MassiveText rules remove a document with fewer than 50 or more than 100,000 words, a mean word length outside 3 to 10 characters or more than 90% of its lines starting with a bullet [3]. C4 kept only lines ending in terminal punctuation and discarded pages with fewer than 5 sentences [4].
Model-based filters keep what a classifier scores highly. DCLM trained a fastText classifier on instruction data and high-scoring forum posts, then kept the top 10% of documents [6]. For FineWeb-Edu, a 70-billion-parameter model scored 460,000 web pages for educational quality from 0 to 5, and a classifier trained on those scores kept pages at 3 or above [1]. The reference data defines quality, and the Dolma authors write that the term "quality filter" "does not appropriately describe the outcome of filtering a dataset" [5].
Filters have side effects. C4's blocklist of bad words removed African-American English and Hispanic-aligned English text at 42% and 32%, against 6.2% and 7.2% for white-aligned and other English [9]. In a study of 28 models of 1.5 billion parameters, quality filtering raised both toxic generation and downstream performance [11]. What goes in before these filters is covered in what data is used to train language models.
Deduplication at each level
Duplicates are removed at several levels. In Dolma, exact URL deduplication removed 53.2% of documents, exact document deduplication 14.9% of the rest and paragraph deduplication 18.7% of paragraphs [5]. FineWeb found near duplicates with MinHash over 5-grams, 112 hash functions in 14 buckets of 8, aimed at documents at least 75% similar [1].
One 61-word sentence appeared over 60,000 times in C4, and models trained on deduplicated data emitted memorized text ten times less often [10]. Run across all 96 snapshots at once, deduplication removed as much as 90% of the oldest snapshots' data, and FineWeb's test models showed little improvement, so the authors deduplicated each snapshot on its own [1]. The same matching problem in structured records is the subject of how entity resolution matches records.
Personal data and unsafe content
The published methods are narrow. Dolma masked email addresses, IP addresses and phone numbers in 0.02% of documents and removed the 0.001% with denser personal data, and removing instead of masking had no effect on model performance [5]. FineWeb anonymized email and public IP addresses for its public release [1]. The open-weight model family's report removed whole domains holding large amounts of personal data [7]. The EU's template for the public summary of a general-purpose model's training content asks for the measures taken to remove illegal content, "such as blacklists, keywords, and model-based classifiers" [26].
Keep evaluation data out
A 2021 study found 1.87% to 24.88% of the target texts of the generation benchmarks it examined in C4 [9]. The paper that introduced a 175-billion-parameter model treated a 13-gram match as overlap, and reported that a bug in its filter ignored some overlaps and that retraining was not feasible [8]. Tulu 3, an open post-training recipe, flagged a test item when more than 50% of its tokens had 8-gram matches with one training example [13]. DCLM asks every submission to disclose a decontamination report [6], and a buyer can ask for one against the benchmarks they use.
Mixing, repetition and size
After cleaning comes the mix. The open-weight model family's report set a final mix of roughly 50% general knowledge tokens, 25% math and reasoning, 17% code and 8% multilingual, chosen by training small models on candidate mixes [7]. DoReMi set domain weights with a 280-million-parameter proxy model and raised average few-shot accuracy by 6.5 points over The Pile's default weights [19]. Up to 4 epochs of repeated data gave negligible changes to loss compared with unique data [20]. A compute-optimal study found that model size and training tokens should be scaled equally, and trained a 70-billion-parameter model on 1.4 trillion tokens [15].
Published fine-tuning sets are far smaller. LIMA tuned a 65-billion-parameter model on 1,000 curated prompts and responses [14]. The open-weight report scored its post-training data for quality and difficulty and removed semantic near duplicates [7]. Record formats for instruction, chat and preference data are a separate subject.
What the delivery should include
JSON Lines stores one valid JSON value per line [32], and Parquet is a column-oriented format "designed for efficient data storage and retrieval" [31]. A buyer can ask for documentation that answers the datasheet's preprocessing questions [12], in a dataset card stating license, language and size [34] or in Croissant metadata [21], and for a removal log for each stage, the decontamination report and the version.
Disclosure rules ask about this processing. California's AB 2013 asks a developer to post whether there was "any cleaning, processing, or other modification to the datasets" and its purpose [27]. On 4 March 2026 a federal court denied a developer's request for a preliminary injunction against the law, finding that it had not identified a cleaning approach distinct from its competitors' in a way that warranted trade secret protection [28]. Article 10 of the EU AI Act lists "relevant data-preparation processing operations, such as annotation, labelling, cleaning" among data governance practices for "high-risk AI systems" [25]. This summary of the texts is research, never legal advice.
In a Rillor data project, the records are labeled, cleaned and de-identified and their quality checked, and you receive the dataset and its datasheet in the agreed format and channel.
Read the full report
The full report sets out removal shares by pipeline, the filter rules and thresholds, deduplication and decontamination settings, the disclosure texts, a buyer's checklist for each stage and all 35 sources.