What data is used to train LLMs depends on the stage of training. A language model first reads trillions of tokens of filtered web pages, code, scientific papers, books and encyclopedias. Much smaller sets of written demonstrations, ranked answers and tasks with checkable answers then shape how it responds. Commercial models "rarely detail their data", as the Dolma authors put it [19], so the figures here come from open reports and dataset cards.
What data is used to train LLMs, stage by stage
| Stage | What the data is | One published example |
|---|---|---|
| Pretraining | Filtered web text plus code, papers, books, encyclopedias and math | OLMo 2: about 3.9 trillion tokens, over 95% from web data [2] |
| Mid-training or annealing | Up-sampled high-quality documents, curated sources, synthetic math | OLMo 2: mixes of 50, 100 and 300 billion tokens [2] |
| Supervised fine-tuning | Prompts paired with written demonstrations | About 13,000 training prompts in a 2022 study [3] |
| Preference tuning | Rankings or pairs of model outputs | 33,000 reward model prompts in the same study [3] |
| Reinforcement learning with verifiable rewards | Tasks whose answers a program can check | Math and precise instruction following in Tulu 3 [4] |
Llama 3, one open-weight model family, follows the same arc in its 2024 report. It pretrained on about 15 trillion multilingual tokens, then annealed on small amounts of high-quality code and math [1]. In a test that added the GSM8k and MATH training sets to annealing, an 8B model's scores on those validation sets rose by 24.0% and 6.4%, while the 405B model's gains were negligible [1]. Post-training ran six rounds of supervised fine-tuning and Direct Preference Optimization, on examples from human annotation or generated synthetically [1].
Web text from Common Crawl, filtered
The open web corpora C4, RefinedWeb, FineWeb, DCLM and RedPajama-V2 are all built from Common Crawl [9, 11, 12, 15, 17], a free, open repository of web crawl data whose home page counts "Over 300 billion pages spanning 15 years" [5]. Its October 2026 crawl holds 2.16 billion pages, or 371.19 TiB of uncompressed content, including 830.1 million URLs not visited before [7].
Raw crawl text needs heavy cleaning. The C4 authors wrote that "the majority of the resulting text is not natural language" [9]. Their rules, applied to an April 2019 snapshot, kept only lines ending in terminal punctuation, dropped pages with fewer than three sentences or with listed bad words, kept one copy of any repeated three-sentence span and required an English probability of at least 0.99 [9]. About 750 GB remained [9].
FineWeb re-extracted text from Common Crawl's raw WARC files, because the text-only WET files kept too much boilerplate [12]. It removed near-duplicates with MinHash within each snapshot, which improved the average score where deduplicating all snapshots together gave only a modest gain [12]. The paper reports 15 trillion tokens from 96 snapshots [12], and the dataset card lists more than 18.5 trillion since version 1.4.0 of 11 July 2025 [13].
Model-based classifiers decide what stays in several recent corpora. For FineWeb-Edu, an instruction model of the Llama 3 family rated 460,000 pages for educational quality on a scale of 0 to 5, and a classifier trained on those ratings kept 1.3 trillion tokens [12]. DCLM applied heuristic filters and deduplication to a 240 trillion token pool, then kept the top 10% of the remaining documents by a fastText classifier's score, which produced the 3.8 trillion token DCLM-Baseline [15]. RedPajama-V2 ships raw text with quality signals for each document, so its users choose the filter [16].
Code, papers, books and other curated sources
The Pile combined 22 subsets into 825 GiB of English text [18]. By share of bytes, its components included Common Crawl web pages at 18.11%, PubMed Central papers at 14.40%, books at 12.07%, arXiv papers at 8.96% and GitHub code at 7.59%, and English Wikipedia was seen three times per pass [18]. Dolma's three trillion tokens include 2,479 billion Llama tokens of web pages, 411 billion of GitHub code, 89 billion of Reddit, 70 billion of academic papers, 6.0 billion of Project Gutenberg books and 4.3 billion of Wikipedia and Wikibooks [19].
Specialist sets fill single domains. The Stack v2 draws on the Software Heritage archive across 619 programming languages [20]. OpenWebMath, a math corpus, filtered more than 200 billion HTML files down to 6.3 million documents and 14.7 billion tokens [21]. Llama 3's final mix was roughly 50% general knowledge, 25% math and reasoning, 17% code and 8% multilingual tokens [1].
LLM training data size
A 2022 scaling study found that model size and training tokens should grow in equal proportion for compute-optimal training, and trained a 70B model on 1.4 trillion tokens [22]. Llama 3 used about 15 trillion tokens, against 1.8 trillion for Llama 2 [1]. Up to four epochs of repeated data changed loss negligibly compared with unique data, and with more repetition the value of added compute eventually decayed to zero [23].
A study revised in June 2024 put the stock of Common Crawl text at about 130 trillion tokens, as a median [24]. It projected that, if trends continued, models would be trained on datasets roughly equal in size to the stock of public human text between 2026 and 2032 [24]. Counts do not compare exactly across sources: Dolma uses the Llama tokenizer [19] and DCLM the GPT-NeoX tokenizer [15].
Post-training data: demonstrations, preferences and checkable tasks
The 2022 instruction-following study collected labeler demonstrations for fine-tuning and rankings of model outputs for reinforcement learning, and its tuned 1.3B model's outputs were preferred to those of the 175B base model [3]. Tulu 3's supervised mixture holds 939,344 samples, and its card says different licenses apply to subsets, some of them non-commercial [25]. UltraFeedback has 64,000 prompts, 256,000 responses and 380,000 pieces of feedback written by a model acting as the annotator, and its card warns that this annotator also makes mistakes [26]. Reinforcement learning with verifiable rewards gives a reward only when a completion is verified to be correct [4].
Synthetic data and its limits
Self-Instruct generated instructions, inputs and outputs from a language model, filtered out invalid or similar ones and reported a 33% absolute improvement on Super-NaturalInstructions [27]. Cosmopedia holds 25 billion tokens of synthetic textbooks, blog posts, stories and how-to articles, about 75% of them prompted from web samples [28]. Tulu 3 conditioned synthetic prompts on about 250,000 personas [4].
A 2024 Nature study found that indiscriminate use of model-generated content in training made the tails of the original distribution disappear, an effect it named model collapse [29]. Keeping a random 10% of the original data in each generation led to only minor degradation [29]. The study covers recursive training on generated data in controlled settings.
Rights, consent and licenses
An audit of more than 1,800 text datasets found license omission rates above 70% and error rates above 50% on widely used hosting sites [31]. A 2024 audit of 14,000 web domains found that restrictions added between 2023 and 2024 left about 5% or more of C4's tokens fully restricted from use [32]. The Stack v2 removed 22,066 files from 1,561 repositories after developers opted out [20].
Two corpora use only open text: the Common Pile v0.1, eight terabytes of public-domain and openly licensed text [33], and Common Corpus, about two trillion tokens that are uncopyrighted or under open licenses [34]. What a license permits for model training is the subject of what a training data license grants, and collecting training data covers source terms, opt-out signals and capture records.
What developers must disclose
The EU AI Act requires providers of general-purpose AI models to publish a summary of the content used for training, on a template from the AI Office [35]. The template of 24 July 2025 sorts sources into publicly available datasets, private datasets from third parties, data crawled and scraped from online sources, user data, synthetic data and other sources [36]. Text size is given as a range: less than 1 billion tokens, 1 billion to 10 trillion, or more than 10 trillion [36]. The obligation applies from 2 August 2025, and models placed on the market before that date have until 2 August 2027 [36].
Duplicates and benchmark contamination
One 61-word sentence appeared more than 60,000 times in C4, and models trained on deduplicated data emitted memorized text ten times less often [30]. A 2021 audit of C4 found inputs from GLUE test sets in the corpus at rates from under 2% to over 50%, depending on the dataset and input field, with both extremes in QNLI [10]. Tulu 3 removed training sets that overlapped its held-out evaluations, found by 8-gram matching [4]. Audits of agent benchmarks report contamination flaws too, as what agent benchmarks measure describes. When corpora are combined, the same document or company can arrive from several sources, and entity resolution decides which records describe the same entity.
Data that no published corpus covers raises the same questions of source type, rights and disclosure. Rillor's data services collect, license, label and clean source data into training and evaluation data for models and agents, and every record keeps its source, its rights and its history.
Read the full report
The full report adds each web corpus's filtering rules, the published mixes in tables, the post-training and synthetic sets, the EU template's source types in full and all 36 sources.