Data collection for model training covers both the data a model learns from and the data it is tested on. Every item needs three facts kept beside it from the first capture: where it came from, the terms it came under and when it was captured. Those facts age quickly. A quarter of the webpages that existed at some point between 2013 and 2023 were no longer accessible by October 2023 [25], and robots.txt restrictions on three public web corpora rose from about 1% of tokens in mid 2023 to 5 to 7% in April 2024 [2]. Disclosure rules in California and the European Union ask developers to describe their training data by source and collection period [15, 16].
This note sets out the method step by step, from the regulators' guidance and the standards that define each part. It is research, never legal advice.
Data collection for model training starts with a source list
The OECD sorts collection mechanisms into two groups: data collected directly from individuals and organizations, and data from third-party providers, which covers commercial licensing, open data and "data scraping" [30]. The U.S. Copyright Office lists routes from "publicly available" data downloaded from the internet and text gathered by web crawlers to licensed or non-public sources [23]. It adds that "'Publicly available' is not synonymous with 'authorized.'" [23]
Name each source before collection starts, with the mechanism used to reach it. Croissant's RAI vocabulary gives recommended values for that mechanism, such as Web API and Document analysis, and a field for the collection timeframe [31]. Datasheets for Datasets asks creators to read its collection questions "prior to any data collection", including one that separates the time of collection from the time the data was created [4]. In a sample of 1,000,000 URLs from the C4 corpus, 92% of pages were estimated to have been written between 2011 and 2019 [27].
Check the terms before the first capture
Terms attach to items as well as to sources. Wikimedia licenses its dump text under the GFDL and CC BY-SA 4.0, but "many images are NOT released under this license" [35]. Common Crawl's terms say crawled content "may be subject to separate terms of use or terms of service from the owners" [10]. Common Crawl may amend its own terms at any time, and they carry a last-updated date, 7 March 2024 on the version viewed [10], so the version in force at capture belongs in the record. Works of the U.S. Government carry no copyright under 17 U.S.C. 105, yet the government may hold copyrights transferred to it [24], so agency sources need an item-level check too.
Published license labels are often wrong. An audit of more than 1,800 text datasets found license omission of 70% or more and error rates of 50% or more on widely used dataset hosting sites [3]. For The Stack, a license detector did not detect a license for more than 80% of the repositories it was run on [28].
In the Copyright Office's view, the knowing use of a dataset of pirated or illegally accessed works "should weigh against fair use without being determinative" [23]. First-party data carries the commitments it was gathered under. FTC staff wrote in February 2024 that a business "cannot then unilaterally renege on those commitments after collecting users' data" [37].
Read opt-out signals as dated facts
RFC 9309 describes robots.txt as rules "that crawlers are requested to honor when accessing URIs", and states that "These rules are not a form of access authorization" [1]. A crawler's next step depends on the status code of the robots.txt fetch: after a 5xx answer it "MUST assume complete disallow" [1].
Article 53 of the EU AI Act requires providers of "general-purpose AI models" to put in place a copyright policy that identifies and complies with rights reservations made under Article 4(3) of Directive (EU) 2019/790 [11]. Under the voluntary General-Purpose AI Code of Practice, published on 10 July 2025, signatories commit to crawlers that read and follow robots.txt as RFC 9309 specifies [12, 13].
The signals move. A 2024 audit of 14,000 web domains found robots.txt restrictions on about 1% of tokens in the C4, RefinedWeb and Dolma corpora in mid 2023 and 5 to 7% in April 2024 [2]. An IETF working group is defining a vocabulary for usage preferences with a training category, and on 7 November 2026 its datatracker lists version 08 as an Internet-Draft [36].
Personal data in collected sources
In a 2023 joint statement, data protection authorities wrote that publicly accessible personal information "is still subject to data protection and privacy laws in most jurisdictions" [22]. The European Data Protection Board (EDPB), in its Opinion 28/2024, lists measures that may include excluding certain sources from collection and excluding websites that clearly object to web scraping, for example by respecting robots.txt files [17].
The EDPB's Guidelines 03/2026, version 1.0, adopted for public consultation on 7 July 2026, recommend recording when data was collected, and state that the absence of a robots.txt file "does not amount to consent" under the GDPR [18]. The CNIL, France's data protection authority, adds that keeping the source URL of each item helps a person or a specific piece of data be found in the database later [21].
What a capture record holds
The web archive format WARC already defines several fields of a capture record. WARC-Date is "the instant that data capture for record creation began", in UTC [5]. WARC-Target-URI is the address the crawler requested, and WARC-Payload-Digest holds a digest of the content with its algorithm name [5].
| Field | What it records | Basis |
|---|---|---|
| Source URL | The address requested | [5, 18, 21] |
| Capture time | The instant capture began, in UTC | [5, 18] |
| Content digest | A digest of the content and its algorithm | [5] |
| Fetch status | The HTTP status of the page and of robots.txt | [1, 5] |
| Opt-out signals | The robots.txt rules and other signals read at capture | [1, 13, 17] |
| Terms | The source's and the item's terms, with their dates | [10, 35] |
| Personal data | Whether the item holds personal information | [16, 17] |
| Collection period | The start and end of the collection | [15, 16, 31] |
| Version | The earlier capture this one revises | [32, 33] |
The W3C PROV-O model states the same links as provenance, with properties such as prov:generatedAtTime and prov:wasRevisionOf [32].
Sources change after capture
Pew Research Center found that 38% of webpages that existed in 2013 were no longer accessible in October 2023, and about one in five pages from the 2021 snapshot had gone two years later [25].
Capture is therefore a series of versions. A re-capture that finds the same content can be logged as a WARC revisit record [5], and a changed page becomes a new version linked to the last one [32]. Prices follow the same rule: reading GPU prices from cloud pricing APIs stores the time of each call in UTC beside the unchanged response.
Keep evaluation data apart
Evaluation data collected from the same sources as training data can overlap with it. One study found train-test overlap in over 4% of the validation set of standard datasets [26]. The C4 documentation study found benchmark inputs inside the corpus at rates from under 2% to over 50% [27]. LiveBench draws its questions from recent information sources and adds and updates them monthly [29]. Dated capture records allow a similar split, with evaluation items limited to those that appeared after the training data was collected. California's AB 2013 defines training to include "testing, validating, or fine tuning" [16], so evaluation data belongs in the same records.
Records the disclosure rules ask for
AB 2013 requires developers to post documentation that includes a high-level summary of their training datasets: their sources or owners, whether they were purchased or licensed, whether they include personal information, and "the time period during which the data in the datasets were collected" [16]. The European Commission's template for the public summary of training content, published on 24 July 2025, asks for crawler names, the period of data collection and the top 10% of domain names by size of content [14, 15]. Providers of models placed on the market before 2 August 2025 should publish their summary no later than 2 August 2027 [15]. A capture log that holds source, terms and dates answers these items directly.
Records gathered from several sources often describe the same company or person, and entity resolution decides which of them match. Datasets that state their limits sets out seven provenance facts for every record, from source and capture time to rights and refresh. The scope comes before any of this, as in how to specify a training dataset before anyone builds it.
Rillor's data collection for model training names the sources before collection starts, checks each source's terms before anything is captured, and delivers the dataset with its capture log, its source list with terms and its version history.
Read the full report
The full report sets out each step from its sources, the capture record field by field with the standard behind it, the figures on license errors, opt-out signals and link rot, a checklist for teams starting a collection, and all 39 sources.