A robots.txt file asks crawlers to stay out of some paths of one host, and model training crawlers are among the crawlers it addresses. RFC 9309, the IETF standard published in September 2022, describes rules "that crawlers are requested to honor when accessing URIs" and states that "These rules are not a form of access authorization" [1]. A disallow line aimed at model training reaches the crawlers it names, from the time they next read the file, on the host that serves it [1, 6, 21]. The standard has no field for what fetched content may be used for [1, 6].
This explainer reads the protocol, the crawler documentation of eight operators as it stood on 8 November 2026, five measurement studies and the laws that refer to machine-readable opt-outs. It is research, never legal advice.
What robots.txt does for model training crawlers
Under RFC 9309, "Crawlers set their own name, which is called a product token, to find relevant groups" [1]. A crawler obeys the group that matches its token, falls back to the * group if the file has one, and follows no rules if neither exists [1]. An opt-out from model training is therefore a list of names kept by the site. The European Union Intellectual Property Office (EUIPO) wrote in May 2025: "To block multiple crawlers, each must be listed with its exact name" [16].
Three of the eight operators publish one name for each purpose [20, 21, 29]. The crawler documentation of one model developer says "Each setting is independent of the others" and that "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models" [20]. An answer engine says its search crawler "is not used to crawl content for AI foundation models" [28].
Tokens that crawl nothing
A search engine operator's documentation says "Google-Extended doesn't have a separate HTTP request user agent string" and that "the robots.txt user-agent token is used in a control capacity" [22]. It governs training of future model generations and grounding, and it "does not impact a site's inclusion in Google Search" [22]. A device maker's page says "Applebot-Extended does not crawl webpages" and "is only used to determine how to use the data crawled by the Applebot user agent" [24].
These tokens carry a use preference inside a crawl rule. Page-level tags carry others. To limit what its search features show, the search operator points site owners to "nosnippet, data-nosnippet, max-snippet, or noindex controls" [23], and the device maker points to the nosnippet meta tag for its "broad world knowledge answers" [24]. An online retailer reads the noarchive tag as "do not use the page for model training" [29].
Fetches a user asks for
Four operators say robots.txt may not govern fetchers that act on a user's request. One model developer writes: "Because these actions are initiated by a user, robots.txt rules may not apply" [20]. The answer engine writes: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules" [28]. A social media company says its user-request fetcher "may bypass robots.txt rules" [27], and the retailer says its own "may not follow all robots.txt directives" [29].
Over 40 days on 36 sites at one institution, a 2025 study found that assistant and model-search bots "have the lowest re-check rates with less than 40% of bots checking robots.txt within a 168 hour window" [33]. When an answer is built from fetched pages, checking LLM citations against their sources sets out how to test what it cites.
An opt-out looks forward
A disallow applies to crawls after it is read. One model developer's page says a block "signals that the site's future materials should be excluded from our AI model training datasets" [21]. The IETF draft for attaching usage preferences says updates to robots.txt "do not retroactively apply to resources" [6].
The delay before a change is seen varies. RFC 9309 says crawlers "SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable" [1]. The retailer's crawlers "fetch host-level robots.txt files or use a cached copy from the last 30 days" [29].
Data already gathered sits outside the file's reach. A non-profit crawl archive publishes a ledger of legal opt-out requests "to alert our users about material that content owners have specifically requested to be excluded from our crawls" [26]. Copyright owners told the U.S. Copyright Office that an opt-out would be "impossible to utilize after training occurs" [17]. Datasets that ship lists of URLs add a gap of their own: the EUIPO notes that a crawler downloading from such a list "may skip the 'discovery phase' where robots.txt is typically checked" [16].
One file, one host
RFC 9309 places the file "in the top-level path of the service" [1], so each host carries its own. One model developer asks site owners to repeat the rules: "Please do this for every subdomain that you wish to opt out from" [21].
Copies elsewhere are outside it. The EUIPO wrote that a reservation through robots.txt on one website "does not guarantee a valid reservation of rights for the licensed copies of the same content hosted on other websites" [16]. Many owners cannot edit the file at all. In a 2025 study of 1,182 artist sites, over 78% sat on one of eight hosting providers, most of which "do not allow for modification of robots.txt" [32].
Third-party datasets sit outside it too. The EU's General-Purpose AI Code of Practice says its crawler commitment "does not affect the application of Union law on copyright and related rights to protected content scraped or crawled from the internet by third parties" [13]. Tying each copy of a work on another host back to one item is a record-matching task of the kind described in how entity resolution matches records across sources.
Signals that state a use
Several vocabularies separate collection from use. The IETF's AI Preferences (aipref) working group has two Internet-Drafts. Its attachment draft adds a Content-Usage line to robots.txt and an HTTP header of the same name, with the example Content-Usage: train-ai=n, and describes "a two-stage arrangement that distinguishes acquisition and usage" [6]. Its vocabulary draft of 14 September 2026 says its contents "DO NOT REFLECT CONSENSUS of the Working Group either in whole or part" [5]. Both showed the datatracker state "I-D Exists" on 8 November 2026, after a charter milestone of August 2026 for sending them to the Internet Engineering Steering Group (IESG) [4, 5, 6].
Other signals are industry specifications. Cloudflare's Content Signals Policy of September 2025 writes Content-Signal: search=yes, ai-train=no. Cloudflare reported that customers had turned on its managed robots.txt for over 3.8 million domains, and said it would add that line to the file it serves for them [30]. The W3C community group report on TDMRep "is not a W3C Standard nor is it on the W3C Standards Track" [7]. RSL 1.0 adds a License: line that points to a license document [9].
One model developer's page says that "As of October 30, 2026" it "has begun recognizing a machine-readable signal" on training that "follows a draft based on work done by the IETF" [21]. The search operator's page of 30 October 2026 says it "intends to support the AIPREF shared machine-readable training preference signal" in early 2027 [22].
The vocabulary draft also says legal agreements that explicitly permit a use "likely apply despite the existence of machine-readable statements of preference" [5]. What a training data license grants covers those written terms.
What the law asks of a reservation
Article 4(3) of Directive (EU) 2019/790 lets rightholders reserve text and data mining of their works "in an appropriate manner, such as machine-readable means in the case of content made publicly available online" [11]. Article 53(1)(c) of the EU AI Act asks providers of "general-purpose AI models" to identify and comply with such reservations, "including through state-of-the-art technologies", from 2 August 2025 [12]. The Code, "a voluntary tool", commits signatories to crawlers that follow robots.txt as RFC 9309 specifies and to other appropriate machine-readable protocols [13].
In Germany, the Higher Regional Court of Hamburg held in December 2025 that a photographer had not shown that a reservation written in natural language was machine-readable in the second half of 2021, and the Federal Court of Justice has set 17 December 2026 for its judgment [19]. The UK government's report of 18 March 2026 says "a broad copyright exception with opt-out is no longer the government's preferred way forward" [18]. In its pre-publication report of May 2025, the U.S. Copyright Office wrote that, for any opt-out Congress might consider, it should not be "sufficient for AI companies to merely honor platform-level flags, like robots.txt" [17].
What a capture record keeps
The European Commission's template for the public summary of training content asks for crawler names, their purposes, a description of crawler behaviour such as "respect of robot.txt and other protocols", and the period of collection [14]. A capture record can keep each signal with its fetch time: the robots.txt file as fetched, the group that matched, any usage line, the page-level tags and the host [1, 6, 23, 29]. Collecting training data sets out the full capture record and the steps before it.
Rillor's data collection for model training checks each source's terms before anything is captured and records where each record came from and when.
Read the full report
The full report adds each operator's crawlers by purpose, dated compliance and adoption figures, the status of each usage vocabulary, the legal texts, the fields a capture record can keep, and all 35 sources.