What Did Google Say About robots.txt and AI Crawlers?
That robots.txt governs access, not use. In the video, Illyes says robots.txt is defined by the internet standard RFC 9309 as “an access-preference protocol” that “only tells bots, ‘You can crawl this page,’ or, ‘You shouldn’t crawl this other page.’” He continues: “It doesn’t dictate what a system does with the data after it has already crawled it. So disallow bluntly means, ‘Do not enter the room.’” Earlier he says the rules “apply to search engine crawlers, AI training crawlers, and other well-behaved crawlers” (transcript, Search Engine Roundtable).
The video is on Google Search Central’s YouTube channel, published October 21, 2026, and its chapter list includes “Generative AI & The New Frontier” at 07:06 (Google Search Central). The quotations here come from the transcript Search Engine Roundtable published with its report; we checked the video’s title, date, speakers and chapters on Google’s own page but could not retrieve YouTube’s captions directly.
| Question | Google’s answer | Source |
|---|---|---|
| What does robots.txt control? | “which URLs the crawler can access on your site” | Introduction to robots.txt |
| Does it govern use of data already crawled? | No: it “doesn’t dictate what a system does with the data after it has already crawled it” | Video (Illyes) |
| Does it keep a page out of Google? | No; “block indexing with noindex or password-protect the page” | Introduction to robots.txt |
| How do you block one AI crawler? | Name “its specific user-agent token explicitly” | Video (Illyes) |
| Can the file force compliance? | No; it “cannot enforce crawler behavior” | Introduction to robots.txt |
| Which fields does Google read? | user-agent, allow, disallow, sitemap; others such as crawl-delay are not supported | robots.txt specification |
| Maximum file size Google reads? | 500 KiB; “Content which is after the maximum file size is ignored” | robots.txt specification |
Does Blocking an AI Crawler Remove Coverage From AI Answers?
Not by itself. A disallow rule asks a compliant crawler not to fetch pages from then on; on Illyes’s account it says nothing about content a system has already collected. Google’s documentation adds that a page disallowed in robots.txt “can still be indexed if linked to from other sites,” in which case Google may show the URL without a description (Google Search Central).
For earned coverage, the page that mentions a company is usually on the publisher’s site, so it is the publisher’s robots.txt that applies, not the company’s. See our guides on whether blocking a crawler removes you from search and noindex versus robots.txt disallow.
How Do You Name an AI Crawler in robots.txt?
By the exact user-agent token its operator publishes. Illyes says “You can’t write a fancy custom rule”: the parser “will just throw it in the trash.” He also says “user-agent tokens can only have letters and dashes,” so emoji and non-English characters cannot work in a token (transcript). Google’s specification says the user-agent field name and its value are case-insensitive and that “All non-matching text is ignored,” so googlebot/1.2 and googlebot* both match googlebot (Google specification).
Google lists its own crawlers and their tokens on a separate page, which the video’s description links to (Google common crawlers). For other companies’ AI crawlers, use the token each operator documents; our guide to blocking or allowing AI crawlers in robots.txt walks through the steps.
What Happens to Formatting Mistakes and Non-English Paths?
Lines the parser does not understand are dropped. Illyes says the parser looks for user-agent, allow, disallow and sitemap, and “If you mess up a line or try to get fancy with your spacing, the system will just silently discard the line it doesn’t understand and move on.” Comment lines starting with a hash are ignored (transcript). Google’s specification likewise says “Google ignores invalid lines in robots.txt files” (Google specification).
For paths with non-English characters or emoji, Prommawin says Google’s parser encodes them automatically, but “other or older crawlers on the web might get confused,” so “you should use the encoded version in your robots.txt rules” (transcript). The specification says Google compares rules against percent-encoded URLs and canonicalizes raw UTF-8 paths, and that if a file is not UTF-8, “Google may ignore characters that are not part of the UTF-8 range, potentially rendering robots.txt rules invalid” (Google specification).
Will Every AI Crawler Obey the File?
No. Google’s documentation says “While Googlebot and other respectable web crawlers obey the instructions in a robots.txt file, other crawlers might not” (Google Search Central). Illyes’s own wording, that crawlers check the file before “any other well-behaved crawler even looks at the content,” carries the same qualification. For checking whether a bot is who it claims to be, see how to verify a crawler is real.
What This Means for Anyone Buying a Placement
Whether an AI system can read a placement depends first on the outlet’s robots.txt, paywall and licensing choices, not on yours. Some publishers block AI crawlers and some license their content; see our check of whether news sites block AI crawlers. Nothing Google said suggests that payment changes this: a paid article is advertising, labeled as such, and no placement guarantees AI citations or permanence.
Our recommendation, not a platform rule: decide separately what you want for your own site on crawling, indexing and AI training, write those choices down, and implement each with the tool built for it, rather than expecting one robots.txt line to cover all three.
Methodology and Limitations
Four limits apply. First, the quotations from the video come from the transcript Search Engine Roundtable published; we confirmed the video’s existence, date, speakers and chapters on Google Search Central’s YouTube page but did not obtain Google’s captions. Second, we read Google’s robots.txt introduction and specification, not each AI company’s crawler documentation. Third, we did not test any crawler’s behavior. Fourth, the guide has no chart, because the only published numeric limit we confirmed is the single 500 KiB size cap.
Conclusion
Robots.txt asks; it does not enforce, and by Google’s account it does not reach back into data already collected. Name the bots you mean to block, use noindex when you need a page out of search, and remember that coverage on a publisher’s site is governed by the publisher.
Frequently Asked Questions
Does robots.txt stop AI companies from training on my content?
Not content they have already crawled. Google’s Gary Illyes said in an October 21, 2026 video that robots.txt only tells bots whether they may crawl, and does not dictate what a system does with data it already has.
How do I block one specific AI crawler in robots.txt?
Name its user-agent token explicitly in a user-agent line, as the crawler’s operator documents it. Google says invented or custom rules are discarded by the parser.
Does robots.txt keep a page out of Google?
No. Google’s documentation says robots.txt is not a mechanism for keeping a page out of Google and recommends noindex or password protection instead.
Can a page blocked in robots.txt still appear in search?
Yes. Google says a disallowed page can still be indexed if other sites link to it, usually shown without a description.
Can I use emoji or non-English characters in robots.txt?
Not in user-agent tokens, which Google says can contain only letters and dashes. For URL paths, Google recommends using the percent-encoded form.
How big can a robots.txt file be for Google?
Google enforces a 500 KiB limit, and content after that size is ignored, according to its robots.txt specification.
Sources and Methodology
This Explainer combines a Google Search Central video, the transcript published with Search Engine Roundtable’s report of it, and Google’s own robots.txt documentation, all read on October 22, 2026. Practical advice is our recommendation, not a platform rule. We did not test any crawler or review AI companies’ crawler pages.
- official Google Search Central (YouTube), What can you actually put in a robots.txt?, October 21, 2026. Title, date, speakers, description and chapter list. Read October 22, 2026.
- trade press Search Engine Roundtable, Google On robots.txt Rules For AI Crawlers, Emojis, International URLs & More, October 21, 2026. Full transcript of the video, used for quotations. Read October 22, 2026.
- platform documentation Google Search Central, Introduction to robots.txt. Purpose, noindex, enforcement and indexing limits. Read October 22, 2026.
- platform documentation Google, How Google interprets the robots.txt specification. Supported fields, invalid lines, encoding, size limit, case sensitivity. Read October 22, 2026.