What Mueller said
The episode, “Do sitemaps still matter?”, published on 1 October with a transcript, is a conversation between Martin Splitt and John Mueller of Google's Search Relations team. Most of it is about Google. The part that matters for AI is a short passage about private sitemaps. A site can give its sitemap an unusual name, leave it out of robots.txt and submit it to Google directly, Mueller said. The cost is that nothing else can find it: Bing would need its own submission, and AI training crawlers cannot be told about it at all, because “they usually don't have a console or any setup where you can submit a sitemap file”.
His advice for sites that want their content in AI systems was to use the standard file name, sitemap.xml, or to rely on RSS feeds, which are usually linked from the head of a web page and so easy for any crawler to notice. “I've seen that happen in my server logs where some AI crawler accesses my sitemap file and they read it, they do something with it,” he said, adding that he had seen the same with his RSS files. He did not say which crawler, and said: “I don't know if they document this anywhere.”
He was blunter about llms.txt, the proposed Markdown file for AI systems. It resembles an old-style HTML sitemap, he said; Google's systems cannot use it as a sitemap because it lacks the strict format; and “the hope is bigger than the reality”. Our count of who publishes one is in does anyone actually publish llms.txt.
What the AI companies document
We took up Mueller's question and read the public crawler pages of eight operators whose bots fetch the web for AI training, AI search or user requests. The test was simple: does the page say the crawler reads sitemaps, and does it offer a way to submit one?
Only one does. Common Crawl, whose archive is a common source of training data and which we covered in what Common Crawl is, says in its FAQ that its crawler “supports the Sitemap Protocol and utilizes any Sitemap announced in the robots.txt file”. OpenAI's crawler overview, Anthropic's help article on its three bots, Perplexity's crawler page, Apple's Applebot page, Meta's web crawler page, Amazon's Amazonbot page and Mistral's crawler page all explain their user agents and how to block them with robots.txt. None of them mentions sitemaps or feeds, and none offers a submission route.
The search engines are the contrast. Google and Bing both accept sitemap submissions through their webmaster consoles, and Bing's help page says its list of known sitemaps includes ones discovered on its own. Silence in documentation does not mean a crawler ignores sitemaps; Mueller's logs suggest at least one does not. It means a site cannot know which do, which is why the location the protocol itself defines matters.
That location is robots.txt. The sitemaps protocol lets a site declare its sitemap with a Sitemap: line and says the directive “is independent of the user-agent line”. In other words, a sitemap declared once is visible to every crawler that reads robots.txt, whatever rules apply to it, which is the closest thing a site has to submitting a sitemap to a crawler that offers no console.
What 30 large companies publish
To see how established companies handle this, we fetched robots.txt from the main website of each of the 30 companies in the Dow Jones Industrial Average, as listed on Wikipedia on 5 October 2026, and checked for sitemap declarations, whether the first declared sitemap worked, whether a sitemap existed at the default /sitemap.xml path, and whether the home page advertised an RSS or Atom feed.
Number of companies, of 30, robots.txt and home pages checked 5 October 2026
Source: Media Morpheus Research, requests to each company's main website on the evening of 5 October 2026, US Eastern time; five sites read through a text-rendering proxy.
The three without a declaration were abc.xyz (Alphabet), amazon.com and www.visa.com. The one declared sitemap that did not work was Procter & Gamble's patents sitemap.
27 of 30 declared at least one sitemap. The three that did not were Alphabet, whose corporate site abc.xyz has a robots.txt with crawl delays but no sitemap line; Amazon, whose robots.txt on amazon.com has none and which, separately, blocks GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and other AI crawlers by name; and Visa, whose www.visa.com returned an empty robots.txt. Visa's US site, usa.visa.com, does declare two sitemaps, but a crawler starting at www.visa.com would not learn that from robots.txt.
Of the 27, 26 pointed to a working sitemap or sitemap index. Procter & Gamble's robots.txt declares a single sitemap for its patents pages, at an address with a doubled slash, and that address returned an ordinary web page rather than XML when we fetched it; its default /sitemap.xml did the same. Large companies declared many sitemaps: Salesforce listed 52, Walmart 34, Honeywell 30 and Microsoft 22.
Mueller's two fallbacks were less common. Of the 25 sites that answered our requests directly, 12 served a sitemap at the default /sitemap.xml path; most of the rest kept theirs elsewhere and declared it in robots.txt, which serves the same purpose. Of the 22 home pages that loaded normally, one, The Walt Disney Company's, advertised an RSS feed in its page head.
What a company can check
Declare the sitemap in robots.txt. It is the only discovery route that every crawler can see without a console, and the protocol makes it apply regardless of user agent. Check that the address in the file actually returns XML.
Check every host. Visa's case shows how a company can have working sitemaps on one host and nothing on the host a crawler is likely to try first.
Keep a feed for news. If a company publishes press releases or articles, a feed linked from the page head is the other route Mueller named, and the one most company home pages in our sample did not offer.
Do not expect a sitemap to get content into AI answers. A sitemap helps a crawler find pages; it does not decide what a model trains on or cites. Whether a crawler is allowed in at all is set by robots.txt rules per user agent, discussed in does blocking a crawler remove you from search, and Google's own crawl of a sitemap depends on how much it wants from the site in the first place.
Methodology and limitations
The Search Off the Record quotations are from Google's published transcript of episode 114, “Do sitemaps still matter?”, released 1 October 2026 and read in full on 5 October 2026; Search Engine Journal's report of 5 October was used only to locate the episode. Crawler documentation was read on 5 October 2026 through a text-rendering proxy, and each page was searched for “sitemap”, “RSS” and “feed”: OpenAI's Overview of OpenAI Crawlers, Anthropic's help article on its crawlers, Perplexity's Perplexity Crawlers, Apple's About Applebot, Meta's Meta Web Crawlers, Amazon's About AmazonBot, Mistral's Mistral crawlers, and Common Crawl's CCBot page and FAQ. The Dow 30 list was taken from Wikipedia's List of Dow Jones Industrial Average companies on 5 October 2026. On the evening of 5 October 2026, US Eastern time, we requested robots.txt, /sitemap.xml and the home page from each company's main domain with a standard desktop browser user agent, followed redirects, and fetched the first declared sitemap. Five sites refused, timed out or served a maintenance page to our direct requests (3M, Caterpillar, Cisco, McDonald's and Johnson & Johnson); their robots.txt files and first declared sitemaps were read through a text-rendering proxy instead, and all five declared at least one working sitemap. Results reflect one moment from one location; bot-protection systems may serve different responses to different visitors. A sitemap declaration shows what a site offers, not what any crawler does with it. We have no server logs of our own showing AI crawler behaviour.
Conclusion
AI crawlers cannot be sent a sitemap the way Google and Bing can, and almost none of the companies running them say whether they read one. Google's John Mueller says he has watched some of them do it, and Common Crawl is the one operator that commits to it in writing, through the robots.txt Sitemap line. Most large companies already make that declaration, but our check of the Dow 30 found the gaps a crawler would trip on: a sitemap pointing at an ordinary web page, an empty robots.txt on the main host, and two of the world's largest companies declaring none at all. The fix costs one line and a working file. What any AI system then does with the pages it finds is a separate question that no sitemap answers.
Frequently Asked Questions
Do AI crawlers read sitemaps?
Some appear to. Google's John Mueller said he has seen an AI crawler read his sitemap and RSS files in his server logs. Of eight AI crawler operators whose documentation we read, only Common Crawl states that its crawler uses sitemaps declared in robots.txt.
Can I submit a sitemap to ChatGPT, Claude or Perplexity?
No. None of the crawler documentation from OpenAI, Anthropic, Perplexity, Apple, Meta, Amazon or Mistral offers a submission route. Google and Bing accept sitemaps through their consoles; AI crawlers do not.
How do I tell AI crawlers where my sitemap is?
Declare it with a Sitemap line in robots.txt. The sitemaps protocol says the directive is independent of the user-agent line, so every crawler that reads robots.txt can see it. Mueller also suggested keeping the standard name, sitemap.xml, or relying on RSS feeds.
Do large companies declare their sitemaps?
Most do. 27 of the 30 Dow Jones companies declared at least one sitemap in robots.txt on 5 October 2026. Alphabet's abc.xyz, Amazon and www.visa.com did not, and one of the 27 pointed to an address that returned an ordinary web page.
Can llms.txt replace a sitemap?
Not in Google's systems. Mueller said llms.txt is closer to an HTML sitemap, lacks the strict format Google needs, and that the hope for it is bigger than the reality.
Why does Search Console say my sitemap couldn't be fetched?
Mueller gave two reasons beyond technical errors: Google may be too busy to fetch it, or crawl demand for the site may be low, which he said is very often based on the perceived quality of the website.
Sources
- official Google Search Central, Search Off the Record, Do sitemaps still matter? – transcript, episode 114, 1 October 2026. Read in full 5 October 2026.
- trade press Matt G. Southern, Search Engine Journal, Google's Mueller Says AI Crawlers Access Sitemaps & RSS In His Logs, 5 October 2026. Used to locate the episode. Read 5 October 2026.
- standards body sitemaps.org, Sitemaps XML format. Sitemap directive in robots.txt is independent of the user-agent line. Read 5 October 2026.
- documentation Common Crawl, FAQ, and CCBot. Read 5 October 2026 through a text-rendering proxy.
- documentation OpenAI, Overview of OpenAI Crawlers; Anthropic, Does Anthropic crawl data from the web?; Perplexity, Perplexity Crawlers; Apple, About Applebot; Meta, Meta Web Crawlers; Amazon, About AmazonBot; Mistral, Mistral crawlers. Each searched for sitemap, RSS and feed. Read 5 October 2026 through a text-rendering proxy.
- documentation Microsoft Bing, Sitemaps, Bing Webmaster Tools help. Read 5 October 2026 through a text-rendering proxy.
- live read Media Morpheus Research, requests to robots.txt, /sitemap.xml, the first declared sitemap and the home page of the 30 Dow Jones Industrial Average companies' main websites, evening of 5 October 2026, US Eastern time; five sites read through a text-rendering proxy. Company list from Wikipedia, List of Dow Jones Industrial Average companies, read 5 October 2026.