llms.txt vs Crawlable Pages at a Glance
Each claim below is one we see in pitches for llms.txt work. The middle column quotes or summarises the primary source that addresses it, as read on October 9, 2026.
| Claim people make | What the source says | Source |
|---|---|---|
| “Google's AI Overviews need an llms.txt file.” | “You don't need to create new machine readable files, AI text files, or markup to appear in these features.” | Google, AI features and your website |
| “ChatGPT search reads llms.txt to decide what to cite.” | OAI-SearchBot “is used to surface websites in search results in ChatGPT’s search features”; OpenAI recommends allowing it in robots.txt. The page does not mention llms.txt as a crawler input. | OpenAI, Overview of OpenAI Crawlers |
| “Claude needs llms.txt to find your content.” | Anthropic lists three bots and says they “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt.” No mention of llms.txt. | Anthropic Help Center |
| “Perplexity uses llms.txt to choose sources.” | PerplexityBot “is designed to surface and link websites in search results on Perplexity”; site owners manage it with robots.txt tags. | Perplexity Crawlers |
| “llms.txt is an official web standard.” | It describes itself as “A proposal to standardise on using an /llms.txt file”, written by Jeremy Howard in September 2024. | llmstxt.org |
| “llms.txt lets you block AI training.” | “robots.txt and llms.txt have different purposes.” llms.txt contains no access rules. | llmstxt.org |
| “A missing llms.txt fails a site audit.” | Lighthouse marks a 404 as “Not Applicable (N/A), as providing the file is optional at the moment.” | Chrome Lighthouse |
Why People Believe AI Search Needs llms.txt
The belief comes from a real problem and some visible adopters. Jeremy Howard's llms.txt proposal starts from a fair diagnosis: “web pages are built for people. An HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise.” A short Markdown index for machines sounds like an obvious fix.
Four other things make the file look more established than it is.
- Visible adopters. In our September 2026 check of 41 domains, 10 of 12 developer and SaaS companies served an llms.txt file, including Stripe, Cloudflare and Shopify. The proposal also says the AI labs “publish llms.txt files for their own developer docs”, naming OpenAI, Anthropic and Gemini.
- Tooling. llmstxt.org lists Yoast SEO, AIOSEO and Wix as generating the file, and says Wix “Generates an llms.txt file for every Wix site”. A file a CMS creates by default looks like a requirement.
- An audit in Chrome. Chrome's Lighthouse llms.txt audit, last updated May 5, 2026, calls the file “an emerging convention”. An audit from Google's browser team is easy to read as a Google Search signal, which it is not.
- The name. llms.txt sounds like robots.txt and sitemap.xml, two files that search engines do read. The proposal itself draws that parallel, then says the files “have different purposes.”
None of that is evidence that an AI search product fetches another site's llms.txt and uses it to decide what to show or cite. That is the claim that matters when someone sells the file as visibility work.
What the Evidence Shows: Google, OpenAI, Anthropic and Perplexity
The four companies behind the largest AI search products document crawlers that fetch ordinary web pages and obey robots.txt. Google's guidance goes further and says no new AI text file is needed. We searched each page for “llms” on October 9, 2026; none describes reading llms.txt from other sites.
Google: no AI text files needed
Google's AI features and your website (last updated December 10, 2025) is the most direct statement any vendor has made. It says “There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary”, and then:
“You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add.”
The page does not use the word llms.txt. It does not need to: an llms.txt file is a new machine-readable AI text file. The same page sets out what does matter. To be a supporting link, “a page must be indexed and eligible to be shown in Google Search with a snippet”, with “no additional technical requirements.” Its list of SEO basics includes “Ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure”, “Making your content easily findable through internal links” and “Making sure that important content is available in textual form”.
Google staff have said the same thing less formally. In April 2025, Google's John Mueller replied in a Reddit r/TechSEO thread (comment by u/johnmu, April 16, 2025; we read it in an Internet Archive copy): “AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it). To me, it’s comparable to the keywords meta tag” (also reported by Search Engine Journal, April 17, 2025). That is a comment, not documentation, and it is 18 months old. Google's documentation is the stronger evidence.
OpenAI: OAI-SearchBot and robots.txt
OpenAI's Overview of OpenAI Crawlers says it uses “OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.” OAI-SearchBot “is used to surface websites in search results in ChatGPT’s search features”, and “To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site’s robots.txt file”. ChatGPT-User visits pages when users ask questions and “is not used to determine whether content may appear in Search.” The only llms.txt on the page is OpenAI's own documentation index, offered to readers of its docs.
Anthropic: three bots, all governed by robots.txt
Anthropic's crawler help article (dated April 7, 2026) lists ClaudeBot for training, Claude-User for fetches “When individuals ask questions to Claude”, and Claude-SearchBot, which “navigates the web to improve search result quality for users.” It says disabling Claude-SearchBot “may reduce your site's visibility and accuracy in user search results”, and that its bots honour robots.txt. llms.txt is not mentioned.
Perplexity: PerplexityBot and Perplexity-User
Perplexity's crawler documentation says PerplexityBot “is designed to surface and link websites in search results on Perplexity” and recommends allowing it in robots.txt and permitting its published IP ranges. Perplexity-User fetches a page “When users ask Perplexity a question” and “generally ignores robots.txt rules” because a user requested it. As with OpenAI, the page links Perplexity's own llms.txt index for its docs; it says nothing about reading yours.
What this adds up to
| Company | Search or answer crawler | Documented control | Says it reads other sites' llms.txt? |
|---|---|---|---|
| Googlebot (AI Overviews and AI Mode use the Search index) | robots.txt for Googlebot; nosnippet, max-snippet, noindex | No; says no AI text files are needed | |
| OpenAI | OAI-SearchBot; ChatGPT-User for user requests | robots.txt tags for OAI-SearchBot; robots.txt rules may not apply to user-initiated ChatGPT-User visits | No |
| Anthropic | Claude-SearchBot; Claude-User for user requests | robots.txt (all three bots) | No |
| Perplexity | PerplexityBot; Perplexity-User for user requests | robots.txt tags; user fetches generally ignore robots.txt | No |
Absence from documentation is not proof that no system ever fetches the file. Server logs may show requests for /llms.txt from tools and agents. It does mean nobody selling the file can point to a vendor document that says it affects AI search results.
What AI Crawlers Actually Read: HTML Pages, Often Without JavaScript
AI search crawlers read the same public HTML pages search engines do, so the page itself is the thing to fix. A December 2024 study by Vercel and MERJ found that none of the major AI crawlers it observed rendered JavaScript, apart from Google's Gemini through Googlebot and Applebot. Content that only appears after scripts run may be missing from their copy.
Fetches in the month before December 17, 2024, as reported by Vercel
Source: Vercel, The rise of the AI crawler, December 17, 2024.
One hosting network in late 2024; volumes vary by site and have changed since. Volume shows how much each crawler fetched, not what it could render or cite.
The Vercel study monitored nextjs.org, the Vercel network and two job-board sites. It reported that OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot, Meta-ExternalAgent, ByteDance's Bytespider and PerplexityBot did not render JavaScript, while “Google's Gemini leverages Googlebot's infrastructure, enabling full JavaScript rendering.” This is a vendor study from 2024 on one network, and crawlers change. It is still the best public measurement we found, and it points the same way as Google's advice to make important content “available in textual form”.
Percentage of fetches to nextjs.org returning 404, monitoring period ending December 2024
Source: Vercel, The rise of the AI crawler, December 17, 2024.
A single site measured by its host. Vercel says ChatGPT and Claude often requested outdated assets from a /static/ folder, and that the exact cause of the high 404 rates remains unclear.
Links matter for discovery as well. Google's link best practices say “Generally, Google can only crawl your link if it's an <a> HTML element (also known as anchor element) with an href attribute.” Google's technical requirements for indexing are three: “Googlebot isn't blocked”, “The page works, meaning that Google receives an HTTP 200 (success) status code”, and “The page has indexable content.” A page that meets those, in server-rendered HTML, is readable by every crawler in the table above. An llms.txt file cannot rescue a page that fails them.
Where llms.txt Still Matters
llms.txt is harmless and has a plausible job outside search: helping coding agents and AI tools navigate documentation. The proposal's August 2026 v2 changes page says “coding agents use them reliably”, and Chrome's Lighthouse page says that without the file “agents may spend more time crawling the site to understand its high-level structure”.
- Documentation sites. The proposal says llms.txt files “are used most heavily for software documentation, where coding agents follow them to find API references and tutorials.” That matches our adoption data: developer companies publish them; news publishers in our sample do not.
- Agents a user points at your site. When a person gives an assistant your URL and asks it to work something out, a clean index can save it from guessing. That is a user-directed fetch, not search ranking.
- Low cost. A hand-written file for a small site takes an afternoon. Lighthouse does not penalise its absence, and its presence does not conflict with anything Google documents.
- A useful editing exercise. Writing a one-paragraph summary and a short list of your most important pages forces the same decisions a good About page or newsroom needs.
The fair position is that publishing one is optional housekeeping. If you want one, our guide on how to create an llms.txt file follows the v2 format. The risk is not the file. It is paying for it as AI visibility while the pages it points to stay blocked, script-dependent or unindexed.
What to Do Instead: Crawlable Pages in Practice
Making your important pages crawlable, indexable and readable as plain HTML is what Google, OpenAI, Anthropic and Perplexity document. Google says a supporting link in AI Overviews must be “indexed and eligible to be shown in Google Search with a snippet”. These six checks cover that ground, in order of impact.
- Allow the search crawlers you want in robots.txt. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot each have their own token, separate from training crawlers such as GPTBot and ClaudeBot. Our guide to blocking or allowing AI crawlers in robots.txt lists them. Check your CDN or firewall too: Google names “any CDN or hosting infrastructure”, and Perplexity documents firewall rules for its bots.
- Confirm the pages are indexed. Use Search Console's URL Inspection tool, as in how to check if a page is indexed by Google. A page that is not in Google's index cannot be a supporting link in AI Overviews.
- Put key facts in server-rendered HTML. View the page source (not the inspector) and confirm your company description, prices, product names and contact details are in the HTML as text, not injected by scripts or shown only in images.
- Link to important pages with ordinary <a href> links. Internal links from your home page and navigation are how crawlers find pages. A sitemap helps search engines discover them; see how to submit a sitemap to Google and Bing.
- Use snippet controls deliberately. nosnippet and max-snippet limit what Google can show, including in AI features. Make sure they are not set on pages you want quoted; our guide to controlling snippets and AI Overviews explains each control.
- Check what the assistants actually say. Ask them about your company and see which pages they cite, following how to check what AI says about your company. That tells you more than any file audit.
None of these steps guarantees a citation. Google states that meeting its requirements “doesn't mean that Google will crawl, index, or serve its content.” They remove the documented reasons a page cannot be used.
Common Objections
The strongest arguments for llms.txt are about agents and future adoption rather than current search. Each objection below is answered from the primary sources already cited.
| Objection | Answer | Source |
|---|---|---|
| “The AI labs publish llms.txt, so they must read it.” | OpenAI and Perplexity link their own llms.txt from their crawler pages, and the llms.txt proposal says OpenAI, Anthropic and Gemini publish them for developer docs. None of the three crawler pages says its crawlers read yours. | OpenAI, Anthropic, Perplexity, llmstxt.org |
| “Thousands of sites use it, so it must work.” | Adoption is concentrated in documentation companies, which benefit when coding agents read their docs. Adoption is not evidence of a search effect. | Our 41-domain study |
| “Chrome audits for it, so Google must use it.” | Lighthouse is a Chrome developer tool and marks a missing file as not applicable. Google Search's own AI guidance says no AI text files are needed. | Lighthouse, Google |
| “Crawlers might start reading it next year.” | Possible. If a vendor documents it, the file takes an afternoon to add then. Crawlable pages are needed either way. | llmstxt.org changes |
| “My server logs show requests for /llms.txt.” | A request shows something fetched it, not that an AI product used it. Verify the requester's identity before drawing conclusions. | How to verify a crawler |
| “It is cheaper than fixing my site.” | It is cheaper, but it points to the same pages. If those are blocked or script-only, the file points crawlers at content they cannot read. | Google, Vercel, 2024 |
Frequently Asked Questions
Does Google use llms.txt for AI Overviews or AI Mode?
Google's documentation does not say it does. Its AI features guidance states: “You don't need to create new machine readable files, AI text files, or markup to appear in these features.” A page needs to be indexed and eligible for a snippet.
Does ChatGPT read llms.txt?
OpenAI's crawler documentation, read on October 9, 2026, describes OAI-SearchBot, controlled through robots.txt, and ChatGPT-User for user requests, to which it says robots.txt rules may not apply. It does not say either reads other sites' llms.txt files. OpenAI publishes an llms.txt for its own developer docs.
Does Claude or Perplexity read llms.txt?
Neither Anthropic's nor Perplexity's crawler documentation says so. Both describe a search bot controlled by robots.txt and a separate fetcher for user requests; Perplexity says its user fetcher generally ignores robots.txt. Perplexity publishes an llms.txt index for its own documentation.
Will adding llms.txt hurt my site?
No source we read says it can. Lighthouse marks a missing file as not applicable and only flags a server error at the path. Keep the file accurate, public-safe and consistent with robots.txt.
What did Google's John Mueller say about llms.txt?
In an April 2025 Reddit comment he said none of the AI services had said they use it and compared it to the keywords meta tag. That is a staff comment, not Google documentation.
Do AI crawlers render JavaScript?
Vercel reported in December 2024 that OpenAI, Anthropic, Meta, ByteDance and Perplexity crawlers did not render JavaScript, while Gemini (via Googlebot) and Applebot did. Put key facts in server-rendered HTML to be safe.
Is llms.txt the same as robots.txt for AI?
No. robots.txt sets crawler access rules. The llms.txt proposal says the two “have different purposes”: llms.txt is a curated Markdown index for agents and grants or refuses nothing.
Who should still publish an llms.txt file?
Sites with substantial documentation that developers query through coding agents, where the proposal says the files are used most. For other sites it is optional housekeeping, not a search requirement.
Sources and Methodology
We read each source below on October 9, 2026 and searched every crawler page for the string “llms”. “Does not mention” means that search found no statement that the crawler reads llms.txt; it does not prove no system ever fetches the file. Quotations are verbatim and dated to the day we read them. The John Mueller quotation is from a public Reddit comment, which we read in an Internet Archive copy captured August 27, 2026 (reddit.com itself blocked our request) and which Search Engine Journal also reported; it is not Google documentation. The JavaScript-rendering finding is Vercel's 2024 measurement of nextjs.org, its own network and two job-board sites, not ours. Adoption figures come from our own September 2026 study. We did not test whether any AI product reads llms.txt, and we make no claim about how any system ranks or cites pages.
- official Google Search Central, AI features and your website, last updated December 10, 2025. Read October 9, 2026.
- official Google Search Central, Google Search technical requirements, last updated December 18, 2025. Read October 9, 2026.
- official Google Search Central, SEO link best practices for Google, last updated December 10, 2025. Read October 9, 2026.
- official OpenAI, Overview of OpenAI Crawlers. Read October 9, 2026.
- official Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?, April 7, 2026. Read October 9, 2026.
- official Perplexity, Perplexity Crawlers. Read October 9, 2026.
- official Chrome for Developers, llms.txt (Lighthouse agentic browsing audit), last updated May 5, 2026. Read October 9, 2026.
- proposal Jeremy Howard, The /llms.txt file, first published September 3, 2024, and Changes: v2 (August 2026). Read October 9, 2026.
- trade press Roger Montti, Search Engine Journal, Google Says LLMs.Txt Comparable To Keywords Meta Tag, April 17, 2025, quoting John Mueller in r/TechSEO. Read October 9, 2026.
- primary record John Mueller (u/johnmu), comment in r/TechSEO, “LLM.txt - where are we at?”, April 16, 2025, Internet Archive capture of August 27, 2026. Read October 9, 2026.
- vendor research Vercel, The rise of the AI crawler, December 17, 2024. Read October 9, 2026.
- our research Media Morpheus, Does Anyone Actually Publish llms.txt?, data collected September 24, 2026.