Does Anyone Actually Publish llms.txt?

What llms.txt is meant to be

The llms.txt proposal was published by Jeremy Howard on 3 September 2024 and has been revised since. Its diagnosis is reasonable: “web pages are built for people”, and “an HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise”.

Its remedy is a markdown file named llms.txt placed at the site root, containing an H1 heading, an optional blockquote summary, and H2-delimited lists of links with notes — a curated map of what matters on the site, aimed at machine readers rather than people.

The proposal asks for more than the index file. Site owners should also provide a “clean markdown version of those pages at the same URL as the original page, either with .md appended or with the extension replaced by .md”. That is a substantial publishing commitment, not a single file.

It is worth being clear that this is a proposal by an individual, not a standard issued by a standards body and not a specification any search or AI vendor has committed to honour. That distinction turns out to matter a great deal.

The second half of that requirement is the part vendors rarely mention when quoting a price. Publishing the index file is trivial; publishing and maintaining a clean markdown twin of every page you care about is an ongoing content operation, and it has to stay synchronised with the human version or it becomes a source of stale, contradictory text about your own company. A proposal to add llms.txt to a site is therefore usually a proposal to add one file and quietly ignore the accompanying obligation that makes it useful.

What we found

We requested /llms.txt from 41 domains across four categories on 24 September 2026, following redirects and treating a file as genuine only if it returned HTTP 200 and was not HTML, since several sites answer any unknown path with their homepage.

News and magazines: zero adoption. Sixteen domains returned a definitive 404 — among them The New York Times, The Guardian, CNN, Forbes, Business Insider, Axios, The Atlantic, Wired, TechCrunch, the BBC, CNBC, Fortune, Entrepreneur, Adweek, Nature and NPR. A further seven, including The Wall Street Journal, Reuters, the Associated Press, Bloomberg and The Washington Post, blocked our request outright and are recorded as not retrievable rather than as not publishing.

Developer and SaaS companies: near-universal. Ten of twelve served one, including Stripe, Cloudflare, HubSpot, Shopify, Notion, Zapier, Netlify, Vercel, Supabase and Linear. Documentation hosts behaved the same way, with files at docs.stripe.com, developers.cloudflare.com, docs.github.com and svelte.dev.

Wire services split. PR Newswire and AccessWire serve one; GlobeNewswire and EIN Presswire return 404. Business Wire blocked the request. PR Newswire's file is the one we had already encountered in an earlier study, published under a comment describing it as an AI context file.

One methodological trap is worth naming because it inflates every casual adoption estimate. A number of sites answer a request for any unknown path with their homepage and an HTTP 200, so a check that only looks at the status code records them as adopters. In our first pass several domains looked like hits until we inspected the body and found HTML. We counted a file only where the response was both 200 and not HTML, which is why our figures are lower than the impression a quick survey gives.

Do the AI labs publish one themselves?

The proposal states that “The AI labs themselves publish llms.txt files”, naming OpenAI, Anthropic and Gemini. Because that claim is load-bearing for the argument, we tried to verify each rather than repeat it.

Anthropic: confirmed, though not where you would guess. A request to docs.anthropic.com redirects twice, via platform.claude.com/docs/llms.txt, and resolves to a file at platform.claude.com whose first line reads “# Anthropic Developer Documentation”. Perplexity, which the proposal does not name, also serves one at docs.perplexity.ai.

OpenAI: not confirmed. platform.openai.com/llms.txt returns 404, and openai.com/llms.txt returns 403 to our client. A 403 means we were refused, not that nothing is there, so we record this as unverified rather than absent.

Google: not confirmed either. developers.google.com/llms.txt returns 404, and ai.google.dev/llms.txt enters an authentication redirect loop that never resolves for an anonymous request. We report what our probe could and could not establish; anyone asserting more about these two domains should say how they checked.

The question nobody answers

Adoption is one thing; consumption is another. The decisive question is whether any AI system actually reads these files, and here the proposal is notably careful. It describes sites publishing them and uses aspirational language about what agents are expected to do. It does not claim that any named AI system currently reads llms.txt.

That silence is the whole story. A file format is only useful if something consumes it, and the burden of establishing consumption sits with whoever is selling the work of producing one.

The one major vendor that addresses the question directly answers it in the negative. Google's guidance for site owners states there are “no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary”, and adds: “You don't need to create new machine readable files, AI text files, or markup to appear in these features.”

Read against a proposal asking you to publish a machine-readable AI text file, that sentence is about as close to a direct refusal as vendor documentation gets.

Why the adoption pattern makes sense

The split we measured is not random, and it explains the file's real function. The companies publishing llms.txt are overwhelmingly those whose product is documentation: developer tools whose users ask coding assistants questions that the assistant answers by reading docs.

For them the file has a plausible, self-interested purpose independent of any search benefit. A clean index of documentation helps anyone building on their API, whether the reader is a person, a script or an agent, and it costs little because the content is already structured.

News publishers have the opposite incentive. Their product is the article itself, many of them are in litigation or licensing negotiations over machine reading, and several block AI crawlers outright. Publishing a curated markdown index inviting machines to ingest the material would cut directly against that position. Zero adoption among news domains is a coherent business decision rather than a technology gap.

That also means the pattern tells you nothing about whether the format works. It tells you which businesses benefit from machines reading their content and which believe they are harmed by it.

What this means for anyone buying coverage

If a vendor proposes to add llms.txt to your site as an AI visibility service, the fair questions are: which AI system reads it, where is that documented, and what changes if it is absent. We could not find a vendor commitment to honour it, and the proposal itself does not assert one.

None of that makes publishing one harmful. It is a small file, it costs little, and if you run a documentation-heavy site it may genuinely help people and tools navigate your material. Treat it as cheap housekeeping with an uncertain payoff, not as an investment with a return.

Be especially wary of the inference from adoption. That Stripe and Cloudflare publish one is evidence about what documentation companies do, not evidence that publishing one produces visibility. The sites most invested in being read by machines are the ones with the least to lose by it.

Media Morpheus sells media and podcast placement, and AI visibility services are adjacent to what we sell. On the evidence here, the only vendor documentation addressing the question says new machine-readable files are not needed — and we would rather tell you that than bill for producing one.

Methodology and limitations

On 24 September 2026 we requested https:///llms.txt for 41 domains across four categories: news and magazines, wire services, developer and SaaS companies, and documentation hosts. Requests followed redirects and used a browser user-agent where an initial request was refused. We counted a file as genuine only where the response was HTTP 200 and the body was not HTML, because several domains answer unknown paths with their homepage, which a naive status check would record as adoption. Domains returning 401, 403 or a connection failure are reported as not retrievable and excluded from denominators rather than counted as non-adopters; this affected seven news domains and Business Wire. For the AI labs we traced redirect chains manually. Limitations: the sample is purposive rather than random, so the percentages describe these categories and not the web; a file's presence says nothing about its quality or maintenance; and we tested only the root path, so a site publishing at a subdomain we did not try would be missed, as very nearly happened with Anthropic.

Conclusion

llms.txt has real adoption in exactly one place: companies whose product is documentation. Among news and magazine publishers, adoption in our sample is zero, and that looks like a deliberate position rather than an oversight, given how many of the same publishers block AI crawlers. The proposal is a thoughtful response to a genuine problem, but it is an individual's proposal rather than a standard, it does not claim any AI system reads the files, and the one major vendor that addresses the question tells site owners plainly that new machine-readable AI files are not needed. Publish one if it helps your readers. Do not buy one as a visibility product.

Frequently Asked Questions

Sources

1. vendor research Media Morpheus, September 2026. Original measurement of /llms.txt across 41 domains on 24 September 2026, following redirects, excluding HTML soft-404s, and recording blocked requests as not retrievable. Method and limitations stated above.
2. official llmstxt.org. The llms.txt proposal by Jeremy Howard, published 3 September 2024: the stated problem, the file format and location, the accompanying markdown requirement, and the claim that AI labs publish such files.
3. official Google Search Central. Guidance on AI Overviews and AI Mode, including the statement that new machine-readable files, AI text files and markup are not needed to appear.
4. vendor research Media Morpheus, September 2026. Earlier measurement of wire service robots.txt, which first identified PR Newswire's llms.txt published under an AI context file comment.