How to Create an llms.txt File at a Glance
| Step | Action | Expected result |
|---|---|---|
| 1 | Decide the scope: site root or a subpath | A target URL such as /llms.txt or /docs/llms.txt |
| 2 | Select the pages an agent most needs | A short, curated list rather than a full sitemap |
| 3 | Write the H1 and blockquote summary | The file opens with the site name and a one-paragraph summary |
| 4 | Add H2 link sections and an Optional section | Every link is a Markdown list item with a short note |
| 5 | Upload the file and serve it as plain text | The URL returns HTTP 200 with Markdown, not HTML |
| 6 | Optionally publish Markdown copies of pages and link to them | .md versions resolve and pages point to them |
What You Need Before Starting
- Upload access to the root of your site (or the subpath you want to cover), or a CMS that can generate the file.
- A plain-text editor that saves UTF-8 without smart quotes.
- A list of your most useful pages and a one-line description of each.
- A robots.txt that already says what you want crawlers to do. llms.txt does not replace it; see how to block or allow AI crawlers in robots.txt.
- Realistic expectations. Read the section on adoption below before you spend more than an afternoon on this.
What Is llms.txt?
llms.txt is a Markdown file, placed at /llms.txt or another path on a website, that gives AI agents a short summary of the site and a curated list of links to the pages they are most likely to need.
The proposal at llmstxt.org was written by Jeremy Howard and dated 3 September 2024; its changes page records a v2 revision in August 2026. The proposal is hosted on GitHub and "open for community input". It is not published by a standards body such as the IETF or W3C.
The proposal distinguishes the file from robots.txt and sitemaps. Robots.txt tells automated tools "what access to a site is considered acceptable, such as for search indexing bots"; llms.txt "is instead used on demand, when an agent needs information about a topic while assisting a user". A sitemap lists every indexable page; llms.txt is meant to be a curated overview small enough to fit in a model's context.
Which AI Systems Read llms.txt? What the Evidence Shows
As of 6 October 2026, the evidence is thinner than the marketing around the file suggests.
- Google says it is not needed for Search. Google's AI features and your website guidance states: "You don't need to create new machine readable files, AI text files, or markup to appear in these features."
- No crawler documentation we checked mentions reading it. The crawler pages of OpenAI, Anthropic, Google, Apple and Perplexity describe robots.txt controls. None of them says the crawler fetches or uses llms.txt.
- AI companies publish the file for their own developer docs. OpenAI's and Perplexity's documentation pages point readers to an llms.txt index, and the llmstxt.org proposal names OpenAI, Anthropic and Gemini as publishers. Publishing a file is not the same as their assistants reading other sites' files.
- Chrome audits for it. Chrome's Lighthouse llms.txt audit calls the file "an emerging convention". It flags a server error at
/llms.txtbut marks a 404 as not applicable, "as providing the file is optional at the moment". - The proposal's own claims. The v2 changes page says: "Thousands of sites now publish an llms.txt file, documentation platforms generate one automatically, and coding agents use them reliably." Those are the author's statements; it does not name a specific assistant that fetches llms.txt automatically.
Our own check of 41 domains in September 2026 found the file on 10 of 12 developer and SaaS sites and on none of 16 news and magazine sites we could read. The detail is in our research on who actually publishes llms.txt. The pattern points to the use case where llms.txt is most plausible: documentation that people query through coding assistants and agents.
Domains serving a genuine llms.txt, of those that returned a readable answer, September 24, 2026
Source: Media Morpheus, Does Anyone Actually Publish llms.txt? (41 domains, data collected September 24, 2026).
Blocked requests are excluded rather than counted as non-adopters; HTML responses were treated as soft 404s.
Step 1: Decide the Scope of the File
Choose whether one file at /llms.txt should describe the whole site, or whether a section such as /docs/ needs its own file.
Why: v2 of the proposal says the file can sit "at the root path /llms.txt of a website or at any subpath", that "a file covers the URLs under its path", and that where more than one applies "agents should use the most specific one".
Process: a small business site needs one root file. A company with a large documentation area can publish /llms.txt for the company and /docs/llms.txt for the docs.
Expected result: a list of one or more target URLs.
Common mistake: putting the file under /.well-known/. The proposal considered that location and chose not to use it.
Step 2: Select the Pages an Agent Most Needs
List the pages that answer the questions people most often ask about your site, product or organisation, and leave out everything else.
Why: the proposal says the file "stays small enough to fit in context" and that "the detail lives behind the links, and is fetched only when needed". It explicitly contrasts this with a sitemap, which covers documents that "in aggregate will be too large to fit in an LLM context window".
Process: start with what a buyer, journalist or developer would need: what you do, pricing or rate cards, key documentation, policies, and contact or support routes. The proposal allows links to external URLs if they help explain your content.
Expected result: a curated list grouped into a few themes.
Common mistake: pasting the whole sitemap. That defeats the purpose of a curated index.
Step 3: Write the H1 and the Blockquote Summary
Open a new file named llms.txt and write a single H1 with the name of the site or project, followed by a blockquote summary.
Why: the spec says the H1 "is the only required section" and the blockquote should contain "key information necessary for understanding the rest of the file". After the blockquote you may add paragraphs or lists, but no headings, with further detail on how to read the links.
Process: this is the skeleton from llmstxt.org:
# Title
> Optional description goes here
Optional details go here
## Section name
- [Link title](https://link_url): Optional link details
## Optional
- [Link title](https://link_url)Expected result: the first line starts with # and the next block starts with > .
Common mistake: writing marketing copy in the summary. The proposal's guidelines ask for "concise, clear language" and to "avoid ambiguous terms or unexplained jargon".
Step 4: Add H2 Link Sections and an Optional Section
Group your links under H2 headings, one Markdown list item per link, each with a short note after a colon.
Why: the spec defines each H2 section as a "file list" in which every item contains "a required markdown hyperlink [name](url), then optionally a : and notes about the file". Its guidelines ask for "brief, informative descriptions".
Process: put secondary links under an H2 titled Optional. The proposal calls this a convention "for secondary information: links an agent can skip when a shorter context is needed". A complete example for a fictional company:
# Example Analytics
> Example Analytics is a web analytics service for small online shops. This file lists the pages that explain what the product does, what it costs and how it handles data.
- Prices on the pricing page are in US dollars and exclude tax.
- The product does not use third-party cookies.
## Product
- [Product overview](https://example.com/product.md): What the service measures and who it is for
- [Pricing](https://example.com/pricing.md): Plans, limits and billing terms
## Policies
- [Privacy policy](https://example.com/privacy.md): What data is collected and how long it is kept
- [Security](https://example.com/security.md): Hosting, encryption and incident contacts
## Optional
- [Company history](https://example.com/about.md): Founding date, team and locationsExpected result: every link is in a list under an H2, and nothing appears between the H1 and the summary.
Common mistake: using H3 headings or bare URLs. The format is meant to be parseable with "classical programming techniques such as parsers and regex", so keep to the stated structure.
Step 5: Upload the File and Serve It as Plain Text
Upload llms.txt to the path you chose in Step 1 and load it in a browser.
Why: a file that returns an error, a login page or your HTML home page is not usable. Lighthouse flags server errors at the path.
Process: on a static site, place the file in the web root. On WordPress, Wix and documentation platforms, check whether your platform already generates one; llmstxt.org lists Yoast SEO, AIOSEO, Wix, Mintlify and GitBook among tools that can. The proposal does not specify a media type; serving it as text/plain or text/markdown with UTF-8 lets browsers and scripts display it as text (our recommendation, not part of the spec).
Expected result: the URL returns HTTP 200 and shows your Markdown.
Common mistake: assuming a 200 status means the file exists. Some sites return their home page with a 200 for any unknown path, which is why our adoption study counted a file only when the body was not HTML.
Step 6 (Optional): Publish Markdown Copies of Key Pages
Create a clean Markdown version of each linked page and point your llms.txt links at those versions.
Why: the proposal says links "should therefore point to LLM-friendly content", and proposes Markdown at the same URL with .md appended (page.html.md) or with the extension replaced (page.md). URLs without a file name use index.html.md or index.md.
Process: v2 adds discovery through standard link relations: rel="alternate" type="text/markdown" for a page's Markdown version and rel="describedby" for the llms.txt that covers it. In HTML:
<link rel="alternate" type="text/markdown" href="/pricing.md">
<link rel="describedby" href="/llms.txt">Or as an HTTP header, which the proposal says can be set in server or CDN configuration without editing pages:
Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"Expected result: each .md URL returns the same content as its HTML page without navigation or scripts.
Common mistake: letting Markdown copies drift from the HTML pages. Stale copies give machines a second, contradictory version of your facts. If you cannot keep them in sync, link to the HTML pages instead.
How to Verify It Worked
- Status and type:
curl -sI https://yourdomain/llms.txtreturns200and atext/content type, nottext/html. - Body: the first line of the response begins with
#, not<!doctype html>. - Lighthouse: the llms.txt audit in Chrome's Lighthouse agentic browsing checks should not report a server error.
- Links: every URL in the file returns 200. A short script or a link checker can confirm this.
- Agent test: the proposal suggests testing "by asking an agent questions about your content, giving it only your llms.txt as a starting point". If the agent cannot find pricing or contact details, the file is missing a link.
- Logs: requests for
/llms.txtin your server logs show something fetched it. They do not tell you that any AI product used it, so verify the requester before drawing conclusions (see how to verify a crawler is real).
Common Problems and Fixes
| Problem | Likely cause | Fix |
|---|---|---|
| Browser downloads the file instead of showing it | Server sends a binary or unknown content type | Serve it as text/plain or text/markdown with UTF-8 |
/llms.txt shows your home page | Catch-all routing returns index.html for unknown paths | Add the file, or exclude the path from the rewrite rule |
| Lighthouse reports an error for llms.txt | The path returns a 5xx server error | Fix the error; a 404 is marked not applicable |
| Links in the file return 404 | Pages moved or .md copies were never published | Update the links, or point them at the HTML pages |
| Agent answers miss key facts | Summary too vague or key page not listed | Add the page under a relevant H2 with a specific note |
| File is very long | Sitemap pasted in, or too many Optional links | Cut to the pages an agent most needs |
| AI crawlers still blocked after adding llms.txt | Access is controlled by robots.txt, not llms.txt | Change the robots.txt rules for the relevant tokens |
How to Choose a Method
| Situation | Recommended method | Why |
|---|---|---|
| Documentation site on a docs platform | Use the platform's built-in generator | llmstxt.org lists Mintlify and GitBook as generating the file |
| WordPress site | Use an SEO plugin's llms.txt feature, then edit the output | llmstxt.org lists Yoast SEO and AIOSEO generators |
| Hand-built or static site with few pages | Write the file by hand | Full control and a curated list in under an hour |
| Large site with separate product docs | One root file plus a /docs/llms.txt | v2 says the most specific file applies |
| Goal is Google AI Overviews or AI Mode | Skip it, or treat it as optional housekeeping | Google says no new AI text files are needed |
| News publisher that blocks AI crawlers | Usually skip it | An index inviting machine reading contradicts a block policy |
What Not to Do
- Do not use llms.txt to block AI training. It has no access semantics. Use robots.txt tokens such as
GPTBotandGoogle-Extended, and see our explainer on training opt-outs. - Do not buy it as a visibility guarantee. Consequence: you pay for a file no documented crawler commits to reading. Correction: ask any vendor which AI system reads it and where that is documented.
- Do not list pages that robots.txt disallows. Consequence: you invite agents to pages your other file refuses. Correction: list only pages you are happy for machines to fetch.
- Do not let it go stale. Consequence: agents follow dead links or out-of-date prices. Correction: review it whenever key pages change.
- Do not put private information in it. The file is public, like robots.txt.
Frequently Asked Questions
Does Google use llms.txt for AI Overviews?
Google's guidance for its AI features says you "don't need to create new machine readable files, AI text files, or markup to appear in these features." It does not say Google reads llms.txt.
Do ChatGPT, Claude or Perplexity read llms.txt?
The crawler documentation from OpenAI, Anthropic and Perplexity that we read on 6 October 2026 does not say their crawlers or assistants read llms.txt. All three companies document robots.txt controls instead.
Is llms.txt an official web standard?
No. It is a proposal by Jeremy Howard hosted at llmstxt.org and on GitHub, open for community input. It has not been published by a standards body such as the IETF or W3C.
Can llms.txt stop AI companies training on my content?
No. The proposal gives the file no access rules. Training opt-outs use robots.txt tokens such as GPTBot, ClaudeBot, Google-Extended and Applebot-Extended.
Can I put llms.txt in a subfolder instead of the root?
Yes. The proposal permits subpaths, and v2 defines their scope: a file at /docs/llms.txt covers the pages under /docs/, and where several files apply, agents should use the most specific one.
What is llms-full.txt?
The llmstxt.org proposal does not define a file by that name as of October 2026. It specifies only llms.txt and Markdown versions of individual pages, so treat any other file name as a tool-specific extension.
Should llms.txt list every page on my site?
No. The proposal describes a curated overview small enough to fit in a model's context, and says a sitemap is not a substitute because it covers too much.
Does the Optional section have to be called Optional?
It is a convention for secondary links that an agent can skip. Under v1 it told tools what to omit; v2 removed that mechanical meaning, so it is now a naming convention.
Does Chrome Lighthouse penalise a site without llms.txt?
No. Lighthouse marks a 404 for llms.txt as not applicable because the file is optional. It flags the page only if requesting the file causes a server error.
Sources and Methodology
We read the llmstxt.org proposal (v2), its changes page, Chrome's Lighthouse audit page, Google's AI features guidance and the crawler documentation of OpenAI, Anthropic, Google, Apple and Perplexity on 6 October 2026. "No documentation says it" means we searched those pages for llms.txt and found no statement that the crawler reads it; it does not prove no system ever fetches the file. Adoption figures come from our own September 2026 study, linked above. The example file describes a fictional company. Time estimates are ours.
- proposal Jeremy Howard, The /llms.txt file, v2, llmstxt.org, first published 3 September 2024. Read 6 October 2026.
- proposal llmstxt.org, Changes: v2 (August 2026). Read 6 October 2026.
- official Chrome for Developers, llms.txt (Lighthouse agentic browsing audit), last updated 5 May 2026. Read 6 October 2026.
- official Google Search Central, AI features and your website, last updated 10 December 2025. Read 6 October 2026.
- official OpenAI, Overview of OpenAI Crawlers. Read 6 October 2026.
- official Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?, updated 7 April 2026. Read 6 October 2026.
- official Google, List of Google's common crawlers, last updated 14 July 2026. Read 6 October 2026.
- official Apple, About Applebot, published 4 September 2026. Read 6 October 2026.
- official Perplexity, Perplexity Crawlers. Read 6 October 2026.
- our research Media Morpheus, Does Anyone Actually Publish llms.txt?, data collected 24 September 2026.