How to Block or Allow AI Crawlers in robots.txt: Separate Training From Search

Time: under an hour for one site (our estimate) · Tools: a plain-text editor and access to your web root or your host's robots.txt setting · Cost: none · Skill level: beginner to intermediate

How to Block or Allow AI Crawlers at a Glance

What You Need Before Starting

• Write access to the file at /robots.txt on every host you want to cover. Google's robots.txt guide says the file must sit at the root of the host and applies only to that protocol, host and port, so blog.example.com needs its own.
• A plain-text editor. Google warns that word processors can add curly quotes and other characters that break the file, and says to save it as UTF-8.
• A copy of your current robots.txt, so you can roll back.
• A decision on each purpose (Step 1). The tokens below only make sense once you know what you want.
• Optional: access to server or CDN logs, and a verified property in Google Search Console, for checking the result.

What Is a robots.txt User-Agent Token?

A user-agent token is the short name a crawler operator tells you to put on a User-agent: line in robots.txt so that a group of rules applies to that crawler.

RFC 9309, the Robots Exclusion Protocol standard published in 2022, calls this the product token and says crawlers "MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group". Some tokens belong to real crawlers that fetch pages. Others, such as Google-Extended and Applebot-Extended, are control tokens: no request in your logs carries that name, and the token only tells the operator how it may use content its main crawler already fetched.

AI Crawler User-Agent Reference (October 2026)

The table lists the tokens each operator documents as of 6 October 2026, what the operator says the token is for, and what the operator's own documentation says about robots.txt. The right-hand column reports operator statements, not our testing.

Sources: OpenAI crawler overview, Anthropic Help Center, Google's common crawlers list, About Applebot, Perplexity Crawlers, Common Crawl CCBot and FAQ. For which of these tokens publishers actually name in practice, see our count of which AI crawlers publishers name in robots.txt.

Step 1: Decide Which AI Purpose You Are Blocking

Write down, for each operator, whether you want to allow or refuse three separate things: model training, AI search indexing, and fetches a user triggers by asking an assistant a question.

Why: operators now split these jobs across different tokens. OpenAI says "each setting is independent of the others", so a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. A single "block AI" rule cannot express that, and blocking the wrong token can remove you from an AI search product you wanted to be in.

Process: use the reference table above. For training, the tokens are GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. For search, they are OAI-SearchBot, Claude-SearchBot and PerplexityBot. Decide separately on CCBot, whose dataset is open to anyone, and note one Google-specific trade-off: Google-Extended covers both Gemini training and grounding in Gemini Apps, so blocking it is not purely a training opt-out.

Expected result: a policy such as "no training, yes to AI search, yes to user fetches".

Common mistake: treating Googlebot as an AI token. Google says AI Overviews and AI Mode are part of Search and that robots.txt rules for Googlebot are "the control" for them (AI features and your website). Blocking Googlebot blocks Google Search crawling. For the search consequences, see whether blocking a crawler removes you from search.

Step 2: Open the robots.txt File at Your Site Root

Open the existing robots.txt for each host, or create a new plain-text file named exactly robots.txt.

Why: RFC 9309 says the rules "MUST be accessible in a file named "/robots.txt" (all lowercase) in the top-level path of the service". Google adds that a site can have only one robots.txt and that it cannot live in a subdirectory. Anthropic's help article tells site owners to add the block "for every subdomain that you wish to opt out from".

Process: on a static site, the file sits in the web root next to the home page. On a hosted CMS, check your host's documentation for its robots.txt setting; Google suggests contacting the host if you cannot reach the root.

Expected result: you can edit the file served at https://yourdomain/robots.txt.

Common mistake: editing the file on www and assuming it covers shop. or blog. subdomains, or the http:// version of the site.

Step 3: Add a Group for Each AI User-Agent Token

Add one group of rules for the tokens you want to treat the same way, with each token on its own User-agent: line, followed by Allow or Disallow rules.

Why: Google's documentation says a group takes "one or more" user-agent lines, so tokens that share a policy can share a group. RFC 9309 says that if no group matches a crawler, it "MUST obey the group with a user-agent line with the "*" value, if present".

Process: pick one of the four rule sets below and edit the paths. Within a group, RFC 9309 says the most specific match wins (the rule with the most characters), and if an allow and a disallow rule are equally specific, the allow rule "SHOULD be used". Paths are case-sensitive; tokens are not.

Rule set A: block training, keep AI search

# Block AI model training, keep AI search crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
Disallow: /admin/

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

This keeps the search and user-fetch tokens of OpenAI, Anthropic and Perplexity, plus Googlebot and Applebot through the * group, while refusing the training tokens. ChatGPT-User and Perplexity-User are omitted because their operators say robots.txt may not govern them.

Rule set B: block every AI token listed

# Block every AI crawler and fetcher named here
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/

This still leaves Google's AI Overviews and AI Mode, which use Googlebot, and Apple's search features, which use Applebot. OpenAI and Perplexity say their user fetchers may not obey the user-fetch lines.

Rule set C: allow AI crawlers explicitly

# Allow AI crawlers, but keep the same private paths closed to them
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /checkout/

A site with no robots.txt is already open: RFC 9309 says a 4xx response means a crawler "MAY access any resources on the server". Explicit allow groups only add value if you also repeat your private paths, as here.

Rule set D: keep one section out of training

# Keep one section out of training, leave the rest open
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /research/
Disallow: /admin/

User-agent: *
Disallow: /admin/

Expected result: every token in your Step 1 policy appears in exactly one group.

Common mistake: copying a token list from an old blog post. Anthropic's article, updated 7 April 2026, lists three bots: ClaudeBot, Claude-User and Claude-SearchBot. Use the names on the operator's current page.

Step 4: Copy Your Private-Path Rules Into Each Named Group

Check every group that names a specific bot and add the Disallow lines from your User-agent: * group that should still apply to it.

Why: Google's robots.txt guide says "a user agent can match only one rule set, which is the first, most specific group that matches a given user agent". Once GPTBot has its own group, it no longer reads the * group. This file opens /admin/ to GPTBot:

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Allow: /

Process: repeat the private paths inside the named group:

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Allow: /
Disallow: /admin/

Expected result: no named bot can reach a path that the * group closes.

How to verify: read each named group on its own and ask what a bot that only sees that group may fetch.

Step 5: Publish the File and Check It Returns HTTP 200

Upload the file, then load https://yourdomain/robots.txt in a private browsing window, as Google's guide recommends, and confirm the rules display as plain text.

Why: the HTTP status changes what crawlers do. Under RFC 9309, a 4xx response means a crawler may access anything, and a 5xx server error means it "MUST assume complete disallow". A robots.txt that errors can shut out every compliant crawler, including search engines.

Process: from a terminal, curl -sI https://yourdomain/robots.txt shows the status line and content type.

Expected result: HTTP/2 200 and a text content type, with your rules in the body.

Common mistake: a CDN or hosting rule that redirects robots.txt to an HTML page, or a staging-site robots.txt that ships to production.

Step 6: Wait Up to 24 Hours, Then Check Logs

Allow up to a day for crawlers to pick up the new rules, then compare your server logs against the policy.

Why: RFC 9309 lets crawlers cache robots.txt but says they "SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable". OpenAI says ChatGPT search can take "~24 hours" to adjust, and Perplexity says changes may take "up to 24 hours".

Process: filter logs by user-agent string, then confirm the source IPs belong to the operator (see How to Verify It Worked below).

Expected result: blocked crawlers request /robots.txt but not disallowed paths.

Common mistake: looking for Google-Extended or Applebot-Extended in logs. Google says Google-Extended has no separate HTTP user agent, and Apple says Applebot-Extended does not crawl.

How to Verify It Worked

• File check: the robots.txt URL returns HTTP 200 with your rules as plain text on every host you edited.
• Google's parser: the robots.txt report in Search Console shows the version Google fetched and any parsing problems. Google also publishes its open-source robots.txt library for local testing (Create and submit a robots.txt file).
• Logs, by user agent: after 24 hours, requests from blocked crawler names should stop at /robots.txt. OpenAI says its crawlers may add a robots.txt marker to the user-agent string when fetching the file.
• Logs, by IP: confirm a request is genuine before drawing conclusions. OpenAI publishes IP lists at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json; Perplexity at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json; Common Crawl at index.commoncrawl.org/ccbot.json; Apple supports reverse DNS in *.applebot.apple.com. Common Crawl warns that it is "aware of crawlers falsely identifying themselves as CCBot". Our guide to verifying a crawler is real walks through the checks.
• Search visibility: if you kept OAI-SearchBot or PerplexityBot allowed, your pages can still be cited in those products. Being allowed is a precondition, not a guarantee of citation.

Common Problems and Fixes

How to Choose a Rule Set

For the standards work on that last row, see our explainer on telling an AI not to train on your site. For how major newsrooms configure these rules, see our check of fifteen news sites.

What Not to Do

• Do not block Googlebot to stop AI Overviews. Consequence: you lose Google Search crawling. Correction: use Google's snippet controls for AI features in Search.
• Do not list secret paths in robots.txt. RFC 9309 says listing paths "exposes them publicly and thus makes the paths discoverable". Correction: protect them with authentication.
• Do not rely on IP blocking alone for Anthropic. Anthropic says blocking its IP addresses "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file". Correction: use a robots.txt block.
• Do not assume a block is retroactive. Anthropic says restricting ClaudeBot signals that "future materials" should be excluded; OpenAI says disallowing GPTBot "indicates" content should not be used in training. Neither page says a block removes content already collected.
• Do not assume user fetchers obey. OpenAI says robots.txt "may not apply" to ChatGPT-User; Perplexity says Perplexity-User "generally ignores" it. Correction: if those visits must stop, that is a firewall decision.
• Do not depend on Crawl-delay everywhere. Anthropic calls it "non-standard" but supports it; Common Crawl obeys it; Apple says Applebot "does not follow crawl-delay". RFC 9309 does not define it.

Frequently Asked Questions

Sources and Methodology

We read each operator's current crawler documentation and the RFC on 6 October 2026 and quote them directly. We did not run crawl experiments for this guide, so statements about whether a crawler follows robots.txt are the operator's own claims, not our measurements. Operator pages change without notice; check the dates below before relying on a token. Time estimates are ours.

1. standard M. Koster, G. Illyes, H. Zeller, L. Sassman, RFC 9309: Robots Exclusion Protocol, IETF, September 2022. Read 6 October 2026.
2. official OpenAI, Overview of OpenAI Crawlers. Read 6 October 2026.
3. official Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?, updated 7 April 2026. Read 6 October 2026.
4. official Google, List of Google's common crawlers, last updated 14 July 2026. Read 6 October 2026.
5. official Google, Overview of Google crawlers and fetchers, last updated 12 June 2026. Read 6 October 2026.
6. official Google, Create and submit a robots.txt file, last updated 21 November 2025. Read 6 October 2026.
7. official Google Search Central, AI features and your website, last updated 10 December 2025. Read 6 October 2026.
8. official Apple, About Applebot, published 4 September 2026. Read 6 October 2026.
9. official Perplexity, Perplexity Crawlers. Read 6 October 2026.
10. official Common Crawl, CCBot and FAQ. Read 6 October 2026.