What Was Announced on October 30, 2026?
Four AI providers said on the same day that they would read one shared robots.txt group as a statement about AI training, so a site can state its preference once instead of naming every company’s crawler. The mechanism is an Internet-Draft titled “Machine-Readable Training Preference Signal,” filed by Kevin Kelley of Anthropic and dated October 30, 2026 (IETF Datatracker). The IETF, the Internet Engineering Task Force, is the body that publishes internet standards; its AI Preferences (AIPREF) working group has been drafting a shared vocabulary for how content may be used by AI systems.
Google’s post, signed by Search Advocate John Mueller, says the control “will help websites express their AI training preferences across multiple AI providers in a single step” and that Google aims “to support this new control along with other AI providers in 2027” (Google Search Central Blog). Microsoft’s post, by Bing principal product manager Krishna Madhavan, says Microsoft “is working with other industry participants to bring a shared AI training preference to site owners and publishers” (Bing Blog). Kelley announced Anthropic’s support on the IETF’s ai-control mailing list the same afternoon (IETF ai-control list).
OpenAI also published a post, which returned an access error when we tried to read it, so we rely on reporting for its contents: Engadget and Unite.AI both report that OpenAI said it would implement the signal “in the coming months” (Engadget; Unite.AI). Treat the OpenAI timing as reported, not read at source.
The token is a different route from the one we described in September, in our explainer on telling an AI not to train on your site. Then, the working group’s attachment draft proposed a Content-Usage directive inside robots.txt. The new draft instead uses a policy token in an ordinary user-agent group, and it comes with public commitments from four of the largest AI providers on the day it was posted.
How Does the __usage_ai-training Token Work?
A site adds a robots.txt group addressed to the user agent __usage_ai-training and uses Disallow and Allow lines to say which paths it does not want used for training. The draft’s own example opts a whole site out except one folder:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/The draft says that example expresses a preference against AI training everywhere except /press-kit/, where it expresses a preference in favor (IETF Datatracker). Three rules in the draft matter more than the syntax:
- It is a usage preference, not a crawl block. The draft says the policy token is not a crawling directive, and that operators who want to limit crawling should use groups for crawlers’ own product tokens. A crawler that honors it can still fetch the page.
User-Agent: *does not count. Wildcard groups must not be read as matching the policy token, so an existing catch-all rule says nothing about training.- Training means one thing. The draft uses the AIPREF definition, “Using an asset to modify the learned parameters of a generative AI model,” and excludes models used only for non-generative tasks.
The draft has no formal standing yet. It is an individual submission, which the IETF page says is “not endorsed by the IETF,” and it expires on May 3, 2027 unless it is revised or adopted (IETF Datatracker). By our count that is 185 days after it was posted.
What Does the Token Switch Off at Each Company?
The same line means slightly different things at each company, because each maps it onto controls it already had. The table is our comparison of the four companies’ own pages, read on October 31, 2026; the OpenAI row relies on reporting and on its existing crawler page.
| Company | What the shared opt-out maps to | What stays separate | Start |
|---|---|---|---|
| Treated as a Google-Extended disallow, if the site does not name Google-Extended itself | Google Search inclusion and ranking, which Google-Extended does not affect | “Early 2027” | |
| Microsoft | Its new AI Training preference, site-wide or by path | AI Grounding, a separate page-level setting; Bing search eligibility | February 1, 2027 |
| Anthropic | Training preference; complements existing crawler controls | ClaudeBot, Claude-User and Claude-SearchBot rules still apply | Started, rollout across services “will take time” |
| OpenAI | Training preference (reported); GPTBot remains its training crawler | OAI-SearchBot, which governs ChatGPT search | “In the coming months” (reported) |
The Google row needs a closer read. Google’s crawler documentation says Google-Extended controls whether content may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI (Google crawler documentation). Google’s announcement adds that “Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps” (Google Search Central Blog). Our reading, which Google has not stated in these words: a site that uses only the shared token to refuse training should expect Google to treat it as a Google-Extended opt-out, which by Google’s own description also covers grounding in Gemini Apps. A site that wants Gemini Apps grounding has to name Google-Extended, and the pages we read do not describe a way to allow grounding while refusing training at Google.
Microsoft draws the line the other way. Its post says “Training and grounding are independent preferences” and “your training setting does not change your grounding setting, and vice versa,” with grounding set per page through a meta tag such as <meta name="Bingbot" content="ai-grounding:disallow"> or an X-Robots-Tag header (Bing Blog).
When Does Each Company Start Honoring It?
Anthropic says it has begun; Microsoft starts on February 1, 2027; Google says early 2027; OpenAI is reported as “the coming months.” Kelley’s post says support is being implemented but that “It’ll take us some time to add support across our services” (IETF ai-control list). Google’s crawler page says it “intends to support” the signal “in early 2027” (Google crawler documentation).
Days after the October 30, 2026 announcement (our arithmetic from Microsoft’s published dates)
Source: Microsoft Bing, “Your content, your choice: clearer preferences for training and grounding.” Read October 31, 2026. Day counts are Media Morpheus Research arithmetic.
Microsoft is the only one of the four companies to publish exact dates. Google says “early 2027,” OpenAI “the coming months” (as reported), and Anthropic says it has started, so they are not charted. The Dec 15 and Sep 30 milestones apply only to sites that use NoArchive or NoCache today.
The chart shows that even the one company with a calendar gives site owners about three months before the token counts and about four months before the old Bing tags stop being read for training and grounding.
Microsoft’s timeline has a deadline attached. It says that after March 1, 2027, NoArchive and NoCache are no longer read as training or grounding preferences, and it urges sites that use them now to move to the new settings (Bing Blog). Any site that relied on NoCache to keep its pages out of Copilot answers has a job to do before then.
Does Opting Out of Training Remove a Site From AI Search?
No, not by itself: the token speaks only to training, and each company keeps search and answer features on separate controls. OpenAI’s crawler page says each setting “is independent of the others,” so a site can allow OAI-SearchBot “in order to appear in search results while disallowing GPTBot” for training (OpenAI). Anthropic says disabling Claude-SearchBot “may reduce your site’s visibility and accuracy in user search results,” while ClaudeBot is the training crawler (Anthropic Help Center). Microsoft says neither of its new preferences changes Bing search eligibility (Bing Blog).
The exception is the Google grounding point above. Google-Extended does not affect Google Search, so AI Overviews in Search are not covered, but Gemini Apps grounding is. Our guide to blocking or allowing AI crawlers in robots.txt lists the search crawlers to keep open, and our explainer on whether blocking a crawler removes you from search covers the crawler-level effects.
Nor does the token undo anything. Engadget notes that it does not reverse past scraping (Engadget), and Anthropic’s own wording is about a site’s “future materials.”
What Should a Company Do With Its Newsroom and Press Kit?
Most companies want their newsroom, boilerplate and press kit read and repeated, so the sensible default is to leave those paths open even if other parts of the site opt out. The draft’s own example, a site-wide Disallow with Allow: /press-kit/, is close to what many communications teams will want. It is our recommendation, not a rule, that the paths to keep allowed are the ones written to be quoted: releases, fact sheets, executive bios and the company boilerplate.
The calculation is different for publishers and for companies whose content is the product, such as research firms, which may reasonably refuse training across the board. For them the trade-off at Google is the real decision, because refusing Gemini training through Google-Extended also refuses Gemini Apps grounding, which Google ties to “clicks via grounding.”
Three practical steps follow from the documents. First, check that no User-Agent: * rule is doing work you thought covered training: under the draft, it does not. Second, decide on Google-Extended explicitly rather than letting the shared token decide for you. Third, if you use Bing’s NoCache or NoArchive today, put a March 1, 2027 reminder in the calendar.
What This Means for Anyone Buying a Placement
For a placement buyer, the token changes who can train on the publication that carries your story, not whether AI search can find it. If a trade title adds a site-wide training opt-out next year, its articles about your company should still be available to the search and answer crawlers it leaves open. What could change is the Gemini Apps case at Google, where an outlet that refuses training without naming Google-Extended may also drop out of grounding.
None of this makes AI visibility something money can buy. Microsoft says plainly that “Neither choice guarantees inclusion, traffic, or payment” (Bing Blog). An outlet’s robots.txt is a question worth asking before a paid placement, alongside the questions in our guide on checking what AI says about your company.
Methodology and Limitations
Four limits apply. First, OpenAI’s announcement returned an access error, so its timing comes from Engadget and Unite.AI and is labelled as reported. Second, the draft is an individual Internet-Draft and can change; we describe version 00. Third, our reading of the Google grounding effect combines two Google pages and is our inference; Google has not published an example of it. Fourth, none of the companies has published a way to check whether its systems are honoring the token on a given site.
Conclusion
A single training line is a real simplification, and the four companies that matter most for AI answers have now said they will read it. But until early 2027 it is mostly a promise, it does not reach search or answer crawlers, and at Google it brings Gemini Apps grounding along with it. Companies should decide path by path, keep their press material open, and treat the shared token as one setting among several rather than the only one.
Frequently Asked Questions
What is __usage_ai-training in robots.txt?
It is a shared policy token from an October 30, 2026 Internet-Draft. A robots.txt group addressed to User-Agent: __usage_ai-training with Disallow or Allow lines states whether those paths may be used to train generative AI models.
Which companies will honor the shared AI training opt-out?
Google, OpenAI, Microsoft and Anthropic said on October 30, 2026 that they will. Anthropic says it has started implementing it, Microsoft starts February 1, 2027, Google says early 2027, and OpenAI was reported as the coming months.
Does the shared token block AI crawlers?
No. The draft says it is not a crawling directive. Crawlers that honor it may still fetch pages; to stop crawling, use rules for each crawler’s own user agent.
Will opting out of AI training remove my site from ChatGPT search or Google AI Overviews?
Not by itself. OpenAI keeps ChatGPT search on OAI-SearchBot, and Google says Google-Extended does not affect Google Search. At Google, though, the opt-out is treated like a Google-Extended disallow, which also covers grounding in Gemini Apps.
Does User-agent: * count as an AI training opt-out?
No. The draft says wildcard groups must not be read as matching the __usage_ai-training token, so a catch-all rule says nothing about training preference.
Should a company opt its press releases out of AI training?
Usually not, in our view. Most companies want releases, bios and boilerplate quoted, so they can opt out elsewhere and keep a press path allowed, as the draft’s own /press-kit/ example does. This is our recommendation, not a platform rule.
Sources and Methodology
This explainer is built from the IETF Internet-Draft and its Datatracker page, Google’s Search Central post and crawler documentation, Microsoft Bing’s announcement, Anthropic’s post to the IETF ai-control list and its crawler help page, OpenAI’s crawler documentation, and reporting by Engadget and Unite.AI on OpenAI’s announcement, all read on October 31, 2026. The company-by-company table, the day counts and the newsroom recommendations are our own analysis, not platform rules. We did not test whether any crawler honors the token today.
- standards IETF Datatracker, draft-kelley-aipref-training-preference-signal-00, Machine-Readable Training Preference Signal. Token, example, definition of training, wildcard and crawling rules, status and expiry. Read October 31, 2026.
- official Google Search Central Blog, Supporting a shared AI training control for websites. Google’s plan, Google-Extended mapping, Gemini Apps grounding, 2027 timing. Read October 31, 2026.
- platform documentation Google, Google’s common crawlers (Google-Extended). What Google-Extended covers; “early 2027” support note. Read October 31, 2026.
- official Microsoft Bing Blog, Your content, your choice: clearer preferences for training and grounding. Training and grounding preferences, syntax, timeline, quotes. Read October 31, 2026.
- official IETF ai-control mailing list, Post by Kevin Kelley, October 30, 2026. Anthropic has started implementing; complements existing controls. Read October 31, 2026.
- platform documentation Anthropic Help Center, Does Anthropic crawl data from the web, and how can site owners block the crawler?. ClaudeBot and Claude-SearchBot roles. Read October 31, 2026.
- platform documentation OpenAI, Overview of OpenAI Crawlers. GPTBot and OAI-SearchBot are independent settings. Read October 31, 2026.
- trade press Engadget, Google joins OpenAI and Anthropic in promising new scraping controls. OpenAI “in the coming months”; no effect on past scraping. Read October 31, 2026.
- trade press Unite.AI, OpenAI to Honor Shared Robots.txt Signal for AI Training Preferences. OpenAI timing and continued robots.txt controls, as reported. Read October 31, 2026.