What is actually being standardised
The AI Preferences working group, chartered in the IETF's Web and Internet Transport area under chairs Mark Nottingham and Suresh Krishnan, exists to “standardize building blocks that allow for the expression of preferences about how content is collected and processed for Artificial Intelligence (AI) model development, deployment, and use.”
Three deliverables are named: a vocabulary for expressing the preferences, one or more documents describing how to attach them to content in IETF protocols and formats, and a method for reconciling multiple expressions that conflict.
That is a narrower remit than the public conversation about AI scraping usually assumes. The group is building a grammar, not a lock, and the distinction matters commercially: a vocabulary settles what a publisher can state unambiguously while leaving entirely open the question of who is obliged to listen.
The vocabulary draft, A Vocabulary For Expressing AI Usage Preferences, is on the standards track and defines three categories. train-ai covers use in training a model. ai-use covers use by an AI system at inference. search covers use for search indexing.
Each takes a single-byte value: y to allow, n to disallow. A complete expression looks like train-ai=y, search=n — permission to train, refusal to be indexed for search. Those two things have never been separable in robots.txt, which is the entire reason the work exists.
Why robots.txt was not enough
The Robots Exclusion Protocol, standardised as RFC 9309, expresses exactly one thing: whether a given user agent may fetch a given path. It has no vocabulary at all for what happens to the bytes afterwards, because when it was written there was only one thing anyone did with them.
That was adequate while the only consumer was a search engine whose interest in the page was to send readers back to it. It stopped being adequate when the same fetch could feed a training corpus, answer a question without a click, or both.
A publisher using robots.txt today faces a binary that does not match its commercial position. Blocking a crawler removes the content from whatever that crawler feeds, including search surfaces the publisher wants. Allowing it concedes training use the publisher may not want. We have covered what happens when publishers choose the blunt instrument in our report on blocking and search visibility.
The workaround the industry reached for was per-agent rules: name GPTBot, name Google-Extended, name CCBot, and hope the mapping between agent and purpose holds. Our own count of which crawlers publishers actually name found that most name none at all, and the ones that do name very few.
AIPREF's answer is to separate the permission to fetch from the permission to use. The fetch stays with robots.txt, where it has always been. The use becomes its own declaration, expressed in its own vocabulary, so a publisher no longer has to encode a commercial position in a list of user-agent names.
How a preference gets attached
The companion draft, Associating AI Usage Preferences with Content in HTTP, defines two attachment points and formally updates RFC 9309 to accommodate one of them. Updating an existing standard rather than replacing it is the significant move: robots.txt stays authoritative and gains a new directive.
The first is an HTTP response header. Content-Usage is a structured field dictionary that “indicates usage preferences regarding the content of the HTTP message”, carried with the representation rather than the resource.
The second is a robots.txt directive of the same name, which can carry an optional path pattern so preferences vary by directory rather than applying to a whole host. The draft's own worked example is compact and worth reading literally:
User-Agent: * · Allow: / · Content-Usage: train-ai=n · Content-Usage: /ai-ok/ train-ai=y. Everything may be crawled; nothing may be trained on; except the /ai-ok/ directory, which may.
One constraint governs the whole design: usage preferences “apply only to those resources that can be crawled according to Allow/Disallow rules”. A preference on a disallowed path is inert, because a compliant crawler was never going to fetch it.
Capabilities of robots.txt as standardised in RFC 9309 against the AIPREF drafts, September 2026
Source: Media Morpheus, September 2026. Classification from RFC 9309 and the draft-ietf-aipref-vocab and draft-ietf-aipref-attach texts.
The final row is the working group's own position: technical enforcement is named as out of scope in the charter, and the vocabulary draft states the specification does not ensure preferences are followed.
The sentence that defines the limits
Section 3.2 of the vocabulary draft is unusually direct about what the specification does not do. It does not “ensure that preferences are followed”, and it does not address “if, how, or when preferences should be followed”. An entity receiving a preference “has a choice whether to follow those preferences”.
The charter says the same thing structurally. Out of scope: “Technical enforcement of preferences; Application layer protocols for authenticating or authorizing clients and/or crawlers; Establishment of registries of preferences relating to content; Auditing and other transparency measures for AI training.”
Read those two together and the shape is clear. The IETF is standardising the sentence, not the consequence — the grammar of refusal rather than any machinery for making refusal bite. That is a deliberate scoping decision, stated twice in two separate documents.
That is not a criticism. It is what a protocol body can usefully do, and it matches how robots.txt has always worked: a convention observed by parties who choose to observe it, with no mechanism compelling anyone.
The vocabulary is also drafted to work “in jurisdictions where expressing preferences results in legal obligations, as well as where there are no associated legal obligations”. That is the hinge. A machine-readable, dated, unambiguous refusal is worth little as a technical control and potentially a great deal as evidence.
Where the work actually stands
Both charter milestones — the vocabulary specification and the attachment specification going to the IESG — were dated 31 August 2026. That date has passed, which makes the current state of each document the most useful thing to check before believing any claim about AIPREF support.
The vocabulary draft carries a September 2026 date and expires 18 March 2027. The attachment draft sits at revision 05, dated 19 August 2026, expiring 20 February 2027, in datatracker state “I-D Exists” within the working group. Neither has been submitted to the IESG.
That is normal for IETF timelines and worth stating plainly rather than dressing up: the milestones have passed and both documents remain working group drafts. Standards work routinely runs long, and a missed milestone is not evidence of trouble, but it does mean nothing here has cleared review.
The practical consequence for a publisher is that nothing here is implementable as a standard today. A Content-Usage line in robots.txt is currently an expression of intent that no crawler is obliged to parse, and most will not.
There is also a caching caveat the draft names directly: crawlers may cache robots.txt “for up to 24 hours”, so a change in preference is not effective immediately even among compliant readers.
What a publisher should do now
Nothing urgent, and that is the honest answer. Adding Content-Usage directives today costs nothing and achieves nothing technically, because no deployed crawler is required to read them and an unparsed line in robots.txt is inert rather than harmful.
What has value now is the record. A dated, public, machine-readable statement of what you permit is the same asset that made robots.txt useful in licensing negotiations long before anyone litigated it, and we have described that dynamic in our report on publisher–AI deals.
Track the attachment draft rather than the vocabulary. The vocabulary looks close to settled; the contested part in work of this kind is always how a preference attaches to content and how competing expressions reconcile, which is the third deliverable and the least finished.
Be sceptical of any vendor selling AIPREF compliance as a product before the drafts reach the IESG. There is nothing yet to comply with, and a file full of directives no crawler parses is not protection.
And keep the distinction straight in your own head: this standard governs what you can say, not what anyone must do. Those have never been the same thing on the web, and this working group has said so in writing.
Methodology and limitations
Quotations are taken from three primary documents retrieved on 26 September 2026: the AI Preferences working group charter on the IETF datatracker, the working group draft A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab), and Associating AI Usage Preferences with Content in HTTP (draft-ietf-aipref-attach). Revision numbers, dates and datatracker states are quoted as displayed on that date; Internet-Drafts change, and any figure here may be superseded by a later revision. We have not surveyed crawler operators on whether they intend to implement the vocabulary, and we make no claim about adoption. The comparison with RFC 9309 is our reading of the two specifications side by side, not a statement by the working group.
Conclusion
The IETF is doing the useful and limited thing a protocol body can do: giving publishers a precise, machine-readable way to say which uses of their content they permit. It is not building a lock, and it says so in the charter. For a publisher the practical position is that the vocabulary is nearly settled, the attachment mechanism is still a working group draft past its milestone, and nothing here is enforceable by the standard itself. The value of writing a preference down today is evidentiary rather than technical — the same reason a robots.txt directive has always been worth more in a negotiation than in a firewall.
Frequently Asked Questions
Will adding Content-Usage to robots.txt stop AI training?
No. The specification explicitly does not ensure preferences are followed, and the working group charter puts technical enforcement out of scope. A compliant crawler that reads the directive still chooses whether to honour it.
Does this replace robots.txt?
No, it extends it. The attachment draft updates RFC 9309 to add a Content-Usage directive, and preferences apply only to resources already crawlable under existing Allow and Disallow rules.
What is the difference between train-ai and ai-use?
train-ai covers use of the content in training or fine-tuning a model. ai-use covers use by an AI system at the point of answering. A publisher can allow one and refuse the other, which robots.txt cannot express.
Is this a published standard yet?
No. Both are working group Internet-Drafts. The attachment draft was at revision 05 dated 19 August 2026 in state “I-D Exists”, with neither document submitted to the IESG despite a charter milestone of 31 August 2026.
Should I implement it now?
There is no technical benefit yet, because no deployed crawler is obliged to parse it. There may be evidentiary benefit in publishing a dated, unambiguous statement of what you permit.
How quickly would a change take effect?
Not immediately. The draft notes crawlers may cache robots.txt for up to 24 hours, so preference changes are not visible to compliant crawlers until the cache expires.
Sources
- official IETF, AI Preferences (aipref) working group charter, retrieved 26 September 2026. Source of the goal statement, deliverables, milestones, chairs and the out-of-scope list.
- official IETF, A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab), standards track. Source of the three categories, the y/n values and the section 3.2 limits.
- official IETF, Associating AI Usage Preferences with Content in HTTP (draft-ietf-aipref-attach), revision 05, 19 August 2026. Source of the Content-Usage header and robots.txt directive, the worked example and the document state.