What we measured
In an earlier report we tested six named AI agents across fifteen news domains. This one asks a different question: not whether a given crawler is blocked, but how many distinct AI crawlers publishers name at all. The answer describes the landscape publishers believe they are managing.
On 24 September 2026 we requested the robots.txt of 25 domains covering national news, business press, trade press, the four major wire services, and three publishing platforms. Twenty-four returned a readable file. NPR did not respond on this attempt, and is excluded rather than counted as naming nothing.
We then extracted every user-agent declared in each file and classified them against a curated list. This matters for honesty: a naive pattern match sweeps in AhrefsBot, SemrushBot and Scrapy, which are SEO and scraping tools rather than AI systems. An earlier pass of ours returned 47 agents by that looser definition; the figure reported here is 35 after excluding them.
Agents that no publisher in our sample named do not appear in the count, so 35 is a floor for how many exist, not a census of the field.
The range between publishers is extraordinary
If publishers were responding to a shared, well-understood threat, their files would look broadly alike. They do not. CNN names 29 AI agents, The New York Times 28, The Atlantic 27, Wired 23 and Bloomberg 22.
At the other end sit publications of comparable stature doing almost nothing. Reuters names 3 AI agents in a file declaring 88 user-agent groups in total. The Wall Street Journal names 4 out of 62. Fortune names 4 out of 8.
Reuters is the sharpest illustration that this is not inattention. A file with 88 declared user-agents is a carefully maintained document; its author simply has not treated AI crawlers as the category to enumerate.
Some of the shortest files are the most AI-focused. TechCrunch declares 14 user-agents of which 13 are AI agents, and the Associated Press declares 14 of which 12 are. Those files exist almost entirely to address AI.
It also helps to see what else is in these files, because AI agents are usually a minority of them. Reuters declares 88 user-agent groups of which 83 are neither AI crawlers nor SEO tools; Bloomberg has 68 such entries and Adweek 96. Those remainders are ad verification bots, regional search engines, archivers and link checkers accumulated over many years. A robots.txt is a sediment of every crawler a publisher has ever had a view about, and the AI layer is just the most recent deposit.
AI-associated user-agents named, 24 September 2026 (SEO and marketing crawlers excluded)
Source: Media Morpheus, September 2026. Original measurement of robots.txt across 25 domains; 24 readable, NPR excluded as not retrievable.
Naming an agent is not the same as blocking it. Some files name agents in order to permit them. robots.txt records stated permission, not crawler behaviour.
Which agents actually get named
The ranking does not match public attention. The most widely named agent is CCBot, the Common Crawl scraper, at 15 of 24 sites. Common Crawl is a non-profit that has been archiving the web since long before generative AI, and it is named more often than any product from an AI company.
Next come three agents that rarely feature in public debate: Bytespider, ClaudeBot and Applebot-Extended, each named at 14 sites, followed by Amazonbot at 13, anthropic-ai and Diffbot at 12, and FacebookBot at 11.
GPTBot, the agent most discussed by a wide margin, is named at 11 sites — behind CCBot and level with FacebookBot. Google-Extended and PerplexityBot are each named at 10.
The tail is where the landscape is visibly still forming. Agents such as FirecrawlAgent and MistralAI-User appear at three sites each, google-cloudvertexbot at five, and Operator and BedrockBot at one apiece. Publishers are adding names as they learn them, which means these files are a rolling record of what each publisher happened to hear about.
Two details in the tail are worth pulling out. Google appears twice under different names: Google-Extended, which governs AI training and grounding, is named at 10 sites, while google-cloudvertexbot, associated with enterprise grounding, is named at 5 — so a publisher who blocked one has not necessarily addressed the other. And the ratio of AI to non-AI agents varies as much as the raw count: Forbes devotes 18 of its 25 declared user-agents to AI crawlers and the Associated Press 12 of 14, meaning those files exist largely for this purpose, while Reuters and The Wall Street Journal maintain large files in which AI is a rounding error.
The wire services name nobody
The starkest result in the dataset is the absence. PR Newswire, GlobeNewswire, AccessWire and EIN Presswire name zero AI crawlers between them. Three of the four declare exactly one user-agent group in total, applying to every crawler alike.
This is consistent with our earlier finding that the wires permit AI crawlers by default, and it makes commercial sense. A wire service is paid to distribute a client's announcement as widely as possible. Restricting who may read it would be restricting the product.
The consequence for anyone buying distribution is worth stating plainly. The material most likely to be machine-readable without restriction is the material a company paid to publish about itself — and that is precisely the category Wikipedia rates generally unreliable and that carries no independent editorial judgement.
Reddit and Substack also name no AI crawlers in their robots.txt, which is a useful corrective to the assumption that every large content platform has locked down. Whatever controls those platforms apply, they are not expressed here.
Two publishing platforms in the sample diverge completely, which is worth noting because they are often discussed interchangeably. Medium declares nine user-agent groups of which seven are AI agents, making its file overwhelmingly about machine readers. Substack declares four and names no AI agent at all. Same business category, opposite postures, and nothing in either company's public positioning would let a buyer predict which was which.
What naming does and does not tell you
Naming an agent is not the same as blocking it. Several files in our sample name agents in order to permit them explicitly, and our count here measures enumeration rather than exclusion. A publisher who names 29 agents has thought about 29 agents; it does not follow that 29 are barred.
Nor does robots.txt establish behaviour. It is a request. It records what a publisher has asked crawlers to do, not what any crawler did, and verifying compliance requires server logs that no outside party can see.
The files are also volatile. NPR's robots.txt was readable when we measured it five days earlier and did not respond on this attempt. Any figure here is a snapshot, and a publisher can rewrite the file in an afternoon.
Finally, absence of a name means a crawler falls through to whatever the wildcard group permits. A file that names no AI agents is not neutral; it is usually permissive, because most wildcard rules restrict administrative paths rather than content.
What this means for anyone buying coverage
If AI visibility is part of why you are buying a placement, the destination's robots.txt is a free, thirty-second check that tells you more than any media kit. The variation we found between comparable publications is far too large to assume.
Do not read a long list of named agents as sophistication, or a short one as negligence. Reuters maintains an elaborate file and names three AI agents; TechCrunch maintains a short one that is almost entirely AI agents. These are different editorial judgements about what matters, not different levels of competence.
Expect this to keep moving. A third of the agents we found are named by five or fewer publishers, which is what an unsettled landscape looks like. A check performed six months ago is not evidence about today.
Media Morpheus sells media and podcast placement, including wire distribution. The finding that wires restrict nobody is commercially convenient for us and worth being honest about: unrestricted machine access to a self-published announcement is not the same thing as credibility, and the two should not be sold as though they were.
Methodology and limitations
On 24 September 2026 we requested https://www.
Conclusion
Publishers do not agree on what they are defending against. Across 24 readable files we found 35 distinct AI crawlers named, with CNN enumerating 29 of them and Reuters 3, despite Reuters maintaining a far larger file overall. The most widely named agent is not a chatbot at all but CCBot, the Common Crawl scraper, ahead of GPTBot. Meanwhile every wire service in the sample names nobody, leaving paid announcements the most freely machine-readable material in the chain. For a buyer, the useful conclusions are that the destination matters far more than the category, that the check is free, and that a third of these agents are named by five publishers or fewer, so the picture will look different again next year.
Frequently Asked Questions
How many AI crawlers are there?
We found 35 distinct AI-associated crawlers named across 24 publisher robots.txt files on 24 September 2026, after excluding SEO tools. That is a floor rather than a total: agents no publisher in our sample named would not appear.
Which AI crawler is named most often?
CCBot, the Common Crawl scraper, at 15 of 24 sites — ahead of GPTBot at 11. Bytespider, ClaudeBot and Applebot-Extended are each named at 14, and Amazonbot at 13.
Do wire services restrict AI crawlers?
None of the four we checked names a single AI crawler, and three declare exactly one user-agent group in total. That is consistent with their commercial purpose: a wire is paid to distribute an announcement as widely as possible.
Does naming a crawler mean blocking it?
No. Some files name agents specifically to permit them. This report counts how many AI agents each publisher enumerates, which measures what they have considered, not what they have barred.
Can I rely on a robots.txt check from a few months ago?
Not really. These files change quickly — NPR's was readable when we measured five days earlier and unavailable on this attempt — and a third of the agents we found are named by five or fewer publishers, indicating the landscape is still forming.
Sources
- vendor research Media Morpheus, September 2026. Original measurement of robots.txt across 25 domains on 24 September 2026, extracting all declared user-agents and classifying AI-associated agents against a curated list with SEO and marketing crawlers excluded. Method and limitations stated above.
- official Google Search Central. Documentation of Google-Extended and the preview controls available to publishers.
- official Wikipedia. Reliable sources/Perennial sources, for the rating of wire service content referenced above.