Do News Sites Block AI Crawlers? We Checked Fifteen.

What we did, and what we found

We fetched the robots.txt of fifteen major news domains on 19 September 2026, following redirects, and parsed each file properly — grouping consecutive User-agent lines so that each named crawler was assessed against the directive block actually governing it.

Fourteen were readable. Politico returned HTTP 403 to every request from our vantage point, so its file is unknown rather than empty, and it is excluded from all counts.

The headline result is that blocking is common but highly selective. Common Crawl's CCBot is blocked by 12 of the 14, and Anthropic's ClaudeBot by 11. OpenAI's GPTBot is blocked by 6, and Google-Extended by 6.

The agents blocked least often are the ones that produce citations. OAI-SearchBot and ChatGPT-User are each blocked by just 4 of 14, and several publishers name them specifically in order to permit them. That asymmetry is the finding, and it is consistent enough across independent publishers to look like a deliberate industry posture rather than a set of accidents.

Two things make robots.txt unusually good evidence for a question like this. It is published deliberately rather than leaked, so it states what a publisher means to permit; and it is machine-readable, so the same check can be run by anyone who wants to verify or repeat it.

Why block training but permit search?

Because the two do different things to a publisher's business, and the distinction is exactly the one OpenAI's own documentation draws.

A training crawler takes text that contributes to a model's parameters. Nothing comes back: no link, no attribution, no reader. As documented elsewhere in this series, content absorbed in training is not retained as a retrievable document with a source attached, so a publisher cannot even demonstrate their material was used.

A search crawler supports answers that carry citations and links. That is closer to the existing bargain with search engines — take the text, send back some readers — which publishers have lived with for two decades.

So the pattern in the data reads as a rational response to an asymmetric deal. Common Crawl, whose archives feed many training corpora, is blocked most often of all. The agents most likely to return a click are blocked least. Publishers are not rejecting AI; they are rejecting the version of it that takes without returning.

Who blocks everything?

Four of the fourteen apply blanket blocks across every agent we checked: the New York Times, CNN, NPR and — with carve-outs described below — Bloomberg.

These are publishers with either litigation, licensing negotiations, or a strong institutional position on the reuse of their archives, and a total block is the clearest available statement of it. The Associated Press blocks GPTBot, CCBot and ClaudeBot while not naming the search agents at all.

At the other end sit publishers who have chosen the opposite posture toward one company. The Atlantic explicitly allows GPTBot, ChatGPT-User and OAI-SearchBot — three separate groups each reading Allow: / — while blocking CCBot, ClaudeBot and Google-Extended. The Wall Street Journal permits OpenAI's agents through a large shared group whose restrictions cover only utility paths such as login and account pages.

Forbes is the sharpest single illustration. It blocks GPTBot outright and explicitly allows OAI-SearchBot. One company, two crawlers, opposite answers — which is only possible because OpenAI publishes them as separately controllable agents, and is a good argument for that design.

The postures also cluster by ownership and business model rather than by editorial politics. Publishers with paywalled subscription businesses and licensable archives block hardest, which is what you would expect if the calculation is about the value of the text rather than about AI as such.

The carve-outs are the most revealing detail

Two publishers block AI crawlers from the site and then admit them to specific areas, and what they choose to admit them to is worth dwelling on.

Bloomberg's rules for GPTBot read Disallow: / followed by Allow: directives for /professional, /company, /latam, /faq and /tc — its corporate, product and terms pages. The Washington Post blocks CCBot and ClaudeBot and then allows /creativegroup/ and /advertising/, which are its advertising and branded-content operations.

So the journalism is closed and the marketing is open. A model asking about either publisher may be reading their commercial self-description while barred from the reporting that gives the masthead its authority.

For anyone thinking about how companies are represented in AI answers, this is a small but genuine finding. Where the highest-quality sources withdraw and promotional material remains reachable, the mix of what these systems can retrieve shifts in a predictable direction — and that shift is being made one robots.txt at a time by publishers acting in their own commercial interest.

How does this compare with the wire services?

It is the exact opposite, and the contrast is the most useful thing in this report.

An earlier study in this series checked the robots.txt of five major press release wires. None named any AI crawler at all — not to block one and not to permit one — so every agent fell through to wildcard rules that restricted only administrative and widget paths. GlobeNewswire went further with an explicit Allow: /news-release/, and PR Newswire publishes an llms.txt directing large-scale crawlers to its full index.

Set the two studies together and the picture is stark. The publishers who produce independent journalism are progressively closing their archives to AI training. The channels that distribute paid corporate announcements are open, and in one case actively inviting.

Neither party is behaving irrationally. A newsroom's archive is an asset it can license; a wire's releases are material its customers paid to have distributed as widely as possible. But the aggregate effect is a slow change in what these systems can read about the world, in which promotional text is increasingly available and reported journalism increasingly is not.

There is a second-order effect worth naming. If reported journalism becomes progressively less readable to these systems while press releases stay open, then the material available to answer a question about a company tilts toward what that company said about itself — not through anyone's design, but as the sum of many separate commercial decisions.

What this means for anyone buying placement

Ask whether the outlet you are buying into permits AI crawlers, because it decides whether the placement can be surfaced that way at all. Checking takes seconds: fetch the domain's robots.txt and look for the agent you care about.

Be precise about which agent matters for your purpose. If the hope is being cited in an AI answer with a link, the relevant crawlers are the search-facing ones, which most of these publishers still permit. If the hope is a model absorbing your coverage in training, that is the pathway most heavily blocked and the one that produces no attribution even when it works.

Notice what the wire comparison implies about the value of scarcity. Coverage on a publisher that blocks training crawlers is less available to those systems, which is a genuine reduction in one kind of reach and says nothing about the coverage's worth to readers.

Media Morpheus sells placement and distribution. What we can tell you honestly is which doors are open, because it is publicly checkable. What nobody can tell you is what any system does after walking through one — a limit this series has now documented from several directions.

Re-check before you rely on it. These files changed while this series was being written, and a permission recorded today is not a commitment for next quarter.

Methodology and limitations

This is an original measurement. On 19 September 2026 we requested https://www.<domain>/robots.txt for fifteen major news domains, following redirects, with a desktop browser user agent, and classified six AI-related crawlers by parsing grouped user-agent blocks: consecutive User-agent lines share the directive block that follows them, and each named agent was assessed against the rules actually governing its group. Full results are recorded in our published data file. Six limits apply. First, Politico returned HTTP 403 to every attempt and is excluded; its rules are unknown, not absent. Second, an initial pass using a line-window heuristic misclassified CNN, the Wall Street Journal and Reuters, because grouped user-agent blocks place the governing directive beyond a fixed window; those results were discarded and the figures here come from the group parser. Third, robots.txt records stated permission at one moment from one vantage point, and sites may serve different files to different requesters. Fourth, it is evidence about policy only: it does not establish that any crawler obeyed it, visited, retained content or cited it. Fifth, fifteen domains is a purposive sample of large English-language outlets, not a representative sample of news publishing. Sixth, these files change frequently and findings should be re-checked rather than assumed.

Conclusion

Major news publishers are not blocking AI indiscriminately; they are drawing a line through the middle of it. Twelve of fourteen block Common Crawl and eleven block ClaudeBot, while OpenAI's search agents are blocked by only four and explicitly welcomed by several, including one publisher that bars OpenAI's training crawler and admits its search crawler on the same page. The distinction tracks whether anything comes back: training takes text and returns nothing attributable, while search returns citations and readers. The sharpest detail is the carve-outs, where two publishers close their journalism to these systems and leave their advertising and corporate pages open. Set against the wire services, which block nothing and one of which publishes a file inviting crawlers in, the direction is uncomfortable — reported journalism is becoming less readable to these systems, and paid corporate announcements are not.

Frequently Asked Questions

Sources

1. vendor research Media Morpheus, 19 September 2026. Original retrieval and parsing of robots.txt for fifteen major news domains. Method and limitations stated above; full results published in our data file.
2. official OpenAI. Crawler documentation for GPTBot, OAI-SearchBot and ChatGPT-User, for the separately controllable agent design.
3. official Google Search Central. Documentation on Google-Extended, for its scope over AI training and grounding.