How Much Traffic Do AI Assistants Actually Refer?

What is a crawl-to-refer ratio?

On 1 July 2025 David Belson and Sam Rhea of Cloudflare published a metric that made the exchange between AI companies and websites legible for the first time. They called it the crawl-to-refer ratio, and Cloudflare now shows it on the AI Insights page of its Radar service.

The calculation is stated plainly in the post. Cloudflare divides “the total number of requests from relevant user agents associated with a given search or AI platform where the response was of Content-type: text/html” by “the total number of requests for HTML content where the Referer header contained a hostname associated with a given search or AI platform”. The result is normalised to a single referral. HTML pages are used because they are the content worth crawling.

Read plainly: a ratio of 1,000:1 means a platform asked a site for a thousand pages for every one visitor it sent back. A ratio below 1:1 means it sent more visitors than it took pages, which is what a referral engine looks like when it is working the old way.

Two aggregation choices matter. Requests for training and requests made on behalf of a user in a live session are counted together under one platform name, even when the company splits them across separate user agents — Cloudflare's own illustration uses AIBot and AIBot-User. And referral traffic originating from Google's network is excluded from the analysis because Google's prefetching means such requests do not represent a person reading anything.

Cloudflare is not a disinterested party here. It sells AI crawl controls, blocks AI training crawlers by default on new domains, and runs a pay-per-crawl scheme built on the HTTP 402 status code that we examined in can you charge an AI crawler. The metric is a good one. It also happens to be the number that makes Cloudflare's product line look necessary, and it should be read with that in mind.

What are the actual ratios?

The fullest published table comes from Cloudflare's João Tomé on 29 August 2025, comparing January 2025 with July 2025.

The spread is the finding. Anthropic's crawlers requested roughly 38,000 pages for every visitor Claude sent back. Google requested five. Between a traditional search engine and a frontier lab there are four orders of magnitude, and the difference is not a matter of degree but of business model: one indexes in order to send you somewhere, the other reads in order to answer in place.

The earlier snapshot in Cloudflare's launch post covers the single week of 19 to 26 June 2025 and puts the range from Anthropic's 70,900:1 down to Mistral's 0.1:1 — Mistral sending ten referrals for every page requested. Comparing that week against the July monthly figure is not like for like, which is exactly the trap these numbers invite.

One ratio deserves separating out. Microsoft sits at 41:1, an order of magnitude worse than Google and three orders better than Anthropic. Microsoft's crawler is Bingbot, which serves both the Bing index and Copilot — the same mixed-purpose arrangement that makes a publisher's blocking decision so awkward, and one of the reasons the robots.txt declarations we catalogued in which AI crawlers publishers name are so inconsistent.

Why are the ratios moving?

Between January and July 2025 Anthropic's ratio fell from 286,930:1 to 38,066:1, a drop of 86.7%. That looks like reform. It is mostly arithmetic.

A ratio falls if the numerator falls or the denominator rises, and Cloudflare's accompanying data points at the numerator. Over the same period OpenAI's ratio moved from 1,217:1 to 1,091:1, a fall of 10.4%; Perplexity's went the other way, from 55:1 to 195:1, up 256.7%; Google's rose from 4:1 to 5:1; ByteDance's collapsed from 18:1 to 1:1 as Bytespider's share of AI crawling fell from 14.1% to 2.4% year on year. Meanwhile GPTBot's share of AI crawler traffic rose from 4.7% to 11.7%, ClaudeBot's from 6.0% to 9.9%, and Meta-ExternalAgent's from 0.9% to 7.5%.

Nothing in that pattern describes assistants sending more people to websites. It describes crawlers changing how often they visit. A publisher watching Anthropic's ratio improve by 87% did not receive 87% more traffic.

The purpose data makes the same point. Averaged over the twelve months to July 2025, Cloudflare classified about 80% of AI crawler requests as training, 18% as search and 2% as user actions. Comparing July 2024 with July 2025, training rose from 72% to 79% and search fell from 26% to 17%. Search-purpose crawling — the kind that can produce a citation with a link — is the shrinking share.

Cloudflare's anniversary report of 1 July 2026, by Arielle Weiss, Zach Albertson and Emily Lanfear, reports that “52% of crawler requests are now for AI training as of June 2026, up from 22% in Spring 2025”, with mixed-use crawlers making up over 36% of activity. Those figures are not comparable with the 2025 series — a 22% training share in spring 2025 cannot be reconciled with a 79% training share in July 2025 unless the denominator and the classification both changed, and the 2026 report does not explain how. It also draws on Cloudflare's Investor Day 2026 presentation as well as Radar. Treat the 2026 percentages as a separate series, not a continuation.

The same report makes claims worth repeating with their provenance attached: that more than 50% of internet traffic is now non-human, that Google accounts for approximately 88% of referral traffic, and that some of the most heavily crawled categories have seen human traffic fall by as much as 40% in under a year. None of those is accompanied by a published dataset.

How much referral traffic actually arrives?

The ratio tells you about the exchange. The absolute numbers tell you whether the returning side is worth building a business on.

Similarweb data reported by Digiday's Sara Guaglione on 10 July 2025: AI referral traffic to news and media sites from AI platforms other than Google grew from 35.3 million global visits in May 2025 to 35.9 million in June, and from 5.6 million to 6.1 million in the United States. The bulk of the June global figure went to Yahoo (2.3 million), Reuters (1.8 million), the Guardian (1.7 million) and Business Insider (1 million).

Set that against the losing side of the same ledger. Search referral traffic to a set of 1,000 web domains fell from 12 billion global visits in June 2024 to 11.2 billion in June 2025, a decline of about 6.7% — roughly 800 million visits a year. The replacement channel was delivering about 36 million visits a month to news and media sites. On Similarweb's own figures the arriving AI traffic is a rounding error against the departing search traffic, which is the point Digiday was making.

Concentration makes it worse for anyone who is not already large. ChatGPT sent 396.8 million referral visits to the 1,000 sites in Similarweb's dataset in June 2025, 81.7% of all AI traffic to those sites. For individual publishers the dependence is close to total: ChatGPT accounted for 99.4% of the Guardian's AI referrals, 95.9% of Business Insider's, 95.4% of Reuters' and 78.5% of Yahoo's. A channel in which one vendor supplies four-fifths to nineteen-twentieths of the volume is not a diversified channel.

The longest-running independent measurement agrees on scale. Rand Fishkin's analysis of Similarweb clickstream data for January to April 2026, published on 9 June 2026, found that less than 1% of all traffic flows from AI tools to external sites, and that only 0.34% of searches reached Google's AI Mode. His earlier work with Datos, covering a panel of several hundred thousand US users on desktop and mobile browsers, had Google at 63.41% of all referral traffic in January 2024 with OpenAI “the smallest of the search/answer engines” despite growing 82% over the study period. Neither panel captures app traffic, which is a real gap and one that cuts in AI's favour.

What that means for a company rather than a publisher: if you are planning around AI referrals as a growth channel, you are planning around something that currently measures below 1% of web traffic and is supplied overwhelmingly by one vendor. The more durable questions are what happens to the coverage itself over time, which we looked at in what happens to placement traffic.

What can the ratio not tell you?

Four limits, three of which Cloudflare states itself.

Native apps are invisible. Cloudflare writes that “traffic referred by Claude's native app does not include a Referer: header, and we believe that the same holds true for traffic generated from other native apps as well”, and concludes that “these calculations may overstate the respective ratios, but it is unclear by how much”. That caveat is load-bearing. A large and growing share of assistant use is in apps. Anthropic's 38,066:1 is an upper bound, not a measurement, and nobody has published the correction.

The ratio counts pages, not value. A crawler requesting ten thousand archive pages and a reader arriving on one high-intent product page are not commensurable quantities, and dividing one by the other produces a number with no units. It is a useful shape. It is not a price.

The series is unstable. Cloudflare says outright that the ratios “will change over time” because of shifting crawling patterns, and its own week-on-week figures move by 19.4% for Google and over 6% for DuckDuckGo and Yandex. Anyone quoting a single ratio without its date is quoting noise.

It is one network's view. Cloudflare sits in front of a large share of the web but not all of it, and its classification of which user agent belongs to which platform depends on operator disclosures and self-identification. Crawlers that do not identify themselves — and, as the Tow Center found when it looked at how often AI search tools cite the wrong source, several do not publish their crawler names at all — are not in these tables at whatever their true volume is. The same identification problem underlies what Common Crawl is and why blocking it does not settle the question.

What should a founder do with this?

Three things follow.

Do not buy AI visibility as a traffic channel. The defensible case for being present in AI answers is that a person reads your name and your claim in the answer. The referral is not the product, because on every dataset here the referral barely exists. A proposal that quantifies AI visibility in expected sessions is quoting from a channel that measures under 1% of web traffic.

Read blocking decisions as economics, not ethics. The ratio is the argument publishers are actually making when they block training crawlers while admitting search agents — the split we found on the ground in our check of fifteen news sites. A crawler at 5:1 is buying something. A crawler at 38,066:1 is not, and the publisher's robots.txt is the only lever available until something is bought.

Measure the mention, not the click. If the exchange now returns attention rather than visits, the metrics that matter are whether an assistant names you, names you accurately and names you when asked the question your buyers ask. Those are harder to count than sessions, and they are the only ones the data supports.

Media Morpheus sells media and podcast placement, including on the argument that coverage improves AI visibility. The crawl-to-refer data bounds that argument on the traffic side specifically. Coverage that AI systems read is coverage that, on Cloudflare's measurement, returns between one visit in five and one visit in thirty-eight thousand depending on which system read it. Whatever the case for placement is, it is not a traffic case.

Methodology and limitations

Ratio figures are from two Cloudflare publications. The metric and its definition come from “The crawl before the fall… of referrals” by David Belson and Sam Rhea, 1 July 2025, which reports a range from Anthropic's 70,900:1 to Mistral's 0.1:1 for the week of 19–26 June 2025. The per-platform table for January and July 2025, the crawl-purpose splits and the bot share changes come from “The crawl-to-click gap” by João Tomé, 29 August 2025. Both rely on requests seen by Cloudflare's network, on user agents attributed to platforms using operator disclosures and industry sources, and on the Referer header, which native apps do not send — Cloudflare states the resulting ratios may overstate the imbalance by an unknown amount. Referrals from Google's network are excluded because of speculative prefetching. Cloudflare sells AI crawl control and monetisation products, so it has a commercial interest in the imbalance being large. The 2026 figures — 52% of crawler requests for training as of June 2026 against 22% in spring 2025, over 36% mixed-use, more than 50% of internet traffic non-human, approximately 88% of referral traffic through Google — are from “Content Independence Day, one year on” by Arielle Weiss, Zach Albertson and Emily Lanfear, 1 July 2026, whose stated sources are Cloudflare Radar and Cloudflare's Investor Day 2026 presentation; those percentages do not reconcile with the 2025 purpose series and are treated here as a separate measurement rather than a continuation. Referral volumes are Similarweb data as reported by Digiday on 10 July 2025, covering news and media sites and a set of 1,000 web domains; Similarweb sells traffic analytics and its panel-based estimates are not server-side counts. The clickstream figures are from Rand Fishkin, SparkToro, 9 June 2026 and 11 March 2024, using Similarweb and Datos panels of US desktop and mobile browser users; neither panel captures traffic inside mobile apps, which understates AI referral volume by an unmeasured amount. The comparison between monthly AI referral visits and monthly search referral visits in this report is our own arithmetic on the Similarweb figures Digiday published, and the two series cover different site sets — news and media sites in one case, 1,000 web domains in the other — so it indicates order of magnitude only.

Conclusion

Almost none, measured against what they take. Cloudflare's July 2025 figures have Anthropic requesting 38,066 pages for every referral, OpenAI 1,091 and Google five; the week before, Anthropic's ratio was nearly 71,000:1. On the receiving end, AI platforms other than Google sent about 35.9 million visits to news and media sites worldwide in June 2025, while search referrals to a thousand domains fell by roughly 800 million visits over the year, and 81.7% of what did arrive came from a single vendor. Both sides of that ledger come with caveats that cut in opposite directions: native app referrals are uncounted, and Cloudflare sells the tools that the imbalance justifies buying. What survives the caveats is the shape. The exchange that built the open web — access in return for visitors — is not the exchange on offer, and any visibility work priced as if it were is priced against a channel that does not yet exist.

Frequently Asked Questions

Sources

1. network operator Belson, D. and Rhea, S., 2025. The crawl before the fall… of referrals: understanding AI’s impact on content providers. Cloudflare, 1 July 2025. Defines the metric; ratios for 19–26 June 2025.
2. network operator Tomé, J., 2025. The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals. Cloudflare, 29 August 2025. Per-platform ratios for January and July 2025 and crawl-purpose splits.
3. network operator Weiss, A., Albertson, Z. and Lanfear, E., 2026. Content Independence Day, one year on: building the business model for the agentic Internet. Cloudflare, 1 July 2026. Sourced to Cloudflare Radar and Cloudflare’s Investor Day 2026 presentation.
4. trade press Guaglione, S., 2025. AI platforms are driving more traffic — but not enough to offset ‘zero-click’ search. Digiday, 10 July 2025, reporting Similarweb data.
5. vendor Fishkin, R., 2026. In 2026, Less than One Third of Google Searches Still Send a Click. SparkToro, 9 June 2026, using Similarweb clickstream data.
6. vendor Fishkin, R., 2024. Who Sends Traffic on the Web and How Much?. SparkToro and Datos, 11 March 2024, updated April 2026.