Is anyone actually measuring this?
Start with the finding a founder will not like: no. There is no published measurement of how much agentic traffic arrives at company websites, how many forms it submits, or how many demo requests it generates. Every number in circulation is either a share of something else or a share published by a company selling bot management.
The closest thing to a series is Cloudflare’s classification of AI crawler requests by purpose. In “The crawl-to-click gap”, published 29 August 2025, João Tomé reported that over the preceding twelve months Cloudflare classified AI crawler requests as 80% training, 18% search and 2% user actions, and over the preceding six months as 82%, 15% and 3%.
Share of AI crawler requests on Cloudflare’s network, twelve months to July 2025
Source: Cloudflare, “The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals”, 29 August 2025.
For the six months to July 2025 the same post gives 82% training, 15% search and 3% user actions. The denominator is requests from user agents Cloudflare attributes to AI platforms — not all traffic, and not all agents. Cloudflare does not define the “user actions” category in the post. Cloudflare sells AI crawl control, bot management and a pay-per-crawl scheme, so it has a commercial interest in these volumes being large and legible.
Note what that 2% is not. It is not a count of agents acting on pages: the category is undefined in the post and almost certainly includes a plain retrieval fetch by ChatGPT-User. And it is not independent — Cloudflare publishes the numbers and sells the controls, which we noted when examining its pay-per-crawl toll booth.
The broader figures are worse on the same axis. Cloudflare’s anniversary report of 1 July 2026 states that more than 50% of internet traffic is now non-human and that mixed-use crawlers make up over 36% of activity, without a published dataset behind either. Imperva’s 2025 Bad Bot Report puts automated traffic at 51% of all web traffic and bad bots at 37%, drawn from Imperva’s own network; Imperva sells Advanced Bot Protection. Both numbers describe automation in general. Neither isolates an agent that clicks a button on a company’s site because a named person asked it to.
What a company can measure for itself is narrower and more useful: grep the server logs for the user agents listed below. That gives you a floor for the retrieval fetches from each operator, and nothing at all about agents running inside someone else’s browser.
Three arrivals that get called the same thing
Most coverage of “AI bots” collapses three different events into one word. The distinction is not academic: it decides which rule you write, and getting it wrong costs either money or visibility.
1. A training crawl
A batch fetch, made to build or refine a training corpus, with nobody waiting for the result. OpenAI’s GPTBot and Anthropic’s ClaudeBot are the named examples; Google’s Google-Extended is a robots.txt token rather than a crawler. This is the arrival the licensing fights are about and the one the emerging opt-out vocabulary was designed for, which we traced in can you tell an AI not to train on your site. It will never fill in a form.
2. A retrieval fetch at answer time
A person has asked a question and the assistant is fetching a page to answer it. OpenAI’s ChatGPT-User and Anthropic’s Claude-User do this live; OAI-SearchBot and Claude-SearchBot build the index that decides which page gets fetched. A human is waiting, and a citation with a link may result — though on the published evidence that link is worth very little, which is the subject of how much traffic AI assistants actually refer. This arrival reads. It does not act.
3. An agent acting on the page
The third arrival navigates. It clicks, it fills fields, it submits, it starts a trial, it books a slot. Google documents it plainly: Google-Agent is “used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request”. Cloudflare’s bot taxonomy carries two behaviours for it — Agent, “user-directed agents visiting a page on behalf of a human”, and Transact, “checkout or other transaction actions on behalf of users”.
This is a session, not a request. It may involve dozens of page views, JavaScript execution, cookie storage and a POST to your form handler. It is the arrival that shows up in your CRM, and it is the one nobody has counted. The robots.txt declarations we catalogued in which AI crawlers publishers name are almost entirely written against the first category, which is a reasonable description of where the industry’s attention has been.
How agent traffic identifies itself
There are three mechanisms, in descending order of reliability, and a fourth case where there is no mechanism at all.
OpenAI documents four agents with full user-agent strings and a published IP file each. GPTBot/1.4 for training, OAI-SearchBot/1.4 for search, OAI-AdsBot/1.0 to validate ad landing pages, and ChatGPT-User/1.0 for “certain user actions in ChatGPT and Custom GPTs”, including GPT Actions. The ChatGPT-User string is given in full as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot, with addresses at openai.com/chatgpt-user.json. Two details worth noting: OpenAI says ChatGPT-User “is not used for crawling the web in an automatic fashion”, and it publishes no separate user agent for ChatGPT’s agent mode.
Anthropic documents three robots, in an article last dated 7 April 2026: ClaudeBot for training, Claude-User for user-initiated access, and Claude-SearchBot for search quality. It publishes no example user-agent strings, only names, and a single IP file at claude.com/crawling/bots.json — which when we fetched it carried a creationTime of 18 August 2026. Anthropic also documents no separate agent for Claude for Chrome, which is the fourth case below.
Google documents nine user-triggered fetchers, a list it says “is not exhaustive”, on a page last updated 19 August 2026. The one that matters here is Google-Agent, whose desktop string is Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36, with addresses in user-triggered-agents.json and a reverse DNS mask of google-proxy-***-***-***-***.google.com for user-owned fetchers. Google adds a note that it “is also experimenting with the Web Bot Auth protocol, using the https://agent.bot.goog identity”.
Google also prints the warning that governs all of this: “The user agent string can be spoofed.” A string is a claim, not evidence, which is the whole subject of how you verify a crawler is what it claims to be. Reverse DNS and a published IP list turn the claim into something checkable; a signature turns it into something cryptographic.
The fourth case has no mechanism. Anthropic’s Claude for Chrome, announced 25 August 2025, is a browser extension. Anthropic’s description of it is explicit about what it does: giving Claude “the ability to see what you’re looking at, click buttons, and fill forms”. It runs in the user’s own Chrome, with the user’s own cookies and logins. There is no distinguishing user agent, no separate IP range and no signature, because at the network level it is the user’s browser. Cloudflare’s own category description names “browser-use agents” as a thing site owners may wish to block, but an agent driving a real browser session from a residential address is not a thing any of these three mechanisms can find. This is the largest single gap in the measurement picture, and it is structural rather than temporary.
Does robots.txt apply when a person asked?
Here the three operators say three different things, in writing, on current pages.
Anthropic applies robots.txt to all three of its robots without carve-out: “Anthropic’s Bots respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt.” It also states the consequence of using it: disabling Claude-User “prevents our system from retrieving your content in response to a user query, which may reduce your site’s visibility for user-directed web search”. It supports the non-standard Crawl-delay extension, and says it will not attempt to bypass CAPTCHAs.
OpenAI carves out the user-directed case: “Because these actions are initiated by a user, robots.txt rules may not apply” to ChatGPT-User. The word is may, which is not a commitment in either direction.
Google carves it out flatly for the whole class: “Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.” That covers Google-Agent.
So a single Disallow: / line has three meanings depending on who reads it, and for two of the three operators it does not govern the arrival this guide is about. If you want to stop an acting agent, robots.txt is not the instrument; a WAF rule is. And the corollary runs the other way too, which is the trap set out in does blocking a crawler remove you from search: the token that reliably obeys you is usually the one whose obedience costs you a citation.
Web Bot Auth, and its actual standing
The proposed fix is to have agents sign their requests. Cloudflare introduced Web Bot Auth on 15 May 2025, in a post by Thibault Meunier that states the problem precisely: “user agent headers alone are easily spoofed and are therefore insufficient for reliable identification”, and IP ranges fail because “connections from the crawling service might be shared by multiple users” and change over time.
Mechanically, per Cloudflare’s reference documentation, the agent operator generates an Ed25519 key, publishes the public key as a JSON Web Key Set at the well-known path /.well-known/http-message-signatures-directory, and attaches Signature and Signature-Input headers to each request. A Signature-Agent header carries key discovery. The origin, or a CDN in front of it, validates the signature.
The standards position deserves stating plainly rather than being implied. The two documents this rests on are individual Internet-Drafts by Thibault Meunier and Sandor Major, not working group products. draft-meunier-web-bot-auth-architecture reached version 05 on 2 March 2026 and was replaced by draft-meunier-webbotauth-httpsig-protocol, which reached version 02 on 19 August 2026; the IETF datatracker lists both as no longer active. There is no working group, and no IETF endorsement. Deployment is running ahead of standardisation, which is the same pattern we found in the opt-out vocabulary drafts.
Deployment is nonetheless real. Cloudflare announced signed agents on 28 August 2025, in a post by Jin-Hee Lee that framed the shift in one sentence: “Bots are no longer directed only by the bot owners, but also by individual end users to act on their behalf.” A signed agent, in Cloudflare’s definition, is directed by an end user rather than a single company, signs its requests via Web Bot Auth, and complies with Cloudflare’s signed agent policy. The first cohort named was ChatGPT agent, Goose from Block, Browserbase, Anchor Browser and Cloudflare’s own Browser Rendering.
As of 1 July 2026 Cloudflare folded signed agents into its Verified bots programme and replaced the distinction with a metadata field: Direct for a bot “operated by a single, narrow operator”, Intermediary for “an agentic service that a wide range of end users can operate”. The documentation names the residual problem honestly. Because an intermediary acts for many users, “this introduces transitive trust: you may trust the intermediary operator, but not necessarily every end user driving it”. Cloudflare says it is experimenting with forwarding information about the end user using the Forwarded header of RFC 7239.
Two limits for a company reading this. Signing only identifies agents that choose to sign, so verification is a way of recognising the well-behaved, never of finding the rest. And the working register of who is signed is today a directory maintained by one CDN, which means the practical answer to “is this agent legitimate” currently depends on a vendor’s allowlist rather than on a standard.
What it does to your analytics and your leads
Two failure modes, running in opposite directions, and one of them is invisible by design.
Google Analytics 4 states that “traffic from known bots and spiders is automatically excluded”, identified using “a combination of Google research and the International Spiders and Bots List, maintained by the Interactive Advertising Bureau”. It then adds the sentence that matters: “At this time, you cannot disable known bot traffic exclusion or see how much known bot traffic was excluded.”
So if an agent is on the list, its visits leave your reports with no counter behind them — you cannot size the thing you are trying to size. And if an agent is not on the list, and it is driving a real browser that executes your tag, it is counted as a human session: a page view, a scroll, an engaged session, and a conversion if it submits your form. Neither error has been publicly measured. Anyone telling you what share of your form fills are agentic is guessing, and so is anyone telling you the share is zero. The equivalent reporting gap on the search side — impressions counted, clicks not broken out — is covered in what Search Console shows for AI answers.
On lead quality there is nothing published at all: no dated study of agent-submitted forms, no measured false-positive rate on demo requests, no benchmark for what an agentically-sourced lead converts at. What can be said is that the infrastructure layer now assumes it. Cloudflare tracks Transact as a bot behaviour covering “checkout or other transaction actions on behalf of users”, which is not a category anyone builds speculatively.
The platforms are also busy routing around the form rather than teaching agents to fill it. Google announced the Agent Payments Protocol on 16 September 2025 with more than 60 organisations, built on “Mandates” — “tamper-proof, cryptographically-signed digital contracts that serve as verifiable proof of a user’s instructions” — to answer three questions it names as authorisation, authenticity and accountability. OpenAI’s Agentic Checkout Spec, carrying an API version of 2025-09-12, asks merchants to implement five REST endpoints so that checkout happens inside ChatGPT against the merchant’s own commerce stack. If that is where this lands, a web form is a transitional surface and a signed structured request is the durable one.
There is also a security dimension, and here the numbers come from the operator rather than from a vendor selling a defence. Anthropic published its own red-teaming of Claude for Chrome alongside the launch.
Share of adversarial test cases that succeeded, autonomous mode
Source: Anthropic, “Claude for Chrome”, 25 August 2025 (updated 18 December 2025). Full set is 123 test cases representing 29 attack scenarios; the browser-specific set is a challenge set of four browser-specific attack types.
Bars are linear against the 35.7% maximum; the 0% bar renders at the 2px minimum. All evaluations were run in Anthropic’s experimental autonomous mode. These are the operator’s own figures on its own product, published against its commercial interest rather than for it, which is a better provenance than most numbers in this field — but they measure attacks on the agent, not agent behaviour on an ordinary site.
The relevance to a company website is direct. Text on your page — including text a visitor put there, in a review field or a comment — is input to a model that may then act. Anthropic’s stated defences are site-level permissions, action confirmations before “high-risk actions like publishing, purchasing, or sharing personal data”, and blocking whole categories such as financial services. Those are the agent operator’s controls, not yours.
Allow, verify, rate-limit or block
Four options. Each carries a cost, and one of them may already have been chosen for you.
Allow. The default and the cheapest. You accept whatever arrives, your analytics are wrong by an unmeasured amount in an unknown direction, and your sales team occasionally works a lead that was never a person. In exchange you are reachable by every retrieval fetch and every agent, which is the only position from which AI visibility is possible at all.
Verify. Validate Web Bot Auth signatures, or check published IP lists and reverse DNS against the user agent, and treat verified agent sessions differently — tag them in analytics, route them out of the sales queue, serve them a clean page. This is the option that buys information rather than protection. Its limits are the ones above: it only sees agents that identify themselves, and the registry is currently a vendor’s.
Rate-limit. Effective against crawlers and close to useless against agents. Anthropic honours Crawl-delay; a retrieval fetch or an agent session is a handful of requests from one address on behalf of one person, and no sane rate limit catches it. Rate limiting is a tool for the first arrival in our three, not the third.
Block. Cloudflare offers three presets — Search, Agent and Training — with Agent defined as “automated activity acting in real time on a person’s behalf, such as chat fetch bots and browser-use agents”, and three mitigations each: block everywhere, block on pages with ads, or allow. The documentation then states a default change worth knowing about: on 15 September 2026 Cloudflare will set updated defaults for new domains under which “bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed”, with mixed-purpose Search-and-Training crawlers blocked by every training-block configuration. If your site is behind Cloudflare and nobody on your team made a decision, a decision has been made.
The trade-off is asymmetric in a way that is easy to get wrong. Blocking a retrieval fetcher removes you from the answer, and Anthropic says as much about Claude-User in its own documentation. Blocking an acting agent removes you from the shortlist a buyer’s agent assembles — the agentic equivalent of not being in the consideration set, and unlike a lost citation it leaves no trace you can find in a report. Against that, an allowed agent costs you measurement accuracy and some sales time. For most companies selling to humans, the visibility loss is the larger number, and it is the one nobody invoices you for.
One gap in the tooling is worth flagging because it shapes what you can even express. Cloudflare’s managed robots.txt, last documented 3 August 2026, writes Content Signals with three terms: search, ai-input and ai-train. There is no signal for an agent may act here, or this form is for people. The vocabulary the industry has built covers reading your content three ways and says nothing about doing things on your site. Until it does, the only place to express that preference is a WAF rule, and the only party who can hear it is whoever runs your edge.
Methodology and limitations
Every user-agent string, IP file and robots.txt statement in this guide was read from the operator’s own current documentation between 24 and 26 September 2026: OpenAI’s “Overview of OpenAI Crawlers” at developers.openai.com; Anthropic’s support article “Does Anthropic crawl data from the web…”, last dated 7 April 2026, and its IP file at claude.com/crawling/bots.json, whose creationTime was 18 August 2026 when fetched; and Google’s “List of Google user-triggered fetchers”, last updated 19 August 2026. Operator documentation is authoritative about intent and unverifiable about behaviour: none of these companies publishes request volumes, and a user-agent string is a claim, which Google states can be spoofed. Purpose-split percentages are from Cloudflare, “The crawl-to-click gap”, 29 August 2025, covering requests seen by Cloudflare’s network in the twelve and six months to July 2025, with platform attribution based on operator disclosures and industry sources; Cloudflare does not define the “user actions” category in that post, so we do not treat it as a measurement of agentic action. The 2026 figures of more than 50% non-human traffic and over 36% mixed-use crawlers are from Cloudflare, “Content Independence Day, one year on”, 1 July 2026, which cites Cloudflare Radar and an investor presentation rather than a published dataset. The 51% automated and 37% bad-bot figures are from Imperva’s 2025 Bad Bot Report, drawn from Imperva’s own network. Cloudflare and Imperva both sell bot management, so in this field the primary counters of the problem are also the primary vendors of the defence, and every figure above should be read with that attached. Cloudflare’s product behaviour, bot taxonomy, Web Bot Auth mechanics and the 15 September 2026 default change are from its developer documentation pages dated 1 July, 28 July and 3 August 2026. IETF draft versions, dates and status were read from the IETF datatracker. Prompt-injection figures are Anthropic’s own, from “Claude for Chrome”, 25 August 2025 with updates to 18 December 2025, measured in an experimental autonomous mode over 123 test cases and a four-type browser-specific challenge set; they describe attacks on the agent and cannot be read as a rate of anything happening on an ordinary company website. We found no published measurement — vendor, academic or platform — of the volume of agent-submitted forms, the share of site sessions that are agentic, or the conversion behaviour of agentically-sourced leads, and we have said so rather than substituting a proxy. We did not run first-party log analysis for this guide; a company wanting a number for its own site should grep its own logs for the user agents listed above and treat the result as a floor, not an estimate.
Conclusion
Probably some of them, and nobody can tell you how many. The mechanism is documented and dated: Google publishes a user agent for an agent that performs actions on request, Cloudflare ships a blocking preset for browser-use agents and a bot category for checkout, and OpenAI and Google are both signing agent requests against a specification that has no working group behind it. What is missing is the number. The closest published figure — 2% of AI crawler requests classified as user actions in the year to July 2025 — measures a different thing, on one network, published by a company that sells the remedy. The agents most likely to be filling in your forms are the ones running inside a real person’s browser, and those are not in anyone’s dataset because at the network level they are that person. Until somebody publishes a method, the defensible position is to verify what identifies itself, tag it in your analytics, leave the retrieval fetchers alone because they are how you get mentioned at all, and check what your CDN switched on for you in September.
Frequently Asked Questions
How much agentic traffic is actually arriving at company websites?
Nobody has published a trustworthy figure. The closest published series is Cloudflare’s classification of AI crawler requests by purpose, which put user actions at 2% over the twelve months to July 2025 and 3% over the preceding six months, against 80% and 82% for training. That is a share of requests from named AI user agents on one network, not a share of a site’s traffic, and Cloudflare sells the products that the numbers justify buying.
What is the difference between a crawler, a retrieval fetch and an agent?
Three different things. A training crawl is a batch fetch with nobody waiting, made to build a training corpus — GPTBot and ClaudeBot. A retrieval fetch happens while a person waits for an answer and may produce a citation — ChatGPT-User, Claude-User, OAI-SearchBot, Claude-SearchBot. An agent acting on a page navigates, clicks and submits on behalf of one named user — Google documents Google-Agent as being used to navigate the web and perform actions upon user request. The robots.txt line you write governs these three differently, and in the third case may not govern them at all.
How does agent traffic identify itself?
Three ways, in descending order of reliability. A Web Bot Auth cryptographic signature, which the agent must opt into. A published IP list checked against a stable user agent — OpenAI publishes one JSON file per bot, Anthropic publishes claude.com/crawling/bots.json, Google publishes user-triggered-agents.json. Or the user-agent string alone, which Google’s own documentation warns can be spoofed. An agent running as an extension inside the user’s own Chrome, such as Claude for Chrome, presents none of these: it looks like the person whose browser it is.
Does robots.txt apply when a person asked the agent to visit?
The three largest operators answer this differently. Anthropic states that its bots, including Claude-User, respect do-not-crawl signals by honouring industry standard directives in robots.txt. OpenAI states that because ChatGPT-User actions are initiated by a user, “robots.txt rules may not apply”. Google states that because the fetch was requested by a user, its user-triggered fetchers “generally ignore robots.txt rules”. One Disallow line therefore has three different meanings.
What does agent traffic do to analytics?
It breaks the count in two directions at once, and Google Analytics 4 will not show you which. GA4 automatically excludes traffic from known bots and spiders using Google research and the IAB International Spiders and Bots List, and states that you cannot disable the exclusion or see how much was excluded. So a listed agent disappears without a counter, and an unlisted agent driving a real browser executes your tag and is counted as a human session — including any form it submits. No published measurement exists of the size of either error.
Should a company block agents?
Not as a reflex, and it is worth checking what your CDN has already decided. Cloudflare’s stated new defaults from 15 September 2026 block bots classified as Training or Agent on pages that display ads for new domains, while leaving Search allowed. The trade-off is asymmetric: blocking a retrieval fetcher removes you from the answer, and Anthropic says so explicitly of Claude-User. Blocking an acting agent removes you from the shortlist a buyer’s agent assembles. Verifying and rate-limiting are the cheaper middle options, and neither works against an agent running inside somebody’s own browser.
Sources
- platform documentation OpenAI, 2026. Overview of OpenAI Crawlers. Read 26 September 2026. Full user-agent strings and IP files for GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User, and the statement that robots.txt rules may not apply to user-initiated actions.
- platform documentation Anthropic, 2026. Does Anthropic crawl data from the web, and how can site owners block the crawler? Dated 7 April 2026. ClaudeBot, Claude-User and Claude-SearchBot; robots.txt, Crawl-delay and CAPTCHA commitments.
- platform documentation Anthropic, 2026. bots.json. Published IP prefixes;
creationTime18 August 2026 when fetched. - platform documentation Google, 2026. List of Google user-triggered fetchers. Last updated 19 August 2026. Google-Agent and the robots.txt carve-out for user-triggered fetchers.
- platform documentation Anthropic, 2025. Claude for Chrome. 25 August 2025, updated to 18 December 2025. Browser-extension capabilities and prompt-injection red-teaming results.
- platform documentation Google, 2026. [GA4] Bot traffic. Automatic exclusion of known bots and spiders; exclusion cannot be disabled or quantified.
- network operator Tomé, J., 2025. The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals. Cloudflare, 29 August 2025. Training, search and user-action shares of AI crawler requests.
- network operator Meunier, T., 2025. Forget IPs: using cryptography to verify bot and agent traffic. Cloudflare, 15 May 2025. The spoofing and shared-IP arguments for signing.
- network operator Lee, J-H., 2025. The age of agents: cryptographically recognizing agent traffic. Cloudflare, 28 August 2025. Signed agents definition and first cohort.
- vendor documentation Cloudflare, 2026. Verified bots. Last updated 1 July 2026. Behaviour taxonomy including Agent and Transact; Direct versus Intermediary; transitive trust and the RFC 7239 experiment.
- vendor documentation Cloudflare, 2026. Block AI Bots. Last updated 1 July 2026. Search, Agent and Training presets and the 15 September 2026 default change.
- vendor documentation Cloudflare, 2026. Web Bot Auth. Last updated 1 July 2026. Ed25519 keys, the well-known directory path and the signature headers.
- vendor documentation Cloudflare, 2026. robots.txt setting. Last updated 3 August 2026. Content Signals vocabulary: search, ai-input, ai-train.
- network operator Weiss, A., Albertson, Z. and Lanfear, E., 2026. Content Independence Day, one year on: building the business model for the agentic Internet. Cloudflare, 1 July 2026. Non-human traffic and mixed-use crawler shares, sourced to Radar and an investor presentation.
- standards draft Meunier, T. and Major, S., 2026. HTTP Message Signatures for automated traffic. Individual Internet-Draft, version 02, 19 August 2026; listed as no longer active. Replaces draft-meunier-web-bot-auth-architecture-05, 2 March 2026.
- vendor Imperva, 2025. 2025 Bad Bot Report. Automated traffic at 51% of web traffic and bad bots at 37%, from Imperva’s own network. Imperva sells Advanced Bot Protection.
- vendor announcement Parikh, S. and Surapaneni, R., 2025. Powering AI commerce with the new Agent Payments Protocol (AP2). Google Cloud, 16 September 2025. Mandates, and the authorisation, authenticity and accountability framing.
- platform documentation OpenAI, 2025. Agentic Checkout Spec. API version 2025-09-12. Five REST endpoints replacing an in-page checkout flow.