Can You Charge an AI Crawler? Cloudflare Built a Toll Booth on a Status Code Nobody Used.

The status code that waited thirty years

HTTP 402 Payment Required has sat in the specification since the 1990s as a reserved code for a future digital payment system that never arrived in the form its authors imagined. For most of the web's history it has been a piece of protocol trivia, quoted in lists of obscure status codes and implemented by almost nobody in production.

Cloudflare's Pay Per Crawl uses it for precisely what it was reserved for, which is rare enough to be worth noting on its own. A crawler requests a page. If it has not signalled a willingness to pay, the server answers 402 and includes the price alongside the refusal, so the crawler learns the terms rather than simply being turned away.

That is a small change with a large implication. Every existing tool for dealing with crawlers is binary — allow, or block. A 402 is neither: it is a counter-offer, and it keeps the conversation open rather than closing the door.

The mechanism matters more than the product. Once refusal can carry a price, the relationship between a publisher and a crawler stops being a permissions question and becomes a commercial one, which is a different negotiation entirely.

It is worth being precise about what this is not. It is not a standard, it is not an IETF specification, and it is not something a publisher can implement by editing a file. It is a feature of one content delivery network, which happens to sit in front of a very large share of the web.

How the transaction works

Cloudflare documents three possible outcomes for an incoming crawler request, and the ordering between them is the important part, because it determines what happens when a publisher's existing rules and its pricing intentions disagree.

If existing WAF or Bot Management rules apply, the request is blocked before any of this happens. Pay Per Crawl sits behind the blocking layer rather than replacing it, so a publisher who has already banned a crawler outright does not accidentally start selling to it.

If the crawler presents payment intent through request headers, it receives HTTP 200 and the content, as any ordinary visitor would.

If it presents nothing, it receives HTTP 402 with pricing attached — a machine-readable statement that the content is available, and what it costs.

Pricing is set by the site owner per zone. That is a coarse unit, and deliberately so: a per-article price would require a market, a clearing mechanism and a catalogue, none of which exists. A flat rate for a property is something one company can operate alone.

What counts as a bot in the first place

Charging a crawler requires identifying one first, and Cloudflare's classification is more interesting than the usual approach of maintaining a list of user-agent strings and hoping the list stays current as new entrants appear.

Its documentation groups automated traffic by behaviour rather than by declared identity, into three categories. Search “collects or indexes your content so it can answer questions about it later”. Agent covers “automated activity acting in real time on a person's behalf”, including chat fetch bots and browser-use agents. Training “crawls your content to train or fine-tune a model, permanently absorbing your data into the model”.

Those three map closely onto the distinction the IETF is standardising in its AI preferences work, which separates search, inference-time use and training into independently expressible categories.

The difference between the two approaches is enforcement. A vocabulary lets a publisher state which of the three categories it permits and relies on the recipient to honour that statement. A network sitting directly in the request path can act on the distinction itself, because it sees every request as it arrives.

That is the honest case for infrastructure-level controls over declarative ones: robots.txt asks, a proxy decides. It is also the case against, since it concentrates the decision in whoever operates the proxy.

Who is actually paying

Here the documentation stops short, and the gap is the story rather than an oversight. Everything technical is specified in detail; the commercial counterparty is simply absent from the record.

Cloudflare describes the mechanism, the status codes, the pricing model and the beta status. It does not name a single AI company participating as a payer.

That is not evidence that none exists. Commercial arrangements of this kind are routinely unannounced, and we found the same opacity across the publisher licensing market, where terms were disclosed in none of the sixteen deals we counted.

But it does define the limit of what can be claimed today. A toll booth is only a toll booth if vehicles stop at it. Absent named counterparties, Pay Per Crawl is a well-designed mechanism whose demand side is undemonstrated in public.

The feature being in closed beta, with access through a signup page or an account executive, is consistent with a product still finding out whether the other side will transact.

What it would take to work

Three conditions would have to hold for this to become a market rather than a mechanism, and none of the three is technical. The engineering is the part that already works.

The first is that major AI companies agree to pay rather than route around. A crawler that declines the 402 and leaves loses the content; a crawler that finds the same material syndicated elsewhere loses nothing, which is why wire distribution complicates this considerably.

The second is a price that means something. Per-zone pricing set unilaterally by each publisher, with no reference rate, is a market without a benchmark — the same structural problem that makes AI licensing deals impossible to value from outside.

The third is coverage. A toll on one network is avoidable by taking the same content from a property behind a different one, and most publishers syndicate.

None of these is an argument against trying, and the attempt is more serious than most proposals in this area. They are simply the reasons this remains a beta rather than a settled market, and they are worth stating plainly before anyone budgets revenue against it or sells it to a client as one.

What a publisher should do now

Treat it as measurement before treating it as revenue. Cloudflare's adjacent features — managing AI crawlers, analysing AI traffic and tracking robots.txt — tell a publisher which automated systems are actually fetching its pages and how often, which is a question most publishers genuinely cannot answer about their own properties today.

That answer has value independent of whether anyone ever pays. We found that most publishers name no AI crawlers at all in robots.txt, which is difficult to do deliberately when you do not know who is visiting.

Keep the blocking layer configured on its own merits. Pay Per Crawl sits behind it by design, so the decision to refuse a crawler outright remains entirely separate from the decision to put a price on it, and neither choice forecloses the other.

Do not budget against it. A closed beta with no named paying counterparties is not a revenue line, and presenting it as one to a board or a client would be indefensible.

And watch which side blinks. If a major model developer publicly agrees to pay a per-crawl rate, the entire permissions conversation changes shape overnight; if none does within a year, the 402 goes back to being trivia.

Methodology and limitations

The mechanism described here — the HTTP 402 response, payment intent presented through request headers, the HTTP 200 outcome for a paying crawler, pricing set per zone, the precedence of WAF and Bot Management rules, and the closed beta status — is taken from Cloudflare's own developer documentation for AI Crawl Control and Pay Per Crawl, retrieved 26 September 2026. The three behavioural bot categories of Search, Agent and Training are quoted from Cloudflare's bot concepts documentation. Cloudflare's documentation does not name participating AI companies and we have not identified any publicly; the absence is reported as an absence, not as evidence that none participate. Header field names are not specified in the documentation we retrieved, so we describe the mechanism rather than quoting a wire format. We have not tested the feature, have no commercial relationship with Cloudflare, and make no claim about revenue any publisher has earned.

Conclusion

Pay Per Crawl is the first serious attempt to make refusal negotiable rather than binary, and it does so by finally using a status code the web reserved and forgot. The design is sound: blocking stays ahead of pricing, payment is signalled in headers, and the price travels with the refusal. What is missing is the other half of a market. Cloudflare's documentation names no AI company paying, the feature is in closed beta, and per-zone pricing set without a reference rate is a number rather than a valuation. For publishers the immediate value is diagnostic — finding out which automated systems are fetching your pages — and the revenue case is unproven.

Frequently Asked Questions

Sources

1. official Cloudflare, AI Crawl Control documentation, retrieved 26 September 2026. Source of the feature set: managing AI crawlers, analysing AI traffic, tracking robots.txt and Pay Per Crawl.
2. official Cloudflare, Pay Per Crawl documentation, retrieved 26 September 2026. Source of the HTTP 402 response, payment intent in request headers, per-zone pricing, the three request outcomes and the closed beta status.
3. official Cloudflare, Bot concepts documentation, retrieved 26 September 2026, for the Search, Agent and Training classification of automated traffic by behaviour.
4. vendor research Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals, João Tomé, 29 August 2025. Share of AI crawling by purpose. Read October 10, 2026.