How Do You Verify a Crawler Is What It Claims to Be?

Why is verification necessary at all?

Because the name a crawler gives is simply a string it chose. Google's crawler documentation sets out three identifying signals: "The HTTP user-agent request header. The source IP address of the request. The reverse DNS hostname of the source IP."

Only the first is supplied by the visitor, and anything can supply it. A request announcing itself as Googlebot is making a claim, not proving one, and the cost of making that claim falsely is nothing.

This matters for publishers in a specific way. Sites frequently treat well-known crawlers preferentially — serving them faster, exempting them from rate limits or bot challenges — which means an impostor using the name inherits that treatment.

The two remaining signals are the answer. The source IP can be checked against published ranges, and the reverse DNS hostname of that IP resolves to the operator's own domain, which a forger cannot fake without controlling the address.

There is a second motive for impersonation beyond preferential treatment. Content behind soft gates — metered paywalls, registration walls, bot challenges — is frequently relaxed for search crawlers so that pages can be indexed, and a request wearing a trusted name inherits that relaxation. Impersonation is therefore a route to content as well as to speed.

Do all crawlers obey robots.txt?

Not identically, and the documentation is careful about it in a way that summaries usually are not.

Google groups its crawlers into three categories: common crawlers used for its products, special-case crawlers, and user-triggered fetchers invoked by an end user through a tool.

Common crawlers "always respect robots.txt rules for automatic crawls". That is a strong statement and it is scoped precisely — to that category, and to automatic crawling.

Special-case crawlers behave differently by design. The documentation gives AdsBot as an example, noting it "ignores the global robots.txt user agent (*) with the ad publisher's permission". A publisher who blocks everything with a wildcard rule has not necessarily blocked that crawler.

The practical lesson for anyone writing a robots.txt file is that a wildcard is not a universal switch. Blocking an agent reliably means naming it.

The category distinction also explains a common support complaint. A publisher who blocks everything and then finds a Google crawler still arriving has usually met a special-case crawler behaving exactly as documented, rather than a malfunction — which is discoverable only by reading the category the crawler belongs to.

How does this bear on the crawler studies in this series?

It marks the boundary of what those measurements established, which we stated at the time and is worth restating with the documentation behind it.

Two original studies here measured robots.txt directives: five wire services, none of which named any AI crawler, and fifteen major news domains, twelve of which block Common Crawl's CCBot. Both reported stated permissions.

A robots.txt file records what a publisher permits. It does not record which crawlers arrived, whether they identified themselves honestly, or whether they honoured the file. Those are separate questions answerable only from server logs, using the verification signals above.

So a publisher wanting to know what actually visited has to check logs and verify the visitors, and a reader of either study should treat the findings as policy rather than behaviour — which is how both reports framed them.

It also sets a fair standard for reading anyone else's crawler research, including work that reaches stronger conclusions than ours did. If a study reports what crawlers did rather than what publishers permitted, it needs server logs and verified requests behind it, and a methodology that does not mention either is describing robots.txt files in the language of behaviour.

What can a publisher actually do?

Verify before trusting, and name agents rather than relying on wildcards.

Verification is mechanical: take the source IP from the log entry, perform a reverse DNS lookup, confirm the hostname belongs to the operator's domain, then resolve that hostname forward and confirm it returns the original IP. Operators including OpenAI also publish IP ranges in machine-readable files, as documented earlier in this series, which allows the same check without DNS.

For blocking, the reliable approach is to name the specific user agents rather than assume a wildcard covers them. The AdsBot example shows that a documented crawler may be designed to ignore the global rule.

And for anything that must not be publicly available, robots.txt is the wrong instrument entirely — as the companion report in this series sets out, Google's own documentation says a disallowed URL can still appear in search results, and recommends noindex or password protection instead.

The forward-confirmation step is what makes the check sound, and it is the one most often skipped. A reverse lookup alone can be satisfied by a hostname the requester controls; resolving that hostname forward and requiring it to return the original IP closes the loop, because only the legitimate operator controls both directions.

Logging enough to make this possible is the prerequisite most sites miss. Verification needs the source IP and user-agent retained for long enough to investigate, and a site that discards or aggregates that detail cannot answer questions about who visited it, however carefully its robots.txt is written.

What does this mean for a company buying coverage?

Mostly it improves how you read claims about crawling and visibility.

A vendor asserting that a particular AI system has crawled a client's coverage is making a claim about server behaviour they usually cannot observe, because the logs belong to the publisher rather than to them.

A publisher asserting that it blocks AI crawlers is describing its robots.txt, which is checkable in seconds and is a statement of policy rather than a technical guarantee.

Both claims are legitimate when stated accurately and both are routinely overstated. The distinction between what a file says, what a log shows and what a verified request proves is the difference between three quite different assertions that tend to be reported as one.

Media Morpheus sells placement and search visibility work. What is publicly checkable — whether a host permits crawling of a given path — we can check with you. What happened in a publisher's server logs is theirs, and nobody selling coverage has it.

The practical asymmetry is worth stating plainly. A buyer can verify what a publisher permits, because robots.txt is public. A publisher can verify what actually arrived, because the logs are theirs. Nobody occupies both positions, which is why claims spanning the two should be read as inference rather than observation.

What this means for anyone buying placement

Read crawler claims precisely. Permission, arrival and honest identification are three separate facts, and only the first is public.

If you run a site, verify before you privilege. Preferential treatment handed out on the strength of a user-agent string is handed to anyone who types it.

Name agents in robots.txt rather than trusting a wildcard, since Google documents at least one crawler designed to ignore the global rule with permission.

And do not use robots.txt to hide anything. It governs fetching by cooperative crawlers, and as the companion report here documents, Google says plainly that it is not a mechanism for keeping a page out of search.

The broader habit this series keeps recommending applies here too: prefer the fact you can check yourself over the one you are told. A robots.txt file, a page's markup, a published fee schedule and an archived copy are all things a buyer can inspect without anyone's cooperation, and they are worth more than assurances precisely because of that.

Methodology and limitations

This report quotes Google's published overview of its crawlers and fetchers for the three identifying signals, the three crawler categories, the statement that common crawlers always respect robots.txt rules for automatic crawls, and the AdsBot example. It also refers to original robots.txt measurements reported elsewhere in this series. Four limits apply. First, the documentation describes Google's own crawlers; other operators document their agents separately and behave according to their own policies. Second, the page examined did not set out Google-Extended's treatment or a general statement about user-triggered fetchers and robots.txt, and nothing is asserted here about either beyond what was published on it. Third, no server log analysis was conducted for this report, so no claim is made about which crawlers actually visit any site. Fourth, platform documentation changes without notice and this describes the position at the date of publication.

Conclusion

A crawler's name is a claim, not a credential. Google documents three identifying signals — the user-agent header, the source IP and the reverse DNS hostname — and only the last two can be checked by the person receiving the request. The documentation is equally careful about obedience: common crawlers always respect robots.txt for automatic crawls, while special-case crawlers may not, with AdsBot documented as ignoring the wildcard rule with the publisher's permission. For the two crawler studies in this series the implication is the one we stated throughout: a robots.txt file records what a publisher permits, and what actually arrived is a different question, answerable only from logs and only after verifying that the visitor was who it said it was.

Frequently Asked Questions

Sources

1. official Google Search Central. Overview of Google crawlers and fetchers, for identifying signals, crawler categories and robots.txt behaviour.
2. official Google Search Central. Introduction to robots.txt, cited for what robots.txt does and does not control.