Does Blocking a Crawler Remove You From Search?

What does robots.txt actually control?

Crawling, not indexing, and the distinction decides everything that follows. Google's robots.txt documentation describes the file as one that "tells search engine crawlers which URLs the crawler can access on your site", then states the limit plainly: it is "not a mechanism for keeping a web page out of Google".

Crawling is the act of fetching a page. Indexing is the act of recording that it exists and may be shown. A file that governs the first does not govern the second, which is the opposite of what most people assume when they add a disallow rule.

The consequence is specific: Google "might still find and index a disallowed URL if it is linked from other places on the web". Links elsewhere tell the index a page exists even when the crawler never reads it.

So a blocked page can appear in results as a bare URL with no description, because the system knows the address and has been forbidden from reading the content that would describe it. Google puts it directly: "if your web page is blocked with a robots.txt file, its URL can still appear in search results."

The naming is part of the confusion. A file called robots.txt containing a directive called Disallow reads like a prohibition on presence, when it is a request about fetching addressed to crawlers that choose to honour it. Nothing in the vocabulary signals that indexing is governed elsewhere.

What should be used instead?

The tools designed for the job. Google's instruction is to "block indexing with noindex or password-protect the page", and its warning is unambiguous: "Don't use a robots.txt file as a means to hide your web pages from Google Search results."

There is a trap in combining them that catches people repeatedly. A noindex directive lives in the page, which means the crawler has to fetch the page to see it. Disallowing the URL in robots.txt prevents that fetch, so the noindex is never read and the page can remain indexed — the two measures together achieving less than either alone.

Password protection works differently and more decisively, because content behind authentication is not retrievable regardless of what any file says.

For a company trying to suppress a page it controls, the sequence matters: allow crawling, serve noindex, wait for recrawl, and only then consider blocking. Reversing that order is the most common way a page intended for removal stays in the index indefinitely.

There is a timing dimension too. A noindex directive takes effect only when the page is next crawled, so removal from an index is never instant and depends on how frequently the site is visited — which is why publishers who need something gone quickly use password protection or removal tools rather than waiting for a recrawl.

Order matters more than choice of tool here.

What does this mean for the AI crawler blocking measured in this series?

It sharpens what those findings do and do not show, which matters because we reported them.

Our own measurement found twelve of fourteen major news domains blocking Common Crawl's CCBot and eleven blocking ClaudeBot, against six for GPTBot and four for OpenAI's search agents. Those are robots.txt directives, and robots.txt governs fetching.

So a publisher blocking an AI crawler has withdrawn permission to fetch its pages going forward. It has not removed anything already collected, as the Common Crawl report set out, and it has not made its URLs unknowable to systems that learn of them by other routes.

This is a reason to read blocking data as a statement of policy rather than as a technical guarantee — which is how both crawler reports in this series framed it, and the robots.txt documentation is why that framing was the correct one.

It also explains why the two studies reported directives rather than outcomes. A robots.txt file is the only part of this that is publicly observable; what any crawler actually fetched, retained or used is not, and no amount of reading the file reveals it.

Why do publishers block anyway?

Because withdrawing permission is meaningful even when it is not absolute.

A robots.txt directive is a public, dated, machine-readable statement that a crawler is not authorised. For a publisher negotiating licensing — the commercial motive documented in the archive report, where back catalogues became licensable assets — establishing that access was refused matters regardless of whether every actor honours it.

It also works against compliant crawlers, which is most of the ones that identify themselves. Common Crawl states that CCBot checks robots.txt before fetching and re-checks whether the file has changed; OpenAI publishes its agents so they can be controlled separately. Blocking works on the parties who asked.

What it does not do is remove a publisher from the index, stop a URL being known, or reach crawlers that ignore the file. Those limits are worth stating because blocking is sometimes sold as more than it is.

There is a further reason that has nothing to do with search. Serving crawlers costs bandwidth, and a large archive fetched repeatedly by many agents is a real infrastructure expense, which gives publishers a straightforward operational motive alongside the commercial one.

The file is also a public record, dated and machine-readable, which matters in a dispute.

Where does this go wrong for companies?

In two recurring ways, both avoidable.

The first is trying to bury a page. A company that wants an unflattering page suppressed adds a disallow rule, the page stays indexed as a bare URL, and the situation is now worse: the address is visible with no content to give it context.

The second is blocking pages a company wants found. Overbroad disallow rules on staging paths, parameters or directories routinely catch pages the business relies on, and because the crawler never fetches them, the failure is silent.

Both follow from the same misunderstanding. The file is an instruction about access for well-behaved crawlers, not a switch controlling public visibility, and treating it as the latter produces outcomes precisely opposite to the intent.

Both failures share a diagnostic property: they are invisible from inside. A page suppressed badly still looks suppressed to the person who filed the request, and a page accidentally blocked still loads perfectly for anyone who visits it directly. Only checking the file against the paths that matter surfaces either one.

What this means for anyone buying placement

Check the host's robots.txt before buying, not after. A placement on a path the publisher disallows is a page search crawlers will not read, which removes most of the search rationale for buying it.

Do not rely on robots.txt to remove anything. If coverage needs to be suppressed, the mechanisms are noindex applied by whoever controls the page, or removal by the publisher — and as the removal report in this series set out, the publisher decides.

Read AI blocking claims precisely. A publisher blocking AI crawlers has refused future fetching; it has not deleted what exists, and it has not guaranteed that non-compliant crawlers stay out.

Media Morpheus sells placement and search visibility work. What we can check for you is publicly checkable: whether a host permits crawling of the path your coverage will sit on. What nobody can offer is removal from an index through a file that its own documentation says does not do that.

And check the specific path rather than the domain, since disallow rules frequently target directories rather than whole sites.

Methodology and limitations

This report quotes Google's published robots.txt documentation in Search Central for the distinction between crawling and indexing, the behaviour of disallowed URLs, and the recommended alternatives of noindex and password protection. It also refers to original robots.txt measurements reported elsewhere in this series. Four limits apply. First, the documentation describes Google's behaviour; other search engines and crawlers implement the robots exclusion standard with their own variations, and non-compliant crawlers ignore it entirely. Second, this report addresses whether pages appear in search, not whether content is used for other purposes such as model training, which is treated separately in this series. Third, no measurement of how often disallowed URLs appear in results was conducted, because Google publishes no such data. Fourth, platform documentation changes without notice and this describes the position at the date of publication.

Conclusion

Blocking a crawler does not remove you from search, and Google says so in its own documentation: robots.txt is "not a mechanism for keeping a web page out of Google", a disallowed URL can still be found and indexed if something links to it, and such a page can appear in results as an address with no description attached. The tools that actually work are noindex and password protection — and combining noindex with a disallow rule defeats it, because the crawler must fetch the page to read the instruction. For the crawler-blocking findings reported earlier in this series, the implication is the one we stated: robots.txt records what a publisher permits, not what a system knows.

Frequently Asked Questions

Sources

1. official Google Search Central. Introduction to robots.txt, for the crawling and indexing distinction and recommended alternatives.
2. vendor research Media Morpheus original robots.txt measurements of wire services and major news domains, reported separately in this series.