What Is Common Crawl, and Why Is Everyone Blocking It?

What is Common Crawl?

A charity that gives away a copy of the web. In its own words, Common Crawl is “a 501(c)(3) non-profit organization dedicated to providing a copy of the Internet to Internet researchers, companies and individuals at no cost for the purpose of research and analysis”.

It has been collecting since 2008. The corpus “contains raw web page data, metadata extracts, and text extracts”, amounting to petabytes of material, and is stored on Amazon Web Services' public datasets alongside academic cloud platforms. Access is free: users “may use Amazon's cloud platform to run analysis jobs directly against it or you can download it, whole or in part”.

For most of its existence this was uncontroversial and largely invisible outside research circles. Linguists, computer scientists, economists and journalists used it to study the web at scale — work that would otherwise require each researcher to crawl the internet independently, which almost none could afford.

What changed is not the archive but its audience. A free, bulk, text-extracted copy of the public web turned out to be an ideal input for training language models, and an institution built to democratise web research became a supply line for a commercial industry.

The design decision that matters is openness. Common Crawl does not select who may use the corpus, because selecting would defeat the purpose of providing a copy of the web to anyone studying it. That single choice is what made the archive valuable to researchers and what has now made it impossible for publishers to treat as a partner.

How does CCBot work?

Conventionally, and with more deference to publishers than its reputation now suggests.

The crawler is “a Nutch-based web crawler that makes use of the Apache Hadoop project” — built on long-established open-source infrastructure rather than anything novel. Its behaviour is the behaviour of a search engine crawler: it checks “first the robots.txt, and if crawling a page is allowed, fetches pages using HTTP GET requests”.

Exclusion is simple and documented. A site adds a directive naming the agent and disallowing it, and Common Crawl states that the crawler “will periodically continue to check if the robots.txt file has been updated”, so the decision can be reversed. There is also an opt-out registry for operators who want a record beyond their own robots file.

This matters for how the blocking data should be read. CCBot is blocked so widely precisely because it obeys the instruction — a crawler that ignored robots.txt would not show up in a study of robots.txt at all. What is being measured when publishers block it is a change in publisher preference, not any change in the crawler's conduct.

Why is it the most blocked crawler?

Because of what sits downstream of it, not because of what it does.

Our own measurement, reported separately in this series, found CCBot blocked by 12 of 14 major news domains — ahead of ClaudeBot at 11, and well ahead of OpenAI's GPTBot at 6 and its search agents at 4 each. Of every AI-related agent tested, the non-profit archive was blocked most.

The explanation follows from the asymmetry documented in that report. Publishers are blocking crawlers that take text and return nothing, and permitting those that return citations and readers. Common Crawl is the purest case of the former: it collects text, publishes it free to anyone, and sends back no traffic, no link and no attribution by design.

There is a compounding factor. Because the corpus is redistributed to everyone, blocking a single AI company's crawler achieves little if the same content remains available through Common Crawl. Blocking the archive is the only way to close that route — which makes the non-profit the first casualty of a dispute it is not a party to.

The ordering is the part worth sitting with. If publishers were reacting to AI companies as such, the commercial crawlers would lead the table. Instead the non-profit does, which suggests the objection is to redistribution rather than to any particular firm.

What does blocking it achieve?

It stops future collection. It does not retrieve the past.

A robots.txt directive is forward-looking: CCBot will stop fetching new pages, and the site's future output stays out of subsequent crawls. Material already collected across seventeen years of archives remains in the corpus and remains downloadable by anyone, because it was gathered when crawling was permitted.

So a publisher blocking today has closed a tap rather than emptied a tank. Models trained on earlier snapshots are unaffected, and so is any research or product built on the existing corpus.

It also does nothing about crawlers that ignore robots.txt entirely. The publishers most concerned about unauthorised scraping are, in blocking a compliant non-profit, restricting the actor that asked permission while the actors who never asked continue unimpeded. That is not an argument against blocking — a publisher may reasonably want to close every route it can — but it is a limit worth being clear-eyed about.

There is also an asymmetry between the parties that rarely gets stated. A publisher blocking the archive loses nothing it can measure, while the archive loses a source it cannot replace and has no commercial means of negotiating for. Where one side faces a decision with no visible cost, the outcome is fairly predictable regardless of the merits.

What gets lost?

Research capability, mostly, and it falls on people with no stake in the AI dispute.

The corpus exists so that studying the web at scale does not require the resources of a search company. Academics analysing language change, misinformation researchers tracing how claims propagate, economists studying online markets, and journalists investigating networks of sites all use it because the alternative is not a cheaper method but no method.

As major publishers withdraw, the archive becomes progressively less representative of exactly the material researchers most need. A corpus of the web missing most quality journalism is a poor basis for studying how information circulates, and the gap is invisible to anyone using the data without knowing what was removed.

This mirrors the pattern reported elsewhere in this series, where publishers restricting the Internet Archive close the preservation route as a side effect of a commercial dispute. In both cases the institution built to serve the public record is the easiest thing to switch off, and the one with the least commercial leverage to object.

What this means for publishers and buyers

For a publisher, the decision is genuinely theirs and the trade-off should be explicit. Blocking CCBot removes future content from a corpus used for both commercial training and public-interest research, and there is no way to permit one and refuse the other — the archive's whole design is that anyone may use it.

For anyone buying placement, the practical point is narrower. Whether a placement's text ends up in the most widely used public web corpus depends on the host publisher's robots file, not on anything the buyer arranges. Checking takes a moment and is worth doing before assuming coverage will feed anything.

It is also a reminder of how little of this is designed. The reachability of a company's coverage by any given system is the sum of unrelated decisions made by publishers about their own commercial interests, and it changes without notice.

Media Morpheus sells placement and search visibility work. We can tell you which doors a given publisher currently leaves open, because robots.txt is public and checkable. What nobody can tell you is what any system does with what it finds — which by this point in the series is the most consistent finding in it.

Methodology and limitations

This report draws on Common Crawl's own published description of its organisation, corpus and crawler, and on our original robots.txt measurement of fifteen major news domains conducted on 19 September 2026 and reported in full separately. Five limits apply. First, Common Crawl's published FAQ and overview do not state crawl release frequency or the size of an individual crawl in pages, and no such figures are asserted here. Second, the blocking measurement covers fourteen readable domains from a purposive sample of large English-language news outlets; it is not representative of publishing generally, and one domain returned HTTP 403 and is excluded. Third, robots.txt records stated permission at one moment and does not establish crawler behaviour. Fourth, the account of who uses the corpus for research is descriptive of documented use cases rather than drawn from a survey of users. Fifth, the observation that blocking does not remove previously collected material follows from how the archive is published rather than from a statement by Common Crawl about retention.

Conclusion

Common Crawl is a registered non-profit that has spent seventeen years giving away a copy of the public web so that researchers without a search engine's budget can study it. It obeys robots.txt, publishes how to exclude it, and re-checks whether a site has changed its mind. It is also, on our own measurement, the single most blocked AI-related crawler on major news sites — blocked by twelve of fourteen, more often than any commercial AI company's agent. The reason is not its conduct but its openness: a corpus available to everyone cannot be licensed to anyone, and closing it is the only way to make a licensing deal exclusive. The publishers have a real commercial case. The cost lands on public-interest research, which had no part in the argument and no leverage in it.

Frequently Asked Questions

Sources

1. official Common Crawl. Published overview and FAQ, for its 501(c)(3) status, corpus description, CCBot's operation and robots.txt handling, and exclusion options.
2. vendor research Media Morpheus, 19 September 2026. Original robots.txt measurement of fifteen major news domains, reported in full separately in this series.