Did Meta Remove "Train" From Its New Business Terms, and Can a Company Still Block Its Crawlers?

What Did the Terms Say When Nieman Lab Reported Them?

Meta’s preview of its updated commercial terms carries a banner saying the updates “will go into effect on October 30, 2026.” The terms apply to anyone using Meta’s products for business purposes, which covers running ads and managing Pages. Nieman Lab, which says it learned of the change from a reader tip, reported on October 15 that the section on rights a business gives Meta authorizes crawling of the business’s websites. It quoted the clause in full, ending: “Meta may use any content from such domains, web pages, or URLs to provide, develop, improve, research, train, and maintain the integrity of, the Meta Company Products, services, and features provided under these Commercial Terms and any other applicable supplemental terms” (Nieman Lab). Nieman Lab also noted that elsewhere in its terms Meta counts “AI at Meta” among its products.

We could not load Meta’s page directly, so we read the Internet Archive’s copies of it. The copies from October 5 and from 11:48 and 16:20 UTC on October 16 contain the sentence exactly as Nieman Lab quoted it.

What Changed on October 16 to 17?

In the Internet Archive copy captured at 03:19 UTC on October 17, and in later copies that day, the “may use any content” sentence is gone. The clause now reads, in full:

“You authorize us (or any entity acting on our behalf) to use automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents) and create test credentials to access, index, cache, analyze, collect, retrieve, or crawl the domains, web pages, or URLs (and any content available through them) that are associated with your products or services, or that are made available through your or users’ use of Meta Company Products and services, and to periodically pull updates from such domains, web pages, or URLs in order to provide, develop, improve, research, and maintain advertising and business products (including business, commercial and related products to help you create and deliver ads and other commercial content, provide users information about your products and services, and connect you with users who may be interested in your products and services) and maintain the integrity and security of Meta Company Products.”

Three things changed in that clause: the word “train” is gone, the stated purpose is now “advertising and business products,” and “integrity” becomes “integrity and security.” The authorization to crawl, the “test credentials” wording and the October 30 date did not change. The paragraph just above it, on content a business posts to Meta, was also shortened in the same edit (Meta, via the Wayback Machine).

We found no announcement of the edit from Meta, and the page does not mark it. We are not saying why the wording changed or whether Meta’s reading of the clause has changed. The archive shows only that the text changed between those two captures.

What Did the Legal Experts Say?

Nieman Lab asked three legal experts to review the earlier text. James Grimmelmann of Cornell Law School said that for news publishers it “clearly represents permission to Meta to train AI on any content from their sites.” He also said the test-credentials provision would technically allow Meta to “sign up as a fake user” to get past registration walls. Matt Topic of Loevy & Loevy said the provision would give Meta access to material “that would likely cost many millions of dollars in a licensing market,” and that publishers “should assume that it would be enforceable.” Benjamin Sobel of the University of Wisconsin compared it to antitrust “tying” (Nieman Lab). Media Copilot’s October 16 summary repeated these views (Media Copilot).

All three were commenting on the version that contained “train.” We have not seen any of them comment on the revised text, and we offer no legal reading of it ourselves.

What Does Meta Say?

Meta declined to answer Nieman Lab’s list of questions. A public affairs spokesperson said in an email that crawling language had been spread across several sets of terms, including terms on generative AI in ads, and that this update consolidates it. He said the terms will not change how Meta crawls publishers and that “content owners can still manage how their content is accessed by using robots.txt.” Meta declined to say whether the test credentials could get past paywalls (Nieman Lab). Those statements came before the October 16 to 17 edit.

Which Meta Crawlers Can Robots.txt Block?

Meta’s developer page names five crawlers and the user-agent token each one sends (Meta for Developers). We read it through an Internet Archive copy from October 5, 2026, because the live page did not load for us.

• meta-externalagent (Meta-ExternalAgent): “crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.”
• meta-webindexer (Meta-WebIndexer): “navigates the web to improve Meta AI search result quality for users.” Meta says allowing it “helps us cite and link to your content in Meta AI’s responses.”
• meta-externalads (Meta-ExternalAds): “crawls the web for use cases such as improving advertising and other business-related products and services.”
• facebookexternalhit (FacebookExternalHit): crawls pages shared in Meta’s apps to build the title, description and thumbnail of the link preview.
• meta-externalfetcher (Meta-ExternalFetcher): “fetches individual links at a user’s request,” including to help AI complete tasks for users.

To block a crawler, Meta says to “add a disallow for the relevant crawler to robots.txt,” and to “allow up to 24 hours” because crawlers may cache the file. Its own example uses the token meta-externalagent. A site that wants to keep that crawler out entirely would add User-agent: meta-externalagent followed by Disallow: /. Meta states two exceptions: Meta-ExternalFetcher “may bypass robots.txt because it performs fetches that were requested by the user,” and FacebookExternalHit “might bypass robots.txt when performing security or integrity checks.” The page does not say the other three bypass robots.txt. Our guides on blocking or allowing AI crawlers in robots.txt and telling an AI not to train on your site cover the mechanics.

Would Blocking Meta’s Crawler Be Unusual?

The chart shows that when Meta’s crawler was new, blocking it was much rarer than blocking OpenAI’s. Fortune reported that Meta had not publicly announced the crawler, and that a robots.txt rule generally needs the crawler’s name (Fortune). The data is from 2024. Our own September 2026 check of 14 large news sites, in Do News Sites Block AI Crawlers?, did not include Meta’s crawlers.

Does Robots.txt Override the Contract?

Nobody we read can say. The experts told Nieman Lab that the earlier language gave Meta an opening to argue its commercial terms take priority over a site’s blocking instructions. Meta’s spokesperson pointed to robots.txt as the control. The revised clause still authorizes crawling and test credentials, so that question still applies to the new text, though the experts’ comments on training referred to wording that has now been removed. Courts have not tested any of this. This is not legal advice.

What This Means for a Company That Advertises on Meta

The terms bind the business account holder and cover the domains tied to its products or services. Our recommendation, not a Meta rule: before October 30, have someone read the current preview rather than a summary of an earlier version, and save a dated copy. List the domains your Meta business account points to, check the robots.txt on each, and decide on purpose about meta-externalagent and meta-webindexer, which serve different ends. Blocking facebookexternalhit is a separate decision, since Meta uses it to build link previews when your pages are shared. Pages behind a login or paywall deserve their own review because of the test-credentials wording. None of this affects coverage on sites you do not control; each publisher’s own account and robots.txt decide that. Nieman Lab also tied the change to Meta One, which limits links some accounts can post; see our guide to Meta One link limits.

Methodology and Limitations

Four limits apply. First, Meta’s terms page and crawler page did not load for us directly, so the before-and-after text comes from Internet Archive copies; we cannot say exactly when between 16:20 UTC on October 16 and 03:19 UTC on October 17 the edit was made. Second, we found no Meta statement on the edit. Third, the experts’ and Meta’s comments predate the edit. Fourth, the blocking-rate data is from 2024, and we did not test any crawler.

Conclusion

The clause that drew the criticism has been rewritten: the word “train” and the “may use any content” sentence are gone, and the purpose is now advertising and business products. The permission to crawl business-linked sites and create test credentials is still there, and Meta still points site owners to robots.txt. Any company with a Meta business account should read the current text before October 30 and make a deliberate robots.txt decision.

Frequently Asked Questions

Sources

This explainer compares Internet Archive copies of Meta's commercial terms preview captured on October 5, October 16 and October 17, 2026, and reads them against Nieman Lab's October 15 report, Media Copilot's October 16 summary, Meta's crawler documentation and a 2024 Fortune report, all read on October 17, 2026. Advice on what a company should do is our recommendation, not a Meta rule. We did not test any crawler.

1. trade press Nieman Lab (Andrew Deck and Hanaa' Tameez), Meta's new commercial terms will allow the scraping of news sites, legal experts say. Supports the original clause text, expert views and Meta's statement. Read through an Internet Archive copy of October 16, 2026.
2. platform documentation Meta, Meta Commercial Terms. Supports the revised clause, the effective date and the opt-out sentence; compared with the copies of October 5 and October 16, 2026 (16:20 UTC) at the same address. Read October 17, 2026.
3. platform documentation Meta for Developers, Meta Web Crawlers. Supports the crawler names, tokens and robots.txt guidance. Read through an Internet Archive copy of October 5, 2026.
4. trade press Media Copilot, Meta's new business terms could allow news-site scraping for AI training. Supports the summary of expert views. Read October 17, 2026.
5. trade press Fortune, A new web crawler launched by Meta last month is quietly scraping the internet for AI training data. Supports the Dark Visitors blocking rates. Read October 17, 2026.