Meta has removed a disputed sentence from commercial terms due to take effect on October 30 after legal experts warned that it could give the company permission to use content from business-linked websites to train AI products.
The deletion, made after Nieman Lab reported on the terms, removes the clearest link between Meta’s website-crawling permission and AI training. But it does not mean the company’s automated access to business websites has gone away. A separate paragraph still authorizes Meta and entities acting on its behalf to crawl domains, pages and URLs associated with a business’s products or services.
Meta told The Verge on October 17 that the consolidated crawling terms apply to its advertising and business products, do not give it new rights over news sites and do not override website owners’ crawler preferences.
What Meta removed from the October 30 terms
The original version contained two consequential pieces of language. One paragraph authorized automated software including scrapers, bots, spiders, crawlers and user agents to access, index, cache, analyze, collect and retrieve content from associated websites. It also allowed Meta to create test credentials and periodically pull updates.
A following sentence said content from those domains and URLs could be used to provide, develop, improve, research, train and maintain Meta products and services. Because Meta’s product definitions include AI offerings, three legal experts consulted by Nieman Lab read the clause as potential contractual permission to train models on publisher websites.
That second sentence has now been removed. Meta spokesperson Daniel Roberts said the company changed the text after hearing the criticism and clarified that content owners can use robots.txt to signal their preferences. The company also stated that the update does not change how it uses web crawlers.
The revision closes the most direct route from the commercial contract to an AI-training claim. It does not settle every question about what Meta can collect under the remaining terms or how content gathered for an advertising or business function may move through the company’s systems.
The crawler authorization is still broad
The remaining language matters because its scope extends beyond material uploaded directly to Facebook, Instagram or WhatsApp. It covers domains, pages and URLs associated with a business’s products or services, along with content made available through use of Meta products.
For a publisher, retailer, creator or software company, that could include a primary website linked from a business profile or used in an advertising campaign. The terms also name several operations beyond simply fetching a public page: indexing, caching, analyzing and collecting content, plus periodically retrieving updates.
The permission to create test credentials deserves separate review. In the original Nieman Lab report, Cornell law professor James Grimmelmann noted that the wording could technically permit Meta to register for access to a site. Meta did not answer the publication’s question about whether that could extend past registration walls or paywalls. Its later statement said the terms are not about scraping news sites, but the published response did not specifically explain the test-credential boundary.
That is not proof that Meta plans to enter restricted areas. It is a reason for businesses with member-only pages, customer portals or licensed archives to have counsel and technical owners examine the final text rather than relying on a social-media summary.
Why robots.txt now matters more
A robots.txt file sits at the root of a website and gives automated agents path-level crawling instructions. Meta says those preferences continue to apply and that the new terms do not supersede them. That assurance directly addresses one of the central concerns raised about the earlier draft.
Site owners should still treat the file as one control, not a complete access policy. A directive can be outdated, target the wrong user agent or leave sensitive paths open. It also does not replace authentication, authorization or rate limits. Pages that must not be publicly collected should not depend on crawler compliance for protection.
Teams also need to distinguish Meta’s different automated visitors. A bot used to create link previews may serve a different function from a crawler used for indexing or AI-related collection. Blocking every Meta-associated agent can break previews, ad workflows or other integrations, while blocking too narrowly can leave the intended restriction ineffective.
What businesses should check before October 30
- Save the final terms and the notice shown in Business Suite. Record which legal entity and accounts accepted them, as well as the effective date.
- Inventory linked properties. List domains, subdomains, landing pages, product catalogs and customer portals connected to Meta business profiles, pixels, ads or commerce tools.
- Review crawler logs. Identify Meta-related user agents, the paths they request, their frequency and whether they reach areas not needed for previews or advertising.
- Revisit
robots.txtand edge controls. Confirm that directives match the current hostnames and paths. Use authentication and application controls for restricted content. - Test business dependencies. Before blocking a crawler, verify the effect on link cards, catalog updates, ad destinations and sharing tools.
- Escalate ambiguous data rights. Publishers and businesses with licensed, confidential or user-submitted material should have legal and privacy teams review the test-credential and associated-domain language.
The October 17 revision is meaningful: Meta removed the sentence that most clearly connected crawled website content with training its products, and it publicly disclaimed any attempt to override robots.txt. The remaining terms still grant substantial automated-access permission. Businesses have less than two weeks to map what is linked to their Meta accounts and decide whether their technical controls and contractual review match that permission.
