OpenAI, Google, Microsoft and Anthropic are moving toward a shared way for website owners to say their pages should not be used to train generative AI models. The proposal uses one new policy token in robots.txt, replacing the need to maintain a separate training rule for every participating company.
The signal is called __usage_ai-training. It was published October 30 in an Internet-Draft written by Anthropic engineer Kevin Kelley. OpenAI has committed to honor it in the coming months, Google is targeting early 2027, Microsoft will begin on February 1, 2027, and Kelley told the IETF working group that Anthropic has started implementing support.
Website operators should not remove their existing AI-crawler rules yet. The shared token is not fully deployed, it expresses a preference rather than enforcing access, and it controls training rather than ordinary crawling or appearance in AI-powered search results.
How to opt out of AI training with the shared token
The whole-site version is short:
User-Agent: __usage_ai-training
Disallow: /That rule tells providers that support the draft not to use any path on the site to modify the learned parameters of a generative AI model. To keep one section available for training, add a more specific Allow line:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/The draft reuses the path matching and longest-match rules already defined for robots.txt. In the second example, the site declines training across the domain but permits it for pages under /press-kit/.
Put the group in the robots.txt file at the root of each relevant hostname, such as https://example.com/robots.txt. A separate subdomain normally has its own file, so a rule on www.example.com should not be assumed to cover docs.example.com.
Three limits that are easy to miss
The wildcard rule does not count
An existing group addressed to User-Agent: * does not express an AI-training preference under the draft. The file must name __usage_ai-training explicitly. This was deliberate: the authors did not want old crawler-access rules to acquire a new legal or operational meaning without a publisher changing the file.
It does not block crawling
The token describes how content may be used. It is not the user-agent string that a bot sends in an HTTP request, and it does not by itself stop a crawler from fetching pages. A site that also wants to restrict access must continue to address the relevant crawler, firewall unwanted traffic or use another access control.
OpenAI, for example, separates GPTBot, which relates to foundation-model training, from OAI-SearchBot, which affects discovery in ChatGPT search. Its crawler documentation says those choices are independent. Blocking the search bot can reduce visibility in ChatGPT search, while blocking GPTBot indicates that content should not be used for training.
It is not live everywhere
The specification is an individual Internet-Draft, not a finalized IETF standard. Provider support is also arriving on different schedules:
- OpenAI: plans support in the coming months and says its existing crawler controls will remain in force.
- Google: its crawler documentation targets early 2027. The existing Google-Extended control continues to govern training future Gemini models and some grounding uses.
- Microsoft: will start honoring the token February 1, 2027. It is also adding page-level settings and a separate AI-grounding preference.
- Anthropic: Kelley wrote to the IETF AI-control mailing list that implementation has begun, without giving a completion date.
Keep existing provider-specific rules during the transition
For now, publishers should add the shared group alongside any controls they already rely on. A site that wants to remain visible in search products while declining foundation-model training might use a transition file like this:
User-Agent: GPTBot
Disallow: /
User-Agent: Google-Extended
Disallow: /
User-Agent: __usage_ai-training
Disallow: /This example does not block Googlebot or OAI-SearchBot, so it is aimed at training preferences rather than general search indexing. It is not a universal template: publishers should inventory the services they want to allow, review current provider documentation, and account for any crawler rules already required by their content management system, CDN or security tooling.
Microsoft’s approach also shows why “training” and “grounding” should not be treated as synonyms. Training changes model parameters. Grounding supplies current pages to an AI system when it answers a prompt. A publisher may accept one use and decline the other, and the shared token only addresses training as defined by the draft.
How WordPress site owners can add the rule
WordPress may generate a virtual robots.txt response when no physical file exists. Before editing anything, open /robots.txt in a browser and identify whether the response comes from WordPress, an SEO plugin, the web server or a CDN.
- Record the current file so existing search and sitemap directives are preserved.
- Add the
__usage_ai-traininggroup through the same layer that currently manages the file. - Clear page, server and CDN caches.
- Request the public URL again from a private browser session or with
curl https://example.com/robots.txt. - Check each important subdomain separately.
- Keep existing provider-specific training rules until the companies’ announced support is actually active and verified.
Do not use a site-visibility setting that blocks all search engines merely to change AI-training preferences. In WordPress, the “Discourage search engines” option is broader than this task and can harm normal discovery.
A common signal reduces maintenance, not enforcement
The shared token solves a real administrative problem. Publisher rules currently depend on knowing which product token each company uses and updating the file as providers change. A purpose-based signal can remain stable even when crawler names or collection systems change.
It remains voluntary. The draft does not create technical enforcement, remove material collected before support goes live or settle copyright questions. It gives publishers a clearer, machine-readable statement and gives participating AI companies one place to read it. For site owners, the sensible move is to prepare the rule now, preserve current controls and revisit the file as each provider documents its final behavior.
