If you run a website, you have spent years telling a dozen different AI crawlers to stay away - one rule for Google's bots, another for OpenAI's, another for Anthropic's. On 30 October 2026, the four biggest AI labs announced a single shared signal that covers them all: one line in your robots.txt that says your content should not be used to train AI models. Here is what it is, how it works, and whether you should use it.
What just happened: four AI labs, one opt-out
Google, Microsoft, OpenAI and Anthropic have all pledged to honour one shared robots.txt policy token for AI training opt-outs, in a plan first reported by The Next Web on 31 October and confirmed by each company's own announcements the same day. The token is __usage_ai-training. Add it once, and all four labs are meant to read the same preference from your site.
The idea comes from Kevin Kelley, a technical lead for standards at Anthropic, who submitted it to the Internet Engineering Task Force (IETF) as an individual Internet-Draft on 30 October 2026. It is currently a proposal - a draft, not a finished standard - and the IETF has not formally endorsed it.
Meta voiced support for the effort but has not committed to implementing it. So for now, the signal covers the four labs, and every other AI company with its own crawler still needs its own rules.
This is not the first attempt to solve the problem: Cloudflare has blocked AI bots by default on new customer sites since July 2025, and a standard called Really Simple Licensing already lets publishers charge bots that visit their pages. But a single opt-out signal recognised by the four largest labs is the first coordinated move of its kind - and it arrived against the backdrop of a surge of scraping and high-profile lawsuits, including The New York Times' December 2023 suit against OpenAI and Microsoft over training data.
The new signal: __usage_ai-training
The neat trick of the proposal is that it invents no new grammar. A site owner writes an ordinary robots.txt rule group, but the User-Agent line names a use rather than a crawler. The example straight from the draft looks like this:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That says: prefer no AI training on anything on this site, except the /press-kit/ folder - handy for a company that would happily let a model learn from its press materials.
Two details matter enormously:
- The token must be named exactly. The familiar catch-all rule
User-Agent: *does not cover the new token - Kelley deliberately made the wildcard blind to it. If yourrobots.txtcurrently blocks every bot, you have expressed no AI-training preference at all until you add the new group by name. (The match is case-insensitive.) - It is a preference about use, not a crawl rule. The draft states it plainly: this policy token is not a directive for crawling behaviour. It says nothing about who may visit your site - only about what the four labs should not do with the content for training purposes. Search keeps working: opting out of training does not stop your pages appearing in Google Search or Bing results.
Google has confirmed that when a site uses the shared token without naming Google-Extended separately, Google will treat that as a Google-Extended block - the agent it uses to crawl for Gemini training data. Microsoft makes the same promise for its own crawler: opting out does not change whether a page can appear in Bing results.
How to add it to your website
1. Find your robots.txt file
It lives at the root of your domain: yourwebsite.com/robots.txt. On WordPress, use an SEO plugin (Rank Math, Yoast) or edit it via FTP. On Wix, Squarespace or Blogger, robots.txt editing may be limited or automatic - check your platform's settings first.
2. Add the new rule group
Paste this at the top of the file, above any existing groups:
User-Agent: __usage_ai-training
Disallow: /Want to keep some folders trainable? Add Allow: lines underneath, one per path, as in the press-kit example above.
3. Keep your existing crawler rules
The new token is a proposal and most of the labs have not implemented it yet. Your current rules - Google-Extended, GPTBot, CCBot and the rest - are still what actually works today. Add the new line; do not delete the old ones.
4. Test it
Google Search Console has a robots.txt tester under Settings; Bing Webmaster Tools has one too. Confirm the file loads correctly and the new group parses.
Note: updating robots.txt only affects future training runs. Content that was already collected into datasets or trained into model weights is not removed by opting out now.
What it does NOT do (the honest limits)
Before you add the lines, understand the boundaries - they are considerable:
- It is voluntary. The token is a statement of preference, not an enforceable block. Rogue crawlers and labs that never signed on can ignore it freely.
- It covers training only. It says nothing about AI-written answers that quote your live pages, and it does not stop crawlers visiting, caching or indexing your content.
- Scraping is still allowed by default. Until you change the setting, your content is fair game. The default is opt-in.
- Microsoft has a separate "grounding" control. Beyond training, Microsoft is adding a page-level AI grounding preference (set via a meta tag or HTTP header) that decides whether your live pages can feed answers generated on the spot by Bing, Copilot and Microsoft 365. Opting out of training does not cover grounding - and Microsoft warns that saying no to grounding will "significantly reduce" a site's visibility in those experiences.
- Rollout is staggered and mostly in the future. The signal does nothing at most of these companies today (see the timeline below). The draft itself expires on 3 May 2027 and still has to pass through the IETF's AI Preferences working group.
Who honours it, and when
| Company | Status |
|---|---|
| Anthropic | Already started implementing, per Kelley; full coverage of its services will take time. (No separate company announcement yet - support is reported via the draft's author.) |
| OpenAI | Aims to roll it out "in the coming months"; existing controls such as GPTBot continue to apply. |
| Microsoft | Honours the preferences from 1 February 2027, as it crawls; after 1 March 2027 it will stop reading its older NoArchive and NoCache tags as training/grounding signals. |
| Aims to support the token in 2027, with progress updates promised. |
Meta, again, has only expressed support - it has not committed to a timeline or an implementation.
Should you opt out? The trade-off
Opting out costs nothing and takes two minutes, but it is not free of consequences. Both Microsoft and Google say in their announcements that opting out of AI training could reduce the chances your content surfaces in AI apps and experiences - the AI Overviews, Copilot answers and chat assistants that increasingly sit between users and the open web.
Consider opting out if:
- Your content is original, expensive to produce, and already being repackaged by AI answers (news publishers, research outlets, paid courses).
- You run a members-only or subscription site where training your competitors' models is pure downside.
- You simply want a clear, on-the-record preference for future licensing conversations.
Consider holding off (or using Allow: exceptions) if:
- You depend on discovery through AI answers and chatbots - opting out trades protection for invisibility.
- Your business benefits from models knowing about your product - documentation, public knowledge bases, marketing content.
- You use Cloudflare's AI-bot blocking or Really Simple Licensing already - those do more of the actual work today than a proposal that takes effect in 2027.
The practical middle ground for most small sites: add the token now so your preference is on the record, keep your existing crawler blocks (they are what actually work today), and revisit the grounding controls - Microsoft's especially - as they go live next year.
FAQ
Does opting out stop Google indexing my site?
No. The token is a preference about training use, not a crawl directive. Your pages keep appearing in Google Search and Bing exactly as before.
Will this remove my content from AI models already trained on it?
No. The signal applies only to future training. Anything already in existing datasets or baked into model weights stays there.
Is one line really enough for all four companies?
Once implemented, yes - that is the whole point of the shared token. But today, keep your per-company rules (Google-Extended, GPTBot, and so on) because the shared signal is not yet live at most of the four.
What about AI companies that are not Google, Microsoft, OpenAI or Anthropic?
They are not covered unless they adopt the proposal. Keep the crawler-specific rules you already have for everyone else.
The opt-out debate is really about what content is worth protecting - a question that applies to how you judge the AI tools you pay for, too. See our guide to telling whether an AI tool is worth paying for, and our explainer on checking whether a photo or video was AI-generated.
No comments:
Post a Comment