What Is __usage_ai-training, and Will It Affect AI Search?

On October 30, 2026, Google and Microsoft published plans for one shared robots.txt line about AI training, and OpenAI announced its support the same day. If you own the public docs at an API or developer tools company, the request that follows is short: opt us out of AI training. Does that line also take your product docs out of AI answers?

The line is a group that names the policy token __usage_ai-training. It comes from an individual Internet-Draft, and the IETF has not adopted it as a standard. In the pages we read, none of the four providers in this guide says it honors the token yet.

It depends on the provider and the product surface. At Google, once support lands, Google plans to read a token-only opt-out as a Google-Extended disallow. That also ends your eligibility for grounding in Gemini Apps and Vertex AI, while AI Overviews and AI Mode run on separate controls. At Microsoft, training and grounding are independent preferences. OpenAI keeps its training crawler and its search crawler on separate rules and has not published how it will implement the token. Anthropic's status is unknown.

What __usage_ai-training is

The token comes from Machine-Readable Training Preference Signal, draft 00, by Kevin Kelley of Anthropic. The IETF Datatracker entry lists it as an individual draft that the IETF has not endorsed and that has no formal standing in the standards process. The author's employer is not a statement about how Anthropic's crawlers behave.

The draft defines AI training as using an asset to change the learned parameters of a generative AI model. It adds no new directives and no new grammar. You write a normal robots.txt group, and the user agent line names the policy token instead of a crawler. The draft gives this example. It is illustrative, so do not paste it over an existing file.

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That group states a preference against AI training for every path except those under /press-kit/. A disallowed path states a preference against the use, and an allowed path states a preference in favor of it. The draft also sets three rules that matter when you review a file:

• If no group names the token, no preference has been expressed.
• A wildcard group (User-Agent: *) does not match the token.
• Providers must match the token exactly, apart from case.

On crawling, the draft is direct: "This policy token is not a directive for crawling behavior." A merged pull request on specification.website names the review risk well. The group looks like a crawler rule, but it only states a training preference and does not block fetching.

Four uses that one request blurs

"Opt out of AI" can mean four things, each with its own controls.

1. Crawl access: whether a crawler may fetch a URL at all. Crawler product tokens such as Googlebot, Bingbot, GPTBot, and ClaudeBot govern this.
2. Ordinary search inclusion: whether a page can appear in classic search results. Neither Google-Extended nor Microsoft's new usage preferences change it.
3. Training: whether content is used to change a model's learned parameters. This is the only use __usage_ai-training addresses.
4. Grounding an answer: whether a model receives your content at answer time and can cite or link it. Google describes grounding as giving the model content from its Search index when the prompt runs.

Training data reflects your pages as they were when a crawler collected them, while grounding and answer-time fetches read live pages or a search index. We covered that difference for removed endpoints in why ChatGPT still recommends the API endpoint you removed. In our view, that makes a training opt-out, for an API company, also a choice about what future models learn about your API.

Provider comparison, as read on November 3, 2026

Grounding sits on a separate control at every provider except Google, where Google-Extended covers both training and Gemini grounding.

Google: the training opt-out also decides Gemini grounding

Google's Search Central post by John Mueller says Google aims to support the control along with other AI providers, on the timing in the table. When a site uses the token to opt out of training and Google-Extended is not explicitly specified, Google "plans to interpret that as if the publisher had disallowed Google-Extended." That behavior is planned and not live.

The effect follows from what Google-Extended covers. It governs training of the Gemini models behind Gemini Apps and the Vertex AI API for Gemini. It also governs grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google's announcement adds that sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. So a token-only Disallow, once Google honors it, will also end that grounding eligibility.

Google-Extended does not affect Search. Google says it has no impact on a site's inclusion in Google Search and is not a ranking signal there. AI Overviews, AI Mode, and generative AI in Discover have their own Search generative AI control in Search Console. Excluding a site removes its traffic and impressions from those features and makes its content ineligible as input for their AI responses. Google says the control is not a ranking or inclusion signal for other parts of Search, and that it does not affect training: "to limit training of the models used to generate responses in Search generative AI features, use Google-Extended."

Our reading of Google's pages: Google's published controls offer no way today to refuse Gemini training while keeping Gemini Apps grounding. An explicit Google-Extended Allow keeps grounding eligibility, and it also allows training. When a file names Google-Extended explicitly, that group decides, and Google does not apply the token mapping.

Microsoft: two preferences on a recrawl schedule

Microsoft's announcement by Krishna Madhavan splits the decision in two: "your training setting does not change your grounding setting, and vice versa." The Bing Webmaster FAQ fills in the details.

Training uses the shared token per site or path. Microsoft adds a page-level extension: a Bingbot meta tag, or an X-Robots-Tag header (the HTTP header form of a robots meta tag), with the value ai-training:disallow. A page-level training setting overrides robots.txt. One caution: the FAQ's explicit Allow example for page-level training uses a grounding value, so confirm that syntax with Microsoft before using it.

Grounding is set per page, and there is no site-wide robots.txt grounding preference. The values are allow, ai-grounding:disallowextended, and ai-grounding:disallow. The extended value also excludes grounding services Microsoft provides to third parties. The announcement names Microsoft-operated grounding experiences such as AI Mode, AI Overview, Copilot, and Microsoft 365.

Keep the milestones in the table apart. The temporary grounding preferences exist only for sites that use NoArchive or NoCache, older robots meta values that Bing reads today as a training or grounding preference. There is no temporary training setting. The February milestone is when Microsoft starts to honor the new preferences as it crawls. The March milestone ends NoArchive and NoCache as a training or grounding signal. In both cases a published setting takes effect only when Bing recrawls the robots.txt file or page where it is published.

None of this changes eligibility for traditional Bing search results, and Microsoft says neither choice guarantees inclusion, traffic, or payment. Both preferences default to Allow when nothing applicable is published. The FAQ says Microsoft's page-level and grounding preferences apply only to Microsoft, so treat that default as Microsoft's policy, not as a rule of the token.

OpenAI and Anthropic: separate bots, unpublished or unknown plans

OpenAI says it "will honor a common signal in robots.txt." It aims to implement the signal in the coming months and will share implementation details later. The signal will not replace OpenAI's existing robots.txt controls.

Those controls are in OpenAI's crawler overview. GPTBot collects content used to make OpenAI's generative AI foundation models more useful and safe. OAI-SearchBot surfaces websites in ChatGPT's search features. Each setting is independent, so a site can allow OAI-SearchBot and disallow GPTBot. For ChatGPT-User, OpenAI says robots.txt rules may not apply. Search results adjust about 24 hours after a robots.txt update. The page does not mention the token.

Anthropic's crawler article covers three bots. Restricting ClaudeBot signals that a site's future materials should be excluded from Anthropic's training datasets. Blocking Claude-User stops retrieval in response to a user query, and blocking Claude-SearchBot stops indexing for search. The article does not mention the token, and we found no first-party Anthropic announcement about it, so its status is unknown.

The case for one line, and where it stops

The strongest counterargument is that the shared token exists so one line means the same thing everywhere. Add it and stop worrying about per-provider controls. OpenAI notes that site operators used to maintain separate rules for every provider, and Google, Microsoft, and OpenAI have each said they plan to support the token. A Shopware issue, written as a merchant user story, asks for Content-Usage and Content-Signal lines in robots.txt "so that I can allow AI search while refusing AI training," and reports that Shopware's robots.txt settings do not accept those lines.

The draft itself limits that idea. It says specific expressions should take precedence over general ones, lets providers document their own tokens as more specific, and lets private agreements override the token. Google reads the token only when Google-Extended is not explicitly specified and says its existing company-specific controls keep working. Microsoft lets a page-level training setting override robots.txt, and its FAQ says each participating company describes how it honors the preference. The token also covers training only, so it says nothing about grounding.

The counterargument holds for one narrow goal. Once a provider honors the token, a site that only wants to state a training preference has one line to maintain for that provider. What happens to AI answers still depends on each provider's grounding control.

Which policy fits your situation

• You want no training anywhere and accept the Google trade-off. Add the token as a Disallow, keep your GPTBot and ClaudeBot rules because those are the training controls OpenAI and Anthropic document today, and record that Google plans to read the token as a Google-Extended disallow, which removes Gemini Apps and Vertex AI grounding eligibility.
• You want to refuse training and keep AI answers using your docs. Microsoft supports this split, because grounding is a separate page-level choice that defaults to Allow. OpenAI separates GPTBot from OAI-SearchBot. Google's published controls do not offer the split for Gemini Apps grounding, so set an explicit Google-Extended rule on purpose.
• You are fine with training. With no token group, no preference is expressed. An explicit Allow in the token group states a preference in favor of training.
• You want out of Google's Search AI features. Use the Search generative AI control. It does not change ranking in other parts of Search and it does not affect training.

How to check the file you deployed

These checks confirm the file you published. They say nothing about how a provider uses it.

1. Request /robots.txt from your public hostname, from outside your network, and confirm the token group is served with the agreed paths and the exact spelling __usage_ai-training. A User-Agent: * group never counts as the token. This command prints the line that names the token and the three lines after it. It follows redirects and stops with an error when the request fails, so empty output with no error means the file was fetched and no line names the token. Replace the hostname with your own.

curl -fsSL https://www.example.com/robots.txt | grep -iE -A 3 '^[[:space:]]*user-agent:[[:space:]]*__usage_ai-training([[:space:]#]|$)'

2. Check every layer that can write or rewrite the file. Cloudflare's "Disallow AI Training" setting works by publishing a preference in robots.txt, and its separate Block option stops Bingbot and Googlebot from reaching the site, search included.
3. Read your existing crawler groups again. They still decide crawl access, and at Google an explicit Google-Extended group decides training and Gemini grounding. The draft allows the token to share a group with crawler tokens, so review any such group line by line.
4. Record the Search Console generative AI control setting for each property, and check that any page-level Bingbot values match the policy.

Microsoft applies preferences only on recrawl, so expect a delay there.

What is still unknown

• Whether any provider honors __usage_ai-training today. No source we read shows that one does.
• OpenAI's implementation date and how the token maps to GPTBot, OAI-SearchBot, and ChatGPT-User.
• Anthropic's position on the token. Its crawler article is silent on it.
• When Google support arrives. Its blog and crawler docs give slightly different timing (see the table).
• Whether Gemini Apps grounding draws on content you exclude through the Search Console control. Neither Google page says.
• Whether Microsoft's training preference covers third parties that license Bing data.

Review checklist to hand over

Give this to the internal owner who asked for the opt-out and to the engineer making the change.

1. Inventory current controls by provider and surface: robots.txt groups for every crawler in the table; the Search Console generative AI control; Bingbot page-level values; NoArchive and NoCache use; and CDN or platform settings that write robots.txt.
2. Write the desired policy as a grid: for each provider, training yes or no, grounding yes or no, and Search AI features yes or no.
3. Agree on the grid with the internal owner, including the Google trade-off between Gemini training and Gemini Apps grounding.
4. Have engineering draft the proposed change as a diff against the live file and page templates.
5. Review the diff for exact token spelling, Google-Extended rules that match the grid, Bingbot page-level values, and a NoArchive and NoCache plan before the March milestone.
6. Deploy, then run the file check in the previous section.
7. Set review dates for the Bing transition window, Google's update, and OpenAI's implementation notice.

Start with the inventory this week. List your current controls by provider and surface, agree on the training and grounding policy with the internal owner, and have engineering review a proposed change against the draft's definitions before it ships.

Sources

• Google Search Central Blog: shared AI training control
• Google common crawlers: Google-Extended
• Search Console Help: Search generative AI control
• Google Search Central: AI features and your website
• IETF Datatracker: draft-kelley-aipref-training-preference-signal
• Machine-Readable Training Preference Signal, draft 00
• Bing: Your content, your choice: clearer preferences for training and grounding
• Bing Webmaster Tools: FAQ, New AI Preferences
• OpenAI: Making it easier for websites to express AI training preferences
• OpenAI: Overview of OpenAI Crawlers
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Cloudflare: Have it both ways: stay discoverable in search while disallowing AI training
• specification.website pull request 291
• Shopware issue 21769
• EkLine: Why ChatGPT still recommends the API endpoint you removed