For three years, opting your website out of AI training meant keeping up with a growing list of crawler names: GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot and more. That may finally end. On 30 October 2026, an IETF Internet-Draft defined a single robots.txt policy token, __usage_ai-training, and Google and Microsoft Bing announced on the same day that they will honour it. OpenAI has said it will too. Here’s how the signal works at the protocol level, how Google and Bing will interpret it, where „training” ends and „grounding” begins, and what to change in your robots.txt.
The problem: one opt-out per AI company
The Robots Exclusion Protocol (RFC 9309) was designed to control crawling, not use. To opt out of AI training, publishers had to block or configure each AI vendor’s user agent separately, and the rules differed between vendors. Some tokens control a real crawler (GPTBot, ClaudeBot). Others, like Google-Extended and Applebot-Extended, aren’t crawlers at all and only control how already-crawled content is used. Every new AI company meant another line, and every forgotten line meant an unintended „yes”.
The IETF’s AI Preferences (AIPREF) working group has been working on a general solution: a vocabulary for usage preferences and a way to attach them to content. That work is still ongoing. The new draft takes a narrower, pragmatic step: it standardises only the AI training preference, using syntax every robots.txt parser already understands.
The draft: draft-kelley-aipref-training-preference-signal
Machine-Readable Training Preference Signal (version -00, 30 October 2026) was written by Kevin Kelley of Anthropic. It’s an individual submission discussed on the AIPREF mailing list, not yet a working group document. Its key design decision is to add nothing new to robots.txt grammar. Instead, a preference is expressed by:
„writing ordinary robots.txt groups keyed by a per-purpose policy token in the User-Agent line.”
draft-kelley-aipref-training-preference-signal-00, Abstract
The example from the draft:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/This means „don’t use my content for AI training, except /press-kit/„. Paths, longest-match precedence and group structure work exactly as in RFC 9309, so existing parsers, validators and CMS plugins can handle it. The meaning of „AI training” comes from the AIPREF vocabulary draft:
„Using an asset to modify the learned parameters of a generative AI model.”
draft-ietf-aipref-vocab-08, AI Training
Models that perform exclusively non-generative tasks are excluded, even if they could generate content. So a spam classifier trained on your pages isn’t covered. A large language model is.
Five rules that matter in practice
- The wildcard doesn’t count. The draft says „wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token.” A blanket
User-Agent: * / Disallow: /blocks crawlers but expresses no training preference under this mechanism. - No group means no preference. If your robots.txt has no
__usage_ai-traininggroup, „no preference has been expressed”. It isn’t read as a yes or a no. - It isn’t a crawl directive. „This policy token is not a directive for crawling behavior.” Crawlers will still fetch your pages for search. To limit crawling, keep using crawler-specific groups.
- Specific beats general. „Specific expressions SHOULD take precedence over general ones.” A group for
Google-Extendedoverrides the shared token for Google. - Contracts beat signals. „Private agreements MAY override preferences expressed via this mechanism.” If you have a licensing deal with an AI company, the deal wins.
How Google will interpret it
In „Supporting a shared AI training control for websites”, John Mueller announced that Google will map the shared token onto its existing control:
„When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended.”
Google Search Central Blog, 30 October 2026
Three things follow from that:
- Google Search isn’t affected.
Google-Extendednever controlled Googlebot, indexing or ranking. Neither will the new token. - It also covers Gemini grounding.
Google-Extendedcontrols both Gemini training and grounding in Gemini Apps. So for Google, the shared opt-out removes you from Gemini answers too. - An explicit
Google-Extendedgroup wins. If you want to stay in Gemini answers, allow Google-Extended explicitly. Google says sites that specifically allow it „are eligible to appear in responses and receive clicks via grounding in Gemini Apps.” - The timeline is 2027. „We aim to support this new control along with other AI providers in 2027.” Google also connects it to regulation: the proposal is „consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act.”
How Bing will interpret it: training and grounding
Microsoft goes further. In „Your content, your choice”, Krishna Madhavan (Principal Product Manager, Bing) introduces two separate preferences. They replace the NoArchive and NoCache tags that Bing has used for AI controls since 2023:
- AI Training: using content to change a generative model’s learned parameters. You set it site-wide with the shared
__usage_ai-trainingtoken in robots.txt, or per page. - AI Grounding: using content as input to inform AI-generated answers in Bing, Copilot and Microsoft 365. You can only set it per page. Bing says there is „no site-wide robots.txt grounding preference.”
<!-- Page level, meta tag -->
<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallow">
# Page level, HTTP response header
X-Robots-Tag: Bingbot: ai-training:disallow
X-Robots-Tag: Bingbot: ai-grounding:disallowextended| Preference | Value | Effect |
|---|---|---|
| AI Training | allow (default) | Content may be used for training |
| AI Training | disallow | No training use |
| AI Grounding | allow (default) | Content may inform AI answers |
| AI Grounding | disallowextended | Grounding only in Microsoft’s own experiences (Copilot, Bing AI answers, Microsoft 365), not in Web IQ or Foundry grounding APIs |
| AI Grounding | disallow | Content isn’t used to inform AI answers |
Key details from Bing’s post: a page-level training preference overrides robots.txt. Preferences control use, not crawl access. An allow won’t override a robots.txt block on Bingbot. Search eligibility doesn’t change either way. Bing is also candid that opting out of grounding reduces your brand’s visibility in AI answers, and that neither choice guarantees „inclusion, traffic, or payment.”
Timeline
If you currently rely on NoArchive or NoCache to keep content out of Copilot, the 1 March 2027 deadline is the one that matters. After it, those tags are no longer read as AI preferences. To keep grounding restrictions during the transition, you need both the new on-page preference and a temporary preference recorded in Bing Webmaster Tools (available from 15 December 2026). Without either, your NoArchive pages default to allow for grounding until Bing recrawls them.
OpenAI announced in „Making it easier for websites to express AI training preferences” that it will honour the shared signal, and aims to implement it „in the coming months”.
Training, grounding, search: which control does what
The most common mistake is mixing up three different uses of your content. Training bakes it into a model’s weights, permanently. Grounding retrieves it at answer time to inform an AI response, often with a link back. Search indexes it to send users to your page. Each control covers a different combination:
The shared token is deliberately narrow. It answers one question, „may you train on this?”, for every participating provider at once. It doesn’t remove you from search. The one exception to „training only” is Google: because it maps the token to Google-Extended, an opt-out there also ends grounding in Gemini Apps.
Recipes for robots.txt
1. Opt out of AI training everywhere, stay in search. The most common choice for content sites. For Google, this also opts you out of Gemini Apps grounding (see recipe 3 to keep it):
User-Agent: *
Allow: /
User-Agent: __usage_ai-training
Disallow: /
Sitemap: https://example.com/sitemap.xml2. Opt out of training, but keep public material available. Useful for press kits, documentation or open-source docs you want models to learn:
User-Agent: __usage_ai-training
Disallow: /
Allow: /docs/
Allow: /press-kit/3. Opt out of training everywhere, but stay in Gemini. The explicit group is more specific, so for Google it overrides the shared token. The trade-off: Google-Extended allows Gemini training too, because Google offers no grounding-only switch:
User-Agent: __usage_ai-training
Disallow: /
User-Agent: Google-Extended
Allow: /4. Belt and braces until 2027. Support for the shared token is still being implemented, so keep your vendor-specific lines for now. Each one is ignored by everyone except its own vendor:
# Shared AI training preference (IETF draft)
User-Agent: __usage_ai-training
Disallow: /
# Vendor-specific controls, still needed until support ships
User-Agent: Google-Extended
Disallow: /
User-Agent: GPTBot
Disallow: /
User-Agent: ClaudeBot
Disallow: /
User-Agent: Applebot-Extended
Disallow: /Note the difference: blocking GPTBot or ClaudeBot stops those crawlers from fetching your site at all. The shared token only states a usage preference. If you only care about training, the shared token is the more precise tool once it’s supported.
Next.js: robots.ts and headers
If your robots.txt is generated by the App Router, the policy token is just another userAgent string:
// app/robots.ts
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{ userAgent: '*', allow: '/' },
{ userAgent: '__usage_ai-training', disallow: '/', allow: ['/docs/', '/press-kit/'] },
],
sitemap: 'https://example.com/sitemap.xml',
}
}Bing’s grounding preference is per page, so set it as a response header for the paths you want to protect, for example paid articles:
// next.config.ts
const nextConfig = {
async headers() {
return [
{
source: '/premium/:path*',
headers: [{ key: 'X-Robots-Tag', value: 'Bingbot: ai-grounding:disallow' }],
},
]
},
}
export default nextConfigVerify what is actually served, not what you think you deployed:
curl -s https://example.com/robots.txt | grep -A3 -i "__usage_ai-training"
curl -sI https://example.com/premium/article | grep -i x-robots-tagWhere this sits in the bigger AIPREF picture
The AIPREF working group is developing a more general system in two documents. The vocabulary draft (draft-ietf-aipref-vocab-08) defines three categories: train-ai, ai-use and search. The attachment draft (draft-ietf-aipref-attach-05, by Gary Illyes of Google and Martin Thomson of Mozilla) carries them in a new Content-Usage robots.txt rule and HTTP header:
# robots.txt (AIPREF attach draft)
User-Agent: *
Allow: /
Content-Usage: train-ai=n
Content-Usage: /ai-ok/ train-ai=y
# HTTP response header
Content-Usage: train-ai=nThat design is more expressive, but it needs new parser support and a vocabulary the working group hasn’t agreed on yet. The vocab draft itself states that its contents don’t yet reflect working group consensus. The __usage_ai-training token is the pragmatic shortcut: it uses the one definition that is widely accepted (training) and syntax that already works. In Google’s words, it gives publishers the IETF training definition „while multistakeholder discussions continue on the rest of the AIPREF vocabulary.”
Limitations to keep in mind
- It’s an individual draft. The Datatracker says it is „not endorsed by the IETF” and has „no formal standing”. The syntax could still change before it becomes a standard.
- It’s a preference, not enforcement. It works only with providers that choose to honour it. Scrapers that ignore robots.txt today will ignore this too. Rate limiting and bot management are still your only technical enforcement.
- It isn’t retroactive. Content already used to train a model stays in that model. The preference applies to content retrieved from the moment the policy is in effect.
- Changes take time. Crawlers can cache robots.txt for up to 24 hours, and Bing applies page-level preferences only after a recrawl. IndexNow can signal changes, but doesn’t guarantee a recrawl date.
For the first time, Google, Microsoft and OpenAI have committed to reading one line written by publishers, not a different line for each of them. The draft is short. Its impact may not be.
Not sure what your robots.txt, headers and meta tags are actually telling AI companies? Talk to the Fireup team. We’ll audit your site’s AI and SEO signals and set up the policy that fits your business.
Sources
- IETF: draft-kelley-aipref-training-preference-signal-00, Machine-Readable Training Preference Signal
- Google Search Central Blog: Supporting a shared AI training control for websites
- Bing Blog: Your content, your choice: clearer preferences for training and grounding
- OpenAI: Making it easier for websites to express AI training preferences
