Short answer: a new robots.txt line, User-Agent: __usage_ai-training, lets a store say “do not train generative AI on my pages” while every crawler and shopping agent keeps the access it had. It blocks nothing by itself. The risk for merchants is the opposite one: copying a popular block list that puts this token in the same group as the agents that send you buyers.

What the draft defines

The mechanism is an individual IETF Internet-Draft, draft-kelley-aipref-training-preference-signal-00, written by Kevin Kelley of Anthropic and last updated on 30 October 2026. Datatracker lists it as an individual submission with no RFC stream yet, so it is a proposal, not a standard.

It adds no new directive. A store writes an ordinary robots.txt group keyed by a “policy token”:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The draft reads that as a preference against AI training everywhere except /press-kit/. Four rules matter for a store:

  • It is not a crawl rule. The draft says the token “is not a directive for crawling behavior”. Crawlers still obey only the groups for their own names.
  • The wildcard does not count. User-Agent: * must not be read as matching the token, so an existing Disallow under * says nothing about training.
  • No group, no preference. A file without a __usage_ai-training line expresses no preference either way.
  • Vendor tokens can still win. Consumers may treat their own tokens (such as Google-Extended) as more specific than the shared one, and private agreements may override it.

“AI training” here means changing the parameters of a generative model, using the IETF aipref vocabulary. Models used only to classify, rank or score are excluded.

Who has committed to read it

Google’s crawler documentation now carries this note under Google-Extended: “Google intends to support the AIPREF shared machine-readable training preference signal (with “user-agent: __usage_ai-training”) in early 2027” (Google’s common crawlers). That is the only vendor commitment we could fetch from the vendor itself. Press reports also name OpenAI and Microsoft, but OpenAI’s post returned HTTP 403 to our fetcher, so we leave those dates out.

The same Google page explains why the distinction matters for a store. Google-Extended controls training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. A store that wanted only “no training” had no Google-specific way to say just that. The shared token is narrower: training only.

The copy-paste trap

The widely copied ai.robots.txt block list added __usage_ai-training on 5 November 2026 (PR #310). It sits in one group of 204 User-agent lines that ends in a single Disallow: /. That same group also names ChatGPT-User, Claude-User, Perplexity-User, OAI-SearchBot and Claude-SearchBot: the user-triggered and search fetchers that read a product page when a shopper asks an assistant about it.

So a store that pastes the list to opt out of training also shuts the door on agents acting for live buyers. If training is your only concern, publish the policy-token group on its own and leave the fetcher groups as they are. Our robots.txt for AI agents piece lists which token belongs to which job.

What a store should do now

  1. Decide training and access separately. They are now two different lines.
  2. If you object to training, add a standalone User-Agent: __usage_ai-training group with Disallow: /.
  3. Check that no group you copied disallows the user-triggered fetchers you want buyers to reach you through.
  4. Keep Google-Extended rules deliberate: they still also cover Gemini grounding.

FAQ

Does __usage_ai-training block AI crawlers?

No. The draft states the token is not a directive for crawling behavior, and crawlers keep following the robots.txt groups for their own names. It only expresses a preference about whether retrieved content may be used to train generative AI models. A consumer may choose to skip fetching pages it would only use for training, but nothing obliges it to.

Does a Disallow under User-Agent: * opt my store out of AI training?

No. The draft says wildcard groups must not be interpreted as matching the __usage_ai-training token. If no group names the token, no training preference has been expressed at all. You need an explicit User-Agent: __usage_ai-training line.

When will Google honour the shared AI training token?

Google’s crawler documentation says it intends to support the token in early 2027. Until then, Google-Extended remains Google’s documented control, and it covers both Gemini training and grounding. The shared token itself is still an individual IETF draft, not a published standard.

Sources

A training preference is a statement about your content; whether an agent can still reach your product page and buy is a separate question, and it is the one that decides the sale.