robots.txt AI training opt-out: what the new token blocks
A draft robots.txt token, __usage_ai-training, says no to AI training without blocking any crawler. Google plans to support it in early 2027.
Research topic
Robots.txt
9 sourced articles on robots.txt, newest first: what changed, what it means for an online store, and what to fix so an AI agent can still find, understand and buy there.
How to Check if AI Shopping Agents Can Buy on Your Store
Five checks you can run yourself, in about fifteen minutes, to see whether an AI shopping agent can find, understand and reach checkout on your store.
terms.txt: A robots.txt Alternative That Lets AI Agents Pay
A new preprint proposes terms.txt, a robots.txt-style file with per-purpose prices and signed receipts for AI agents. It has no adopters yet.
Cloudflare's September 15 AI Defaults: What Merchants Must Check
Cloudflare's September 15 AI defaults, corrected September 16 from its own notice: Agent blocked on ad pages, Block AI Bots migrated, no Googlebot trap.
GPTBot User Agent: The Exact String and How to Verify It's Real
The exact GPTBot user-agent string, how it differs from OAI-SearchBot and ChatGPT-User, and why real verification means OpenAI's published IP ranges.
Content-Signal in robots.txt: Google Says It Does Nothing
Google told a merchant that Cloudflare Content-Signal robots.txt tags have no effect on any AI crawler, unlike enforced bot-blocking rules.
robots.txt for AI Agents: The 2026 Copy-Paste Block
OpenAI, Anthropic, Perplexity, Google, Meta and Apple each publish distinct AI-crawler user-agent tokens for robots.txt, listed here with sources.
Microsoft Clarity Now Flags AI Bots That Ignore robots.txt
Microsoft Clarity's free Violations card shows what share of AI bot traffic to a site is requesting URLs its robots.txt disallows.
Google-Agent Ignores robots.txt — What Actually Blocks It
Google's new Google-Agent fetcher is documented to ignore robots.txt by design, so merchants need IP blocking or auth instead.
📡 The agentic-commerce brief
What changed in the protocols — and what it means for your store. Connect it to your agent and it pulls the latest itself; no inbox required.
Add it to your agent (Claude Code, Claude Desktop, Cursor — anything that speaks MCP):
MCP server config
{
"mcpServers": {
"agentready-brief": {
"type": "http",
"url": "https://www.agentready.market/mcp/docs"
}
}
}Then ask it for the AgentReady brief —
it calls get_agentready_brief and reads any entry in full. Prefer a feed
reader? Subscribe by RSS.
No agent? Get it by email instead
Email delivery isn't live yet — nothing will land in your inbox until it is. Leave your address and you'll be on the first issue when it ships.