robots.txt AI training opt-out: what the new token blocks
A draft robots.txt token, __usage_ai-training, says no to AI training without blocking any crawler. Google plans to support it in early 2027.
Research topic
Access
14 sourced articles on access, newest first: what changed, what it means for an online store, and what to fix so an AI agent can still find, understand and buy there.
Cloaking AI bots: why serving agents a different page backfires
Serving AI crawlers a different page than shoppers is cloaking: Google's spam policy bans it, and a 2025 SPLX test showed agents repeat what they are fed.
Why AI shopping agents get blocked by websites, and what to check
Walmart, Delta, Yelp and eBay told TechCrunch why personal AI agents fail on their sites; three of the four causes are bot-defence settings, not policy.
Personal Agent Protocol: what it means for your online store
Sierra and Meta's Personal Agent Protocol puts shoppers' agents on OAuth sessions that start as guests; v0.1 is due this month, so nothing to ship yet.
Safari 27's MCP Server: AI Agents Can Now Drive Safari
WebKit's Safari MCP server lets any MCP agent navigate, click, type and screenshot a live Safari window, so agents can now walk a store's checkout.
Do AI Shopping Agents Render JavaScript? The SSR Fix
Google's own JavaScript SEO documentation confirms not all crawlers run JavaScript, which is why product pages need server-rendered HTML.
Ninth Circuit refuses to rehear Amazon v. Perplexity
The full Ninth Circuit declined to rehear the AI shopping agent CFAA ruling. What is now settled, what still isn't, and why September 15 makes it urgent.
Cloudflare's September 15 flip: what our rescan measured
Pre-registered study: every Cloudflare-fronted store in our corpus is rescanned after the September 15 AI-traffic defaults, access-pillar deltas published live.
Why AI Shopping Agents Abandon Your Checkout
An AI agent that reaches your product page still fails to buy at one of several concrete, checkable points in a store's checkout path.
Cloudflare's September 15 AI Defaults: What Merchants Must Check
Cloudflare's September 15 AI defaults, corrected September 16 from its own notice: Agent blocked on ad pages, Block AI Bots migrated, no Googlebot trap.
GPTBot User Agent: The Exact String and How to Verify It's Real
The exact GPTBot user-agent string, how it differs from OAI-SearchBot and ChatGPT-User, and why real verification means OpenAI's published IP ranges.
robots.txt for AI Agents: The 2026 Copy-Paste Block
OpenAI, Anthropic, Perplexity, Google, Meta and Apple each publish distinct AI-crawler user-agent tokens for robots.txt, listed here with sources.
Microsoft Clarity Now Flags AI Bots That Ignore robots.txt
Microsoft Clarity's free Violations card shows what share of AI bot traffic to a site is requesting URLs its robots.txt disallows.
Google-Agent Ignores robots.txt — What Actually Blocks It
Google's new Google-Agent fetcher is documented to ignore robots.txt by design, so merchants need IP blocking or auth instead.
📡 The agentic-commerce brief
What changed in the protocols — and what it means for your store. Connect it to your agent and it pulls the latest itself; no inbox required.
Add it to your agent (Claude Code, Claude Desktop, Cursor — anything that speaks MCP):
MCP server config
{
"mcpServers": {
"agentready-brief": {
"type": "http",
"url": "https://www.agentready.market/mcp/docs"
}
}
}Then ask it for the AgentReady brief —
it calls get_agentready_brief and reads any entry in full. Prefer a feed
reader? Subscribe by RSS.
No agent? Get it by email instead
Email delivery isn't live yet — nothing will land in your inbox until it is. Leave your address and you'll be on the first issue when it ships.