There is no single “AI bots” line for robots.txt. Six vendors run their own crawlers and fetchers under their own user-agent tokens, each controllable independently — and several run more than one token that do different jobs. Below is the current block to explicitly allow the main ones, plus what each line actually does and where it stops working.

The copy-paste block

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

User-agent: Meta-ExternalFetcher
Allow: /

User-agent: Applebot-Extended
Allow: /

Robots.txt already defaults to “allow” for anything not disallowed, so these lines don’t change access by themselves. What they do is make the policy explicit and auditable — useful once a WAF, CDN rule, or a blanket Disallow: / from a staging-site copy-paste is silently blocking a token nobody remembered to check.

What each token actually does

  • OpenAI runs three: GPTBot fetches pages as candidate training data, OAI-SearchBot builds the index behind ChatGPT Search, and ChatGPT-User fetches a page live because a user asked ChatGPT to look at it. OpenAI’s own docs note that because ChatGPT-User fetches are user-triggered, robots.txt “may not apply” to it — allowing it here is a policy statement, not a guaranteed gate (OpenAI — Overview of OpenAI Crawlers).
  • Anthropic runs ClaudeBot (training-data crawl), Claude-User (a user’s live request), and Claude-SearchBot (search-quality crawl). Anthropic is the one vendor here whose documentation states plainly that all three honor Disallow, including the non-standard Crawl-delay extension (Claude Help Center).
  • Perplexity documents PerplexityBot (index crawl) and Perplexity-User (live per-query fetch) and states both respect robots.txt (Perplexity — Perplexity Crawlers; Perplexity Help Center — How does Perplexity follow robots.txt?). Treat that claim with more caution than the others: Cloudflare published evidence in August 2025 that Perplexity ran undeclared crawlers rotating user-agents, IPs and ASNs specifically to get around sites that had disallowed the declared ones.
  • Google-Extended isn’t a crawler with its own fetch signature — Google still fetches with the regular Googlebot user agent. It’s a control token: disallowing it opts a site out of having Googlebot-crawled content used for Gemini/Vertex AI training and grounding, without touching Search ranking or indexing at all (Google Search Central — Google’s common crawlers). Allowing Googlebot does not implicitly allow Google-Extended — they’re separate lines.
  • Meta runs Meta-ExternalAgent (AI training/product-improvement crawl) and Meta-ExternalFetcher (live fetch triggered by a Meta AI product responding to a user prompt) (Meta for Developers — Web Crawlers).
  • Apple splits Applebot (powers Siri, Spotlight and Safari suggestions — has crawled for years) from Applebot-Extended, a separate 2024-introduced token that controls whether crawled content trains Apple Intelligence’s generative models. Blocking Applebot-Extended doesn’t remove a page from Apple’s search surfaces (Apple Support — About Applebot).

Where robots.txt stops being the whole answer

Three of these vendors (OpenAI, Perplexity, and — per our earlier coverage — Google’s separate Google-Agent fetcher) now run user-triggered agents whose own documentation says robots.txt isn’t necessarily binding on them, because the fetch traces back to a specific user action rather than autonomous crawling. Anthropic is the current holdout that commits to honoring Disallow across the board. Practically: robots.txt is still the right first move — it’s free, it’s the documented signal every vendor’s own crawler checks first, and it’s what most AI-crawler-detection tooling audits against — but it is not a firewall. A merchant that actually needs to stop a specific fetcher needs the IP-range or Web Bot Auth-signature controls those docs describe, not just a Disallow line.

The check that actually matters

Allowing these tokens only gets an agent past the front door. Whether it can then resolve a product’s price, variants and how to complete checkout is a separate, harder problem — the one an AgentReady scan is built to check, distinct from crawl access alone.

Sources