If your robots.txt has a Content-Signal: line telling AI crawlers whether they can use your pages for search, grounding, or training, Google says it isn’t reading it — and neither, as far as Google’s own John Mueller knows, is anyone else.

What Mueller actually said

Responding on Reddit to a merchant testing Content-Signal directives and an llms-author.txt file, Google’s Search Advocate John Mueller said the Content Signals directive “has no effects whatsoever for any crawler or LLM,” calling it “bloat and future maintenance” in a robots.txt file with no behavioral payoff — as far as he’s aware, nobody parses it (Search Engine Roundtable, Search Engine Journal). In the same thread he confirmed Google doesn’t use llms.txt or the proposed llms-author.txt for Search or author disambiguation either.

That second point isn’t new from Google. Its own Search Central guidance on optimizing for generative AI features already states that maintaining an llms.txt “won’t harm (nor help) your visibility or rankings in Google Search, as Google Search ignores them” — a line we cited when Chrome Lighthouse shipped an Agentic Browsing score that scores sites on having one anyway. What’s new here is Google extending the same “we don’t read this” answer to Content-Signal, a directive built by a different vendor for a different job.

What Content-Signal actually is

Cloudflare introduced the Content Signals Policy in September 2025 as a robots.txt extension with three values: search (index and link to a page), ai-input (use it to ground a live AI answer), and ai-train (use it to train a model). It’s meant to let a publisher express more granular consent than a blanket Disallow, and Cloudflare has rolled it into the managed robots.txt file it serves for millions of domains on its network, defaulting previously-training-blocked sites to search=yes, ai-train=no.

Why this doesn’t undermine Cloudflare’s actual bot blocking

It’s worth keeping two different things Cloudflare ships apart. Content Signals is a preference string — text in robots.txt that a crawler has to choose to read and honor. Cloudflare’s separate bot-blocking — verified crawler identity plus firewall/WAF rules that actually reject a request — is enforcement, not a suggestion, and Mueller’s comment doesn’t touch it at all. The gap he’s pointing at is specific: writing “no ai-train” in a text file doesn’t stop a crawler from training on your content unless that crawler’s operator chose to build support for reading and respecting that line, and by Google’s own account, it hasn’t.

What this means for a merchant’s robots.txt

  • Don’t treat Content-Signal as a control. If you need to actually stop a crawler, that’s a firewall/WAF rule or an IP-range block against a verified bot identity — not a preference tag. Google’s ordinary crawler still honors plain Disallow in robots.txt; it’s the newer, more granular signaling layer (Content-Signal, llms.txt, llms-author.txt) that nothing confirmed is actually reading.
  • Adding it costs little, so it’s not harmful — Cloudflare’s own three-value vocabulary is cheap to set and may get picked up by other crawlers later — but budget zero enforcement value against Google today, and don’t report it to a client as a working access control.
  • Separate “did we express intent” from “did we block access” in any agent-readiness audit. Our own copy-paste robots.txt block for the named AI crawler tokens (GPTBot, ClaudeBot, PerplexityBot, and the rest) is the layer that actually changes what gets fetched; Content-Signal, for now, is a note left in a file most readers skip.

The distinction matters because it’s the same one AgentReady’s scan makes everywhere else: a discovery file or a policy string tells you what a site declared, not what actually happens when an agent shows up and tries to act on it. Robots.txt directives are a declaration a crawler can honor or ignore; only an actual agent-driven purchase tells you if your store works.

Sources