Does robots.txt block Google’s new AI agent? No. Google’s own crawler documentation now lists Google-Agent — the fetcher behind Project Mariner-style browsing agents acting on a user’s behalf — as a “user-triggered fetcher,” and Google’s docs are explicit that this class of fetcher generally ignores robots.txt entirely, because the request traces back to a specific human action rather than autonomous crawling.

For merchants who’ve been leaning on Disallow rules to keep agent traffic away from staging pages, internal search results, or account areas, that’s worth checking today, not after a scan turns up unexplained hits in server logs.

What Google-Agent actually is

Google-Agent is described in Google’s crawling documentation as the identity used by Google-hosted AI agents — including Project Mariner-style browsing agents — when they fetch a page in direct response to a user’s prompt. That puts it in a different bucket from Googlebot, which crawls autonomously to build the search index and does honor robots.txt.

Because the fetch is a “direct consequence of a user’s action,” per Google, the same logic Google applies to its other user-triggered fetchers applies here: robots.txt doesn’t govern it. A merchant’s robots.txt can look fully agent-friendly by every normal heuristic and Google-Agent will still fetch a page it’s disallowed from, because the docs classify it as behaving more like a browser at a user’s request than a crawler.

Google isn’t alone — and isn’t the strictest, either

Google-Agent joins a growing list of “user-triggered” agent fetchers that don’t treat Disallow as binding:

  • OpenAI’s ChatGPT-User: OpenAI revised its crawler documentation on December 9, 2025, removing language that had applied robots.txt tags to ChatGPT-User alongside OAI-SearchBot and GPTBot — the current docs scope robots.txt compliance to the latter two only (OpenAI crawler docs).
  • Anthropic’s Claude-User is the documented holdout: Anthropic states that all three of its bots — ClaudeBot, Claude-User, and Claude-SearchBot — honor standard robots.txt directives, including Disallow and the non-standard Crawl-delay extension (Anthropic crawler support page).

So the practical map for a merchant auditing agent access today is: Google and OpenAI’s user-triggered fetchers bypass Disallow, Anthropic’s doesn’t — and that split can change again with the next documentation revision from either vendor.

What actually stops (or verifies) Google-Agent

Since robots.txt isn’t a lever here, Google’s own documentation points to two other controls:

  • IP-range blocking. Google publishes the IP ranges its user-triggered fetchers use in dedicated JSON files, and the traffic resolves via reverse DNS to *.gae.googleusercontent.com or google-proxy-*.google.com depending on whether the fetch runs on Google-owned or user-owned infrastructure. A WAF or CDN rule keyed to that published range (not the spoofable user-agent string) is the only firm opt-out today.
  • Web Bot Auth signature verification. Google has started signing a subset of Google-Agent’s requests with an HTTP Message Signature under the identity agent.bot.goog, per its “Guide to Authenticating Requests with Web Bot Auth” (marked Experimental). We covered how that cryptographic layer works and who else verifies it in our Web Bot Auth piece — Google frames this as a limited pilot, not a full rollout, so IP-range checks remain the fallback for now.

Neither of these is a robots.txt-style toggle a merchant can flip from a CMS settings page — both require server- or CDN-level configuration.

Why this matters for merchants, not just SEOs

The reflex is to treat “ignores robots.txt” as purely an access-control problem to lock down. But for a store trying to be buyable by AI agents, the same mechanism cuts both ways: a shopping-capable browsing agent acting on a real shopper’s behalf will reach product and pricing pages regardless of Disallow, which is arguably the outcome a merchant wants — as long as the pages it reaches are actually structured well enough for that agent to understand price, availability and how to complete the purchase. The robots.txt line only ever controlled whether the fetch happened; it never controlled whether the agent could parse and act on what it found. That’s the part an AgentReady scan checks.

Sources