How do you allow Google-Agent through a CDN or web application firewall? Google now answers that itself. Its crawling documentation gained an Allow Google-Agent traffic page (last updated November 17, 2026) with three verification rules and a troubleshooting list of six ways a site keeps blocking legitimate Google-Agent requests after the operator thinks it has allowed them. Several of them only bite past the homepage: on search, add-to-cart and the APIs behind them.

Google-Agent is, in Google’s words, the name used by agents hosted on Google infrastructure “to navigate the web and perform actions upon user request.” We covered in July why robots.txt does not govern it. This page is the other half: what does.

How Google-Agent identifies itself

Per the new page and Google’s user-triggered fetchers reference:

  • User-Agent: every request carries a User-Agent header containing Google-Agent, in a mobile and a desktop Chrome-based variant.
  • IP address: every request comes from a range listed in user-triggered-agents.json. When we fetched it on November 20 the file held 20 prefixes (12 IPv4, 8 IPv6) and a creationTime of November 19, 2026, so a copy pasted into a firewall last month may already be stale.
  • Signature, sometimes: Google is “experimenting” with Web Bot Auth under the https://agent.bot.goog identity. Participating requests carry a Signature-Agent: g="https://agent.bot.goog" header plus Signature and Signature-Input headers labelled g, verifiable against the key directory at https://agent.bot.goog/.well-known/http-message-signatures-directory.

The signature is optional, so the user agent and the IP range are the checks every request can pass.

The three rules Google gives

If your CDN, WAF or bot-detection service does not verify Google-Agent automatically, Google suggests:

  1. Confirm the User-Agent header contains Google-Agent.
  2. Confirm the IP address is in user-triggered-agents.json.
  3. If a signature is provided, verify Signature and Signature-Input for Signature-Agent: g="https://agent.bot.goog" according to Web Bot Auth.

Then, for verified requests, skip bot challenges. A user-agent match alone is spoofable; the IP check is what makes the rule safe to write.

Six ways a store still blocks it

Google’s troubleshooting list, with what each means for a store:

  • Rule priority. The allow rule must evaluate before broad cloud or data-centre ASN blocks, managed bot rules and category-level “AI agent” block policies. A blanket AI block placed higher wins.
  • Interactive challenges. Skipping HTTP blocks is not enough: the rule must also skip CAPTCHAs and JavaScript challenges, including in-page CAPTCHA widgets triggered on search or form submission.
  • In-journey actions and APIs. The allowance has to cover on-site search, filtering, product detail pages, add-to-cart, booking, and the XHR, GraphQL or api.* endpoints behind them. An allow rule scoped to page loads lets the agent browse and stops it at the cart.
  • Shared-IP rate limits. Concurrent requests from many users share the same egress pool, so per-IP rate limits need to account for or exempt the verified ranges.
  • IP range updates. IP rules must stay synced with the current file.
  • Proxy headers. Load balancers and intermediate proxies must preserve the original client IP, the User-Agent, and the three signature headers, or the firewall behind them has nothing to verify.

What a merchant should check this week

  • Ask your CDN or bot vendor whether it verifies Google-Agent natively; Google says some do and defers to them for exact steps.
  • If you wrote the rule yourself, test it on the cart, checkout and search endpoints, not just the homepage.
  • Automate the IP-range refresh rather than pasting the list once.
  • Look for any “block AI agents” category toggle sitting above your allow rule. If you are on Cloudflare, re-read what its September 15 defaults change switched on for your zone.

An outside scan cannot prove your Google-Agent rule works: it does not come from Google’s IP ranges or sign as agent.bot.goog. What it can show is the wall a generic agent meets. An AgentReady scan requests /cart and /checkout with an agent user agent and records whether they answer or return a challenge (401, 403, 429, 503 or a managed challenge page).

FAQ

Does robots.txt control Google-Agent?

No. Google lists Google-Agent among its user-triggered fetchers, which act on a user’s request rather than crawling autonomously. Google’s guidance for controlling it is at the CDN, firewall or bot-detection layer: match the Google-Agent user agent and the IP ranges in user-triggered-agents.json.

Do I need Web Bot Auth to allow Google-Agent?

Not today. Google describes Web Bot Auth signing under the agent.bot.goog identity as an experiment, and only participating requests carry the signature headers. Its rules say to verify the signature “if provided”, so the user-agent and IP-range checks remain the baseline for every request.

Why would Google-Agent load my product pages but fail at checkout?

Google’s troubleshooting list names this case: an allow rule that covers page loads but not add-to-cart, search or the XHR, GraphQL or API endpoints behind them. Interactive CAPTCHAs on form submission and per-IP rate limits on Google’s shared egress pool can stop the agent mid-journey in the same way.

Sources

Letting an agent read the product page is discovery; letting it through the cart is the transaction, and that is the rule most allow-lists forget.