How do you allow Google-Agent through a CDN or web application firewall? Google now answers that itself. Its crawling documentation gained an Allow Google-Agent traffic page (last updated November 17, 2026) with three verification rules and a troubleshooting list of six ways a site keeps blocking legitimate Google-Agent requests after the operator thinks it has allowed them. Several of them only bite past the homepage: on search, add-to-cart and the APIs behind them.
Google-Agent is, in Google’s words, the name used by agents hosted on Google infrastructure “to navigate the web and perform actions upon user request.” We covered in July why robots.txt does not govern it. This page is the other half: what does.
How Google-Agent identifies itself
Per the new page and Google’s user-triggered fetchers reference:
- User-Agent: every request carries a
User-Agentheader containingGoogle-Agent, in a mobile and a desktop Chrome-based variant. - IP address: every request comes from a range listed in
user-triggered-agents.json. When we fetched it on November 20 the file held 20 prefixes (12 IPv4, 8 IPv6) and acreationTimeof November 19, 2026, so a copy pasted into a firewall last month may already be stale. - Signature, sometimes: Google is “experimenting” with Web Bot Auth under the
https://agent.bot.googidentity. Participating requests carry aSignature-Agent: g="https://agent.bot.goog"header plusSignatureandSignature-Inputheaders labelledg, verifiable against the key directory athttps://agent.bot.goog/.well-known/http-message-signatures-directory.
The signature is optional, so the user agent and the IP range are the checks every request can pass.
The three rules Google gives
If your CDN, WAF or bot-detection service does not verify Google-Agent automatically, Google suggests:
- Confirm the
User-Agentheader containsGoogle-Agent. - Confirm the IP address is in
user-triggered-agents.json. - If a signature is provided, verify
SignatureandSignature-InputforSignature-Agent: g="https://agent.bot.goog"according to Web Bot Auth.
Then, for verified requests, skip bot challenges. A user-agent match alone is spoofable; the IP check is what makes the rule safe to write.
Six ways a store still blocks it
Google’s troubleshooting list, with what each means for a store:
- Rule priority. The allow rule must evaluate before broad cloud or data-centre ASN blocks, managed bot rules and category-level “AI agent” block policies. A blanket AI block placed higher wins.
- Interactive challenges. Skipping HTTP blocks is not enough: the rule must also skip CAPTCHAs and JavaScript challenges, including in-page CAPTCHA widgets triggered on search or form submission.
- In-journey actions and APIs. The allowance has to cover on-site search, filtering,
product detail pages, add-to-cart, booking, and the XHR, GraphQL or
api.*endpoints behind them. An allow rule scoped to page loads lets the agent browse and stops it at the cart. - Shared-IP rate limits. Concurrent requests from many users share the same egress pool, so per-IP rate limits need to account for or exempt the verified ranges.
- IP range updates. IP rules must stay synced with the current file.
- Proxy headers. Load balancers and intermediate proxies must preserve the original
client IP, the
User-Agent, and the three signature headers, or the firewall behind them has nothing to verify.
What a merchant should check this week
- Ask your CDN or bot vendor whether it verifies Google-Agent natively; Google says some do and defers to them for exact steps.
- If you wrote the rule yourself, test it on the cart, checkout and search endpoints, not just the homepage.
- Automate the IP-range refresh rather than pasting the list once.
- Look for any “block AI agents” category toggle sitting above your allow rule. If you are on Cloudflare, re-read what its September 15 defaults change switched on for your zone.
An outside scan cannot prove your Google-Agent rule works: it does not come from
Google’s IP ranges or sign as agent.bot.goog. What it can show is the wall a generic
agent meets. An AgentReady scan requests /cart and /checkout with an agent
user agent and records whether they answer or return a challenge (401, 403, 429, 503 or
a managed challenge page).
FAQ
Does robots.txt control Google-Agent?
No. Google lists Google-Agent among its user-triggered fetchers, which act on a user’s
request rather than crawling autonomously. Google’s guidance for controlling it is at the
CDN, firewall or bot-detection layer: match the Google-Agent user agent and the IP
ranges in user-triggered-agents.json.
Do I need Web Bot Auth to allow Google-Agent?
Not today. Google describes Web Bot Auth signing under the agent.bot.goog identity as
an experiment, and only participating requests carry the signature headers. Its rules
say to verify the signature “if provided”, so the user-agent and IP-range checks remain
the baseline for every request.
Why would Google-Agent load my product pages but fail at checkout?
Google’s troubleshooting list names this case: an allow rule that covers page loads but not add-to-cart, search or the XHR, GraphQL or API endpoints behind them. Interactive CAPTCHAs on form submission and per-IP rate limits on Google’s shared egress pool can stop the agent mid-journey in the same way.
Sources
- Google for Developers: Allow Google-Agent traffic (last updated 2026-11-17, fetched 2026-11-20)
- Google for Developers: Google user-triggered fetchers
- Google: user-triggered-agents.json IP ranges (creationTime 2026-11-19, fetched 2026-11-20)
Letting an agent read the product page is discovery; letting it through the cart is the transaction, and that is the rule most allow-lists forget.