Correction, September 16, 2026. Cloudflare’s customer notice of September 15 (“Updates to managing AI crawlers on your account”, emailed to account holders) changes two things this article said in August. The Googlebot warning is withdrawn: the recommended Training setting is now “Disallow AI Training”, which the notice says lets Googlebot, Applebot and Bingbot keep indexing while telling them not to train; a legacy “Block AI Bots” switch is migrated to that setting, not to a block. Existing zones are affected after all: a zone that had “Block AI Bots” on is migrated, over the week following September 15, to Search: Allow, Training: Disallow AI Training, Agent: Block on pages with ads. Zones that never touched the setting stay on Allow. The original text is kept below with the superseded passages marked; the merchant checklist has been updated.

On September 15, 2026, Cloudflare changes its default rules for AI traffic. If your store is served through Cloudflare — or ever will be — two of those changes decide whether an AI shopping agent can load your site at all. (The August version of this paragraph added that one of them could cut you off from Google; see the correction above.) The changes are stated in Cloudflare’s own announcement, which we fetched and quote below.

This is the access layer underneath the two Cloudflare measurements we have already taken apart: the Agent Readiness Score, which grades whether an agent can find and parse your site, and the AEO Visibility Dashboard, which grades whether assistants cite you. Neither survives a default that stops the agent from loading the page in the first place.

What changes on September 15

Cloudflare now classifies AI crawler traffic into three buckets — Search (indexing to answer later), Agent (acting in real time on a person’s behalf), and Training (collecting data to train models) — and the announcement sets new defaults for them, verbatim:

“On September 15, 2026, we’ll be setting new defaults for each of these three classifications. For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.”

Cloudflare’s reasoning is explicitly about ad-monetized attention:

“An ad is a signal that a website owner meant for a person to land there and see it — something monetizable that fuels the business. So, on those pages, we treat human attention as the end goal, and keep away the bots that may prevent this attention (i.e., Training and Agent bots).”

For a content site, that logic holds. For a store, the Agent bucket is not an attention thief — it’s a shopping assistant carrying a customer’s intent to buy. ChatGPT fetching your product page to complete a purchase for a live buyer is Agent-class traffic. Blocked at the door, that traffic doesn’t degrade into a human visit; the order simply happens somewhere else.

The Googlebot trap for existing customers (superseded September 16)

This section describes the August 17 announcement. Cloudflare’s September 15 customer notice replaced this mechanism: legacy “Block AI Bots” zones are migrated to “Disallow AI Training”, which keeps Googlebot, Applebot and Bingbot indexing. Kept for the record; do not act on it.

The default flip above applies to new domains onboarding to Cloudflare. Existing zones keep their settings — but a second September 15 change reaches them directly:

“Multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors … Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).”

Read that again from a merchant’s chair: if at any point you (or an agency, or a security checklist) clicked Cloudflare’s one-click “Block AI Bots”, then from September 15 that setting starts blocking Googlebot itself, because Googlebot crawls for both Search and Training and the most restrictive rule now wins. Cloudflare says it will notify customers and that the choice can be changed in Security settings any time before the date.

Two smaller shifts land the same day and are worth knowing: Cloudflare’s “Verified bot” label no longer means “allowed by default” (a verified bot is now only allowable within an allowed category), and a new “content use” setting defaults to reference — index, excerpt, and link back.

Who is affected, precisely

  • New domains onboarding to Cloudflare after September 15: Agent and Training blocked by default on pages that display ads; Search allowed. A new store that runs ads on its own pages starts life invisible to shopping agents unless someone changes the default.
  • Existing customers who had “Block AI Bots” on (corrected September 16): migrated over the following week to Search: Allow, Training: Disallow AI Training, Agent: Block on pages with ads. Googlebot, Applebot and Bingbot keep indexing. The Agent class, the one a shopping assistant falls into, is blocked on any page Cloudflare sees ads on unless the operator changes it. (The August version of this bullet said these zones would start blocking Googlebot; Cloudflare’s notice says they will not.)
  • Existing customers who never touched bot settings: no automatic change — but the controls (including on the Free tier) are live now, and the ecosystem’s baseline is moving around you: every new Cloudflare-fronted competitor starts agent-blocked, which makes deliberate agent access a differentiator you can choose on purpose.

What to check before September 15

  1. Find out whether “Block AI Bots” was ever switched on. If it was, your zone is being migrated to per-class rules during the week after September 15, with the Agent class blocked on pages that show ads. Open Security → Bots, read the three new settings, and set the Agent class deliberately. (The August version of this step was about Googlebot; that warning was withdrawn by Cloudflare’s September 15 notice.)
  2. Split your intent by bucket, not by “AI”. Blocking Training crawlers is a legitimate content-protection choice that costs you no shoppers. Blocking the Agent class makes the store unbuyable by AI assistants. Cloudflare’s three-way controls exist precisely so you don’t have to choose “all or nothing”.
  3. Test it from the outside. Your settings page tells you what you asked for; only a probe tells you what an agent actually experiences. Our free scan sends real Agent-class requests (ChatGPT-User, Claude-User, Perplexity-User and others) at your store and reports which ones get in — the check flags Cloudflare specifically when we detect it in front of your site.

Context worth having while you’re in robots.txt territory: Cloudflare’s Content Signals Policy already writes Content-Signal: search=yes, ai-train=no into the managed robots.txt it serves for over 3.8 million domains. Those are preference signals, not enforcement — Google has publicly said its crawlers don’t act on them — while the September 15 changes are enforcement: requests actually blocked at Cloudflare’s edge. Don’t confuse the two layers when auditing what protects (or starves) your store.

FAQ

Does September 15 change anything if my store is not behind Cloudflare?

Not directly — the defaults are enforced at Cloudflare’s edge for Cloudflare zones. Indirectly, yes: a large share of the web sits behind Cloudflare, so assistants and shopping agents are adapting to a web where agent access is a merchant choice rather than a given, and stores that allow it deliberately stand out.

Will blocking Training crawlers hurt my AI-assistant sales?

No. Training crawlers collect data for model training; the shopping flows run on Agent-class fetchers acting for a live user, which are a separate classification in Cloudflare’s system. Blocking Training while allowing Agent and Search is a coherent — arguably the optimal — merchant posture.

How do I know if shopping agents can reach my store today?

Probe it as an agent, from outside. AgentReady.market’s free scan includes an Agent-class access check that requests your pages with real shopping-agent user-agents and reports which are admitted, which are blocked, and whether Cloudflare is in front of the site.

Sources

Access is the first link in the chain an agent has to complete to buy from your store — AgentReady’s scan checks the whole chain: access, structured data, catalog truth and the checkout rails.