Does GPTBot actually respect your Disallow rules, or does it just request the page anyway? Until now, answering that meant grepping raw CDN logs by hand. As of June 23, 2026, Microsoft Clarity will tell you for free.

What the Violations card shows

Clarity’s Bot Analytics dashboard has tracked AI crawler hits since January 2026. The new piece, announced by Ihab Rizk on the Microsoft Clarity Blog, is a Violations card that checks every bot request against a site’s own robots.txt and reports the share that hit a disallowed path.

Three things make it useful rather than a raw count dump:

  • It’s a percentage of total bot traffic, not just a raw number — so a violation rate is comparable across sites of wildly different crawl volume.
  • It’s filterable by operator, bot name and activity type, so you can isolate, say, just OpenAI’s crawlers against just your /checkout paths.
  • It tracks trend over time, so a spike after a robots.txt edit or a new crawler rollout is visible instead of buried in daily log noise.

Turning it on requires a project admin to connect a supported CDN — Fastly, Amazon CloudFront or Cloudflare — through the AI Visibility section of Project Settings; without one, Clarity only sees JS-instrumented pageviews, not the raw bot requests the check depends on.

One limit worth being explicit about: this counts requests that reached a disallowed path, not requests that were blocked. robots.txt has no technical enforcement mechanism — it’s a request, not a lock — so Clarity is measuring who ignored the sign, not stopping anyone at the door.

Why the timing matters for retail specifically

The card lands alongside a March 2026 report, “The Future of Search and Discovery for Agentic Commerce”, from Retail Economics, AWS, Botify and DataDome — drawing on Botify’s analysis of roughly 200 retail and e-commerce sites, a DataDome security scan of 698,214 live sites, and a 6,000-consumer survey across the UK, US and France (ppc.land). Two numbers from it are directly relevant to what a Violations card would surface on a typical storefront:

  • AI bot traffic to retail sites grew 5.4x over the course of 2025.
  • OpenAI’s crawlers request a retail page roughly 198 times for every visit they actually refer — versus about one Google visit for every six Googlebot crawls. For OpenAI specifically, the relationship with a retail site looks far more like scraping for its own index than like sending traffic.
  • 79.7% of the sites DataDome scanned had no protection against AI agent spoofing — meaning most sites can’t reliably tell a real crawler from something impersonating one, which is a separate but related blind spot from robots.txt compliance.

None of that 198:1 ratio tells you whether those crawls are disallowed crawls — that’s exactly the gap Clarity’s new card is built to close on a site-by-site basis, for whichever retailer turns it on.

What it doesn’t answer

A clean Violations card tells you your crawl rules are being respected. It says nothing about whether an agent that does fetch your product page can then actually complete a purchase — add to cart, authenticate, check out. Crawl compliance and transaction completion are different layers, and a storefront can score perfectly on one while quietly failing the other (a login wall the agent can’t clear, a cart that silently drops items, a checkout step with no machine-readable path at all). That gap is what an AgentReady scan is built to test directly, and it’s the same theme behind our look at Clarity’s separate Citations dashboard: free monitoring tools are proliferating fast, but they consistently stop at “is the agent showing up,” not “can the agent buy.”

Sources

As of July 2026.