If you’ve asked “how do I know that request claiming to be an AI shopping agent actually is one” — the answer forming across the industry right now is Web Bot Auth: a way for a bot or agent to cryptographically sign its own HTTP requests, instead of asking a site to trust a spoofable User-Agent string or IP address. It started as a crawler-verification spec. In 2026 it’s become the identity layer that Visa, Mastercard and American Express are building agentic checkout on top of — which makes it relevant to merchants, not just security teams.

(Updated August 2026: this guide now walks the full web bot authentication flow end to end, with the real signature headers our own crawler sends in production — we implement the spec we’re describing.)

Web bot authentication, end to end

Web Bot Auth is built on RFC 9421 (HTTP Message Signatures). The whole mechanism is three moves:

  1. The bot signs. The agent holds a private key (Ed25519 in every deployment we’ve seen) and signs each request over a small set of covered components — the target host and a Signature-Agent header naming who is calling. The signature carries a creation and expiry timestamp, so a captured request can’t be replayed later.
  2. The bot points to its keys. The Signature-Agent header names a directory URL; the operator serves its public keys there, at the standardized path /.well-known/http-message-signatures-directory, as ordinary JWKs.
  3. The site verifies. The origin (usually its CDN/WAF) fetches that directory once, caches it, and checks each request’s signature against the published key. Match → this really is the operator it claims to be. No shared secret, no allowlisted IP range to keep in sync.

That’s the entire trust move: identity stops being an assertion (a User-Agent string anyone can type) and becomes a proof (a signature only the key-holder can produce). Standardization is underway at the IETF: a chartered working group (webbotauth), a standards-track authentication spec due to the IESG by April 2026, and a Best Current Practice document on key management due August 2026 (IETF webbotauth charter). Cloudflare originated the approach in 2025 and documents the mechanics in its bot-verification reference.

What a signed request looks like — ours

We don’t just write about this spec: the AgentReady audit crawler implements it. When its signing key is configured, every request our auditor sends carries RFC 9421 signature headers of exactly this shape:

Signature-Agent: "https://www.agentready.market"
Signature-Input: sig1=("@authority" "signature-agent");created=1755100000;
  expires=1755100300;keyid="<RFC 7638 thumbprint>";alg="ed25519";tag="web-bot-auth"
Signature: sig1=:MEUCIQDx…base64-signature…:=

Three details worth copying if you implement it yourself:

  • The covered components are minimal on purpose — @authority (the host being called) plus the Signature-Agent header itself, per the web-bot-auth profile. Enough to bind the signature to both the caller and the callee; small enough to survive proxies that rewrite everything else.
  • Signatures expire in minutes (ours: 300 seconds). A logged request is not a reusable credential.
  • The public key is one URL away. Ours is served, as a JWK with its RFC 7638 thumbprint as keyid, at https://www.agentready.market/.well-known/http-message-signatures-directory — the same standardized path Google uses for its own directory (below). Any origin can fetch it and verify us, today, with no relationship to us.

Who’s actually verifying it

The news isn’t the cryptography — it’s that verification has spread past Cloudflare to the rest of the stack a store might sit behind:

  • AWS WAF, Vercel, Akamai and HUMAN Security now verify Web Bot Auth signatures, and Amazon’s own Bedrock AgentCore Browser signs outbound agent requests with it in preview specifically to skip CAPTCHAs at sites that recognize the signature.
  • Google published its own guide, “Google’s Guide to Authenticating Requests with Web Bot Auth” (marked Experimental), including a public-key directory at agent.bot.goog/.well-known/http-message-signatures-directory — Google is now a first-party participant in the spec, not a bystander.
  • Cloudflare and GoDaddy partnered to pair Web Bot Auth’s cryptographic signatures with GoDaddy’s DNS/PKI-based Agent Name Service, extending crawler management and agent-identity verification to GoDaddy’s much larger base of smaller sites.

Why this matters more than “just” bot management

The part that should get a merchant’s attention: Visa’s Trusted Agent Protocol (TAP) and Mastercard’s Agent Pay both authenticate AI shopping agents using Web Bot Auth as the signing layer, and American Express is building its agentic-commerce offering on the same foundation (Cloudflare — “Securing agentic commerce: helping AI Agents transact with Visa and Mastercard”). In practice: an agent registers with a payment network, its public key lands in that network’s directory, and it signs its checkout request with the matching private key — letting a merchant’s site tell “a Visa-registered shopping agent” apart from an unverified scraper before the payment step, the same way ACP and UCP already split checkout into two non-interoperable rails.

What to check on your own store

  • Does your CDN/WAF (Cloudflare, Akamai, AWS WAF, Vercel, HUMAN Security) have Web Bot Auth verification turned on, and is it configured to let signed agents through rather than treat all automated traffic the same?
  • If you accept Visa or Mastercard, is your checkout flow prepared to see Web Bot Auth-signed agent requests arrive under TAP/Agent Pay, rather than only human, cookie-based sessions?
  • Is this on your radar as a Pillar 1 (access) check, separate from — but feeding into — whether an agent can actually complete a purchase once it’s let in? That transaction-level gap is what an AgentReady scan checks directly — and since August 2026 the scan probes the identity layer itself: our signed crawler sends one extra unsigned control request under the same agent user-agent and reports whether your edge treats the two differently (informational for now — it never lowers a score).

FAQ

Is Web Bot Auth the same thing as robots.txt?

No — they answer different questions. robots.txt states your policy (which bots you invite), but it cannot verify who is knocking: any scraper can wear GPTBot’s User-Agent. Web Bot Auth is the identity layer: a cryptographic signature proving the request really comes from the operator it claims. You need both — a policy, and a way to know who you’re applying it to.

Does Web Bot Auth replace the User-Agent string?

No. Agents still send a User-Agent, and sites still read it to know which family a bot belongs to. What changes is trust: the User-Agent is a claim anyone can type, while a Web Bot Auth signature can only be produced by the holder of the operator’s private key — published for verification at their /.well-known/http-message-signatures-directory.

Do merchants need to implement web bot authentication themselves?

Mostly no. Verification lives in the CDN/WAF layer — Cloudflare, Akamai, AWS WAF, Vercel and HUMAN Security already do it. A merchant’s job is configuration, not cryptography: check that verification is enabled, and that verified agent traffic is allowed through instead of being lumped in with unverified scrapers. Signing is the bot operator’s side of the handshake.

Sources

Originally published July 2026; expanded August 2026. Web Bot Auth is still a draft IETF spec — check the working group’s datatracker page for the current milestone status before making implementation decisions.