GPTBot is OpenAI’s training-data crawler, and it identifies itself with a user-agent string of this shape:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot

Two practical rules fall out of that string immediately. First, match on the token GPTBot, not the full string: the version segment moves over time (1.0, 1.1 and 1.2 have all been observed in the wild), so any log filter, WAF rule or robots.txt line pinned to an exact version will silently rot. Second, the string proves nothing — any scraper can send it. Verification is an IP question, not a string question (below).

GPTBot is one of three OpenAI fetchers

Seeing “GPTBot” in your logs — or blocking it — tells you about exactly one of OpenAI’s three fetchers (OpenAI — Overview of OpenAI crawlers):

  • GPTBot — crawls pages as candidate training data for OpenAI’s models.
  • OAI-SearchBot — builds the index behind ChatGPT search; this is the one that makes a store citable in answers.
  • ChatGPT-User — fetches a page live because a user asked ChatGPT to look at it; OpenAI notes robots.txt “may not apply” to these user-triggered fetches.

Each has its own robots.txt token and its own published IP ranges. Disallowing GPTBot does not touch the other two — a store can opt out of training and still be fully present in ChatGPT search. The full copy-paste robots.txt block for all vendors’ tokens is in our robots.txt for AI agents guide.

Verifying a hit is really GPTBot

Because the user-agent is trivially forged, OpenAI publishes the egress IP ranges for each fetcher as JSON:

  • https://openai.com/gptbot.json
  • https://openai.com/searchbot.json
  • https://openai.com/chatgpt-user.json

Each file is a list of prefixes[].ipv4Prefix CIDR blocks (for example, 20.171.206.0/24 and 52.230.152.0/24 are currently in the GPTBot list). The check that holds up: a request is genuine GPTBot only if the UA token matches and the source IP falls inside the published ranges. Community mirrors like openai-crawlers-ip-ranges re-fetch these files weekly and flatten them into plain-text CIDR lists ready for nginx/WAF ingestion — that repo exists precisely because site owners kept seeing GPTBot-signed traffic from unofficial IPs.

The robots.txt lines

# allow training crawl
User-agent: GPTBot
Allow: /

# or opt out of training only
User-agent: GPTBot
Disallow: /

Robots.txt defaults to allow, so the Allow block changes nothing by itself — its value is making the policy explicit and auditable. The failure mode we actually see in scans is the inverse: robots.txt says allow, the WAF says 403. A CDN bot-fight rule or a blanket bot-management preset blocks the GPTBot user-agent at the edge, and nothing in robots.txt reveals it.

How we test it

Every AgentReady scan runs a differential probe: it fetches the store’s homepage once as a normal browser and again presenting the real GPTBot user-agent (plus ClaudeBot and PerplexityBot), then compares status codes and content length. A 403 on the GPTBot probe with a 200 on the browser baseline means the store is blocked for AI crawlers in practice, whatever robots.txt declares — that’s a hard fail on the access pillar, because an agent that can’t get in can’t do anything else.

Sources