Short answer: llms.txt doesn’t make a page more fetchable — a controlled experiment found it changes nothing about whether an agent’s fetch succeeds. But a separate dataset of 1,033 real agent runs shows it does change agent behavior: when a site ships one and an agent finds it, that agent follows its links and grounds answers in it at meaningfully higher rates than when it isn’t there. Two different questions, two different answers — and a merchant who only asks the first one is missing the reason to bother with the file at all.

Both datasets come from a public, CC BY 4.0 release published by the ora.ai research lab with Vercel in August 2026 — 1,033 traced agent runs across 25 developer-tool sites (Stripe, Notion, Datadog, Zapier and others) and 190 controlled fetch probes, all reproducible from the raw CSVs. We pulled the data ourselves and recomputed every number below directly from it.

The fetchability grid: almost nothing matters except two things

The controlled half of the dataset serves one documentation page under 19 configurations — adding llms.txt, AGENTS.md, sitemap.xml, JSON-LD, Open Graph tags, a Markdown mirror, a canonical tag; removing headings or the meta description; adding redirects or user-agent cloaking — and probes each with a plain, non-JavaScript fetch and a JavaScript-executing one. A probe succeeds if the response contains a planted answer string.

Recomputed from data/fetchability.csv: 16 of the 19 configurations succeed 10/10, on both client types. The three that don’t:

  • Outright blocking (403 to agent user agents) — 0/10 on both clients.
  • JavaScript-only rendering (the answer sits inside a JS payload with nothing in the static HTML) — 0/10 on a plain fetch, 10/10 with JS execution.
  • Every other tested feature — llms.txt included — makes no measurable difference to whether the fetch succeeds.

That’s a real result, not a rounding artifact: adding llms.txt to a page that already serves plain HTML doesn’t unlock anything a plain fetch couldn’t already read. If your product page renders server-side and doesn’t 403 agent traffic, it was already fetchable — the file adds nothing at that layer.

The trace data: llms.txt changes what agents do next

Fetchability isn’t the whole question, and the 1,033 traced runs measure the part the grid can’t: what an agent actually does once it can read your site. Recomputed from data/traces.csv:

  • Among the 853 runs against sites that ship an llms.txt, agents fetched it in 38.6% of runs.
  • Of the runs that fetched it, 36.0% went on to fetch a page llms.txt had listed — meaning the agent used it as a navigation map, not just a page it happened to read.
  • Of the runs that fetched llms.txt, 36.5% ended with the final answer traced back to content from that file specifically, rather than docs, the homepage, or another page.
  • Overall, 81.7% of all runs in the dataset ended with a grounded answer — llms.txt is one contributor among several, not the dominant one.

The number that matters most for a merchant deciding whether it’s worth maintaining: how agents found the file at all. Across every run, 85.6% of llms.txt fetches arrived via a link an agent followed, not a path it guessed. The same pattern holds even more strongly for adjacent discovery files: .well-known fetches were linked 92.7% of the time, openapi.json 96.7%, and every single AGENTS.md fetch in the dataset (38/38) arrived by link.

What this means for your store

The dataset’s 25 sites are developer tools and SaaS products, not e-commerce — worth naming as a real caveat before applying it to a storefront. But the mechanism it isolates generalizes: retrievability and discoverability are separate failure modes, and llms.txt only fixes the second one.

  • Fix rendering and access first. If product or category pages don’t render without JavaScript, or you 403 agent user agents, that’s the only thing in this data that actually blocks a fetch — no discovery file compensates for it. See our llms.txt for e-commerce guide for what the file should contain once that’s true.
  • Link it, don’t just publish it. An llms.txt earns its keep by being reachable from somewhere an agent’s crawl actually visits — homepage, footer, robots.txt — not by existing at /llms.txt alone. This dataset shows agents mostly find these files by link, not by guessing the path.
  • Expect a behavior effect, not a fetchability score. llms.txt won’t move whether a page can be read; it moves what an agent does once it’s already reading — closer to what a transaction actually needs.

FAQ

Does llms.txt improve whether AI agents can access my pages?

Not by itself. A controlled test across 19 site configurations found llms.txt made no difference to fetch success — the only two things that blocked a fetch were serving the answer exclusively inside client-side JavaScript and returning a 403 to agent user agents. If your pages already render as plain HTML and don’t block agent traffic, they’re already fetchable with or without the file.

Is llms.txt worth implementing for a store, then?

The data says yes, for a different reason than fetchability. Among traced agent runs, a third of the runs that read an llms.txt on a site that shipped one went on to follow a link it listed, and about the same share grounded their final answer in it — real navigation and citation effects, not something a plain HTML page produces on its own.

How do AI agents typically find a site’s llms.txt file?

Overwhelmingly by following a link, not by guessing the URL. Across the traced runs, 85.6% of llms.txt fetches arrived via a link the agent had already encountered, and the figure was even higher for .well-known (92.7%), openapi.json (96.7%) and AGENTS.md (100%). An llms.txt with no inbound link on the site is unlikely to be found at all.

Sources

  • agentready-org/standard — public GitHub repository holding the dataset behind the “Make your site readable by AI agents” report, written by the ora.ai research lab in collaboration with Vercel (commit f52dca9, fetched 2026-08-28). Note for readers: the standard published in that repository is hosted at agentready.org and is not affiliated with AgentReady (agentready.market), operated by Villepinte Studio — a name collision, not a shared project.
  • data/traces.csv — 1,033 traced agent runs, one row per run; percentages above were recomputed directly from this file.
  • data/fetchability.csv — 190 controlled fetch probes across 19 site configurations; success counts above were recomputed directly from this file.
  • data/README.md — dataset methodology (site list, models, harnesses, collection window).

We audit the transaction, not just discovery — but a page an agent can’t even fetch never gets the chance to be bought from, which is why we check both.