I read 700 of the highest-ranked sites' robots.txt files on 9 November 2026, matched each one
against the AI crawler tokens their operators document, and published the whole reading — domain
by domain, with the exact line that decided each verdict — as open data. 407 of the 700 files
could actually be read; 145 hosts refused the request outright. Those are large, high-ranked
sites, not small businesses, so do not read the shares below as a picture of small business.

Two numbers from that reading are the reason for this post.

Of the sites that block GPTBot, 80.3% wrote a rule that names it. That block was a decision:
somebody sat down and kept their pages out of a training corpus.

Of the sites that block OAI-SearchBot — the crawler that fetches a page at the moment someone
asks a question, in order to answer with it and link back — 43.6% never chose it at all. The
block came from a rule aimed at every bot alike.

That is the whole problem in one sentence: the file people wrote to keep out training crawlers is
also keeping them out of answers that would have sent them visitors. And it usually happens
through one of four readings of RFC 9309 that are easy to get backwards.

1. * is a fallback, not an addition

This is the one that bites hardest, in both directions.

User-agent: * Disallow: /admin/ User-agent: GPTBot Disallow: / 

The GPTBot group does not inherit Disallow: /admin/. If any group names the crawler, the *
group is ignored for it entirely — not merged, not consulted for paths the specific group is
silent about. So a file like this:

User-agent: * Disallow: /private/ User-agent: GPTBot Disallow: /drafts/ 

allows GPTBot into /private/. Everything you meant to close for everyone, you have just opened
for the one crawler you were paying attention to.

The inverse is the same rule and it is why a generator that writes a group per crawler has to say
so: adding User-agent: GPTBot to a file whose * group disallows half the site relaxes the
policy for GPTBot unless you repeat those paths inside its own group.

2. Group selection is exact, not a prefix

A group for Google-Extended does not govern Googlebot. A group for Googlebot does not
govern Googlebot-Image. The matcher compares the product token itself, case-insensitively, and
nothing else.

Every parser I tried this against got it right. Every human reading I have seen of a file like

User-agent: Google-Extended Disallow: / 

got it wrong in the same direction: people read it as "I have blocked Google". It blocks one
Google product, the one used for Gemini training, and leaves the search index untouched. Which,
for most sites, is exactly what they wanted — they just did not know they had got it.

3. Longest match wins, and Allow breaks the tie

Disallow: /blog/ Allow: /blog/public/ 

/blog/public/x is allowed: the longer pattern wins. And when an Allow and a Disallow match
with the same length, the Allow wins. People write the pair above expecting the first line to
dominate because it came first; order does not enter into it.

4. An empty Disallow: allows everything

User-agent: GPTBot Disallow: 

That is the documented way to say "no restrictions". Read as a zero-length prefix — which every
path matches — it would mean the opposite, and a file that was meant to open everything would
close everything. If you write a matcher yourself, this is the test case to add first.

What I ended up building

The answer a site owner actually wants is not "allowed" or "blocked". It is "allowed or blocked,
because of this line, in this group" — because that is the only form of the answer they can
check against their own file. Every parser I looked at returns the boolean and throws the
provenance away, so our matcher returns the deciding rule, and the free checker quotes it with
its line number.

Two free things came out of it, no sign-up on either:

The generator deliberately does not offer Googlebot or Bingbot. A switch that blocks Googlebot
de-indexes your site, and that is not a control anybody should hand out next to nine AI crawlers
in the same list.

The 9 November reading, domain by domain with the deciding lines and the per-suffix figures, is at
https://www.gotopseo.com/api/public/ai-crawler-blocking/sites. If you re-run it and get
something different, I would like to see it.