Most robots.txt problems are not "Google ignored my rules". They are the file saying something other than what the owner thinks it says. We audited six small static sites last month and the same four patterns kept showing up, in the same order of frequency.

Here is the table we ended up using, and the two commands behind it.

What is wrong What it looks like in the file The check that names it
Left over from staging Disallow: / under User-agent: * Fetch the file and read the first group, not your notes from last month
Served, but not as text A 200 whose content-type is text/html curl -sI and look at the header, not the body
Training and retrieval swapped GPTBot allowed, OAI-SearchBot disallowed Compare the two groups line by line
Sitemap points at another host Sitemap: https://something.pages.dev/... Compare the declared host with the domain you think you are checking

The two commands, which work when any tool is down:

curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' https://example.com/robots.txt curl -sS https://example.com/robots.txt | head -40 

The content-type case deserves its own paragraph

A host that answers 200 with an HTML error page is the worst of the four, because the status looks healthy in every dashboard. The body starts with <!doctype html> and every rule you thought you had is gone. This is not hypothetical: it is what we found on our own site when /ads.txt answered 200 with the homepage HTML, and it is the reason our checklist starts with a header read rather than a file read.

Precedence is documented, and it is not what most people assume

Google's robots.txt reference states the rules plainly: the most specific user-agent group wins, and inside a group the longest matching path decides, with Allow winning an equal-length tie. That order matters when a file has both Disallow: /private and Allow: /private/ok. If you have never read that page, the four lines that matter are worth ten minutes.

What the specification does not say is that any of it is enforced. robots.txt is a request. A crawler that ignores it is not violating a standard; it is making a different choice. That distinction changes what you should expect from the file: it will not stop a scraper, and it will not keep a page out of an index that chooses to ignore it.

The two switches people mix up

Training crawlers and retrieval crawlers are separate decisions, and the names do not make that obvious:

  • GPTBot, Google-Extended, ClaudeBot, CCBot — training access.
  • OAI-SearchBot, PerplexityBot — retrieval, meaning whether you can appear in those answers at all.

Google-Extended is the one that gets misread most often. It controls Gemini training. It does not touch Googlebot, and blocking it does not remove you from Search. Sites that block it "for privacy" and then wonder why their AI-overview presence is uneven are usually looking at the wrong switch.

We built the checker we wanted

Reading four things across a fetched file, at a glance, turned out to be the repetitive part, so we made it a page: robots.txt checker fetches the live file from the host and reports the four checks above, plus a per-crawler verdict with the exact rule that decided it.

Two things we deliberately did with it. It states what it does not check — wildcards and $ anchors are reported as written, not evaluated per URL, and nothing in it can tell you whether a crawler obeys the file. And it gives you the curl replacement, because a checker whose own behaviour you cannot reproduce is just another dashboard.

If you only take one thing from this: a 404 on robots.txt is fine. A 5xx is not. And "we have a robots.txt" is a statement about a file's existence, not about what it permits.