I tested 10 websites of SEO and web-design agencies, the people other companies pay to make their sites findable, to see how ready they are for AI search (ChatGPT Search, Perplexity, Claude). The sample is small and the sites stay anonymous. The point isn't to shame anyone. It's that even specialists miss the same handful of things, and the misses that hurt are not the trendy ones.

What I checked

For each homepage I fetched the raw HTML (no JavaScript) and the standard files, the way a non-rendering crawler would:

  • robots.txt: status code, syntax, and rules for AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) vs. AI training crawlers (GPTBot, ClaudeBot)
  • /llms.txt: present, and actually Markdown rather than an HTML page
  • JSON-LD structured data in the initial HTML
  • canonical and og:url, and whether they point to the URL being served

The numbers

Check Result (out of 10)
Blocked AI search crawlers 0
Blocked AI training crawlers (GPTBot / ClaudeBot) 2, both via a CDN-managed robots block
Real Markdown /llms.txt 2
/llms.txt missing (404) 7
/llms.txt returning 200 with an HTML error page 1
No JSON-LD in the initial HTML 4
No og:url 7
robots.txt served with a non-200 status 1
Canonical pointing to a URL that redirects 1

The good news first: nobody blocked the search crawlers. The two sites that blocked GPTBot and ClaudeBot did it through a CDN-managed block, which is a reasonable policy choice and doesn't stop ChatGPT Search or Claude search from reading the pages.

The three findings that actually matter

The missing llms.txt files got the most attention when I started, but they're the least important. It's an optional convention. These three were real bugs, and each one fails silently.

1. A correct robots.txt served with a 404 status. The file had the right content, including its Sitemap: line, but the server returned HTTP 404. Crawlers read the status code first. Under RFC 9309 a 4xx robots.txt is treated as if there were none, so the sitemap hint inside it is never seen. The page in a browser looks perfectly fine.

2. A canonical that points to a redirect. The homepage is served at https://www.example-agency.com/, but its canonical says https://example-agency.com, which 301-redirects back to www. That gives search and answer engines two contradictory signals about which URL is the real one.

3. Every missing page returns 200. On one site, any made-up URL returned HTTP 200 with a "page not found" message. That's a soft 404: search engines report these, and crawlers waste time on URLs that aren't pages. /llms.txt was just the first place it showed up.

A smaller one: a robots.txt whose first line was /User-agent: *, with a stray slash. It happened to be harmless because the rule allowed everything anyway, but it shows how typos in this file fail with no error anywhere.

Check your own site in two minutes

curl -sI https://yoursite.com/robots.txt | head -1 # must be 200 curl -sI https://yoursite.com/this-should-not-exist | head -1 # should be 404, not 200 curl -s https://yoursite.com/ | grep -o '<link rel="canonical"[^>]*>' # should match the URL you fetched 

If you'd rather not use the terminal, I built a free web checker that runs these checks and shows search vs. training crawler access separately: AI Crawler Checker. The same checks are available as an open-source MCP server for Claude Desktop or Cursor: geo-seo-mcp-pro on GitHub.

What I'd take away

  • Check status codes before content. Two of the three real bugs were status-code problems that look fine in a browser.
  • Make sure your canonical matches the exact URL you serve, including www.
  • Blocking training crawlers and allowing search crawlers is a valid setup. Just make sure it's the setup you meant.
  • llms.txt is a nice extra. The basics matter more.

Disclosure: I built the checker and the MCP server mentioned above. Small sample, checked on 2026-11-18 from a non-rendering client; results for JavaScript-heavy sites may differ in a real browser.