How to check whether AI crawlers can actually read your site

Getting a site online is easy now, but getting found takes much longer, and more of that happens inside AI assistants, which can only mention pages they managed to fetch. Whether they can fetch yours is something you can check in about five minutes, and the answer doesn't always match what your robots.txt suggests.

Blocking can happen in two places. robots.txt is a request that well-behaved bots read before fetching anything, while your host, CDN or firewall can refuse a bot outright even when robots.txt says it's welcome. It's easy to fix the first and never look at the second, so these steps look at both.

One thing to know before you start: the crawlers that collect pages for training or indexing (GPTBot, ClaudeBot, PerplexityBot) are separate from the ones that fetch a page when someone asks an assistant a question (ChatGPT-User, Claude-User, Perplexity-User), and each family has its own user agent and its own robots.txt group. The checks below cover both.

1. Read what robots.txt says to each bot

curl -s https://yourdomain.com/robots.txt | grep -i -A2 -E "^user-agent: (gptbot|chatgpt-user|claudebot|claude-user)" 

Here is the real output for our own site, run on 5 November 2026 at about 11:23 UTC:

User-Agent: GPTBot Allow: / Disallow: /app$ -- User-Agent: ChatGPT-User Allow: / Disallow: /app$ -- User-Agent: ClaudeBot Allow: / Disallow: /app$ -- User-Agent: Claude-User Allow: / Disallow: /app$ 

Each bot has its own group, and each group allows the whole site and disallows only the exact URL /app (the $ anchors the match to the end of the path, so /app/anything is covered by a separate Disallow: /app/ line). If a bot you care about has no group of its own, it falls back to the User-agent: * group, so read that one too.

Google-Extended is a robots.txt token only. It controls whether Google uses your content for Gemini, it doesn't change Search or AI Overviews, and it never appears as a user agent in your logs, so there is nothing to test with curl.

2. Ask the server what it answers to each bot

Most robots.txt checks stop before this step, which is where the second kind of blocking shows up. Save the script below as ua-check.sh. It requests your homepage once as a normal browser and once with each bot's user agent, follows redirects, and prints only the status codes.

#!/usr/bin/env bash # usage: bash ua-check.sh https://example.com/ SITE="${1:-https://example.com/}" check() { # name, user agent printf '%-16s %s\n' "$1" "$(curl -sL --max-time 15 -o /dev/null -w '%{http_code}' -A "$2" "$SITE")" } check "browser" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36" check "GPTBot" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)" check "OAI-SearchBot" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" check "ChatGPT-User" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)" check "ClaudeBot" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" check "Claude-User" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)" check "PerplexityBot" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" check "Perplexity-User" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)" 

Run against our site just after step 1:

$ bash ua-check.sh https://growthradar.app/ browser 200 GPTBot 200 OAI-SearchBot 200 ChatGPT-User 200 ClaudeBot 200 Claude-User 200 PerplexityBot 200 Perplexity-User 200 

You want every bot line to match the browser line. If a bot gets a 403 or 429 where the browser gets a 200, something in front of your app is refusing it, and the same goes for a 200 whose body is a "checking your browser" page. To catch that hidden challenge, send the same full user-agent string from the script and read the first few hundred bytes of the body. If the browser line fails too, the result is inconclusive, because your server may be rejecting curl itself. The user-agent strings above are the published ones at the time of writing and operators do change them, so confirm against each operator's docs if a result looks odd.

3. Check llms.txt if you publish one

curl -sL -o /dev/null -w "%{http_code}\n" https://yourdomain.com/llms.txt 

Ours returned 200. The file is optional, the format is described at llmstxt.org, and no major AI provider has said it relies on it, so treat it as housekeeping. If you do publish one, it should load as plain text with a 200 and not redirect to your homepage.

What this can't tell you

A curl with a crawler's user agent sent from your own connection is not what the real crawler experiences. Many firewalls and CDNs also check the source IP against the ranges the bot's operator publishes, so a request that only claims to be GPTBot from a home or office address can be treated differently from the genuine one, in either direction. A clean result means this URL wasn't blocked by user agent at the moment you ran it, but it doesn't prove the real bot gets through, and it says nothing about rate limits or rules on other paths. Your own access logs, filtered to the operators' published IP ranges, are the closer evidence.

If you'd rather not run curl

We build Growth Radar, and it has a free AI crawler checker that makes the same two kinds of check in the browser, with no signup.

Which bots does your own server answer differently from a browser? If you find one, we'd like to hear what was blocking it.

Disclosure: AI helped draft this article. We ran every command ourselves and pasted the output as it came back.