TL;DR
- 38 of 100 well-known sites serve an
llms.txtfile (42% of the 91 we could check). The split is sharp: 25 of 35 developer-tool sites, 2 of 18 news and media sites. - 24 of the 96 sites whose
robots.txtwe could read (25%) block at least one of GPTBot, ClaudeBot and PerplexityBot from/. Among news and media sites it is 14 of 17 for ClaudeBot. Only 10 of 96 block OAI-SearchBot, the crawler OpenAI documents for ChatGPT search. - Our own audit tool had two blind spots here: it could not load 27 of the 100 homepages, and it reported an
llms.txtfor 5 sites where we found only HTML pages. We report numbers from direct fetches, not from audit scores.
This is a snapshot of 100 sites we chose by hand on one day. It is not a ranking of the web, and it does not say whether any of these settings changes whether AI products cite a site.
What we measured
We wanted a few facts that anyone can re-check with curl:
- Does
https://<domain>/llms.txtreturn a text file? - What does
robots.txtsay about GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and CCBot for the path/? - Does the site declare a sitemap, or serve
/sitemap.xml? - Does the homepage HTML contain JSON-LD and an
og:title?
The domains are a fixed list of 100: 35 SaaS and developer tools, 10 AI companies, 18 news, media and publishing sites, 20 consumer and marketplace sites, and 17 education, reference and other sites. The list is in the data folder. We also sent every domain through SEOBolt's free audit endpoint, then compared its output with the direct fetches. SEOBolt is our product; the comparison is there to show where it is wrong.
Results
| Check | Result | Out of |
|---|---|---|
Serves a text llms.txt | 38 (38%) | 100; 9 could not be checked |
Declares a Sitemap: in robots.txt | 79 (82%) | 96 readable robots.txt |
Declares a sitemap or serves /sitemap.xml | 85 (85%) | 100 |
| Homepage has JSON-LD | 51 (61%) | 83 homepages we could fetch |
Homepage has og:title | 73 (88%) | 83 homepages we could fetch |
Blocking means a Disallow rule wins for the path / in the group that applies to that crawler (its own group, otherwise *):
| Crawler | Blocks / | Of 96 |
|---|---|---|
| GPTBot | 20 (21%) | 96 |
| ClaudeBot | 24 (25%) | 96 |
| PerplexityBot | 18 (19%) | 96 |
| OAI-SearchBot | 10 (10%) | 96 |
| ChatGPT-User | 11 (11%) | 96 |
| Claude-SearchBot | 15 (16%) | 96 |
| Google-Extended | 13 (14%) | 96 |
| CCBot | 24 (25%) | 96 |
By category (blocking counts are out of the sites with a readable robots.txt):
| Category | llms.txt | GPTBot blocked | ClaudeBot blocked | PerplexityBot blocked |
|---|---|---|---|---|
| SaaS and developer tools | 25 of 35 | 2 of 35 | 2 of 35 | 1 of 35 |
| AI companies | 4 of 10 | 0 of 9 | 0 of 9 | 0 of 9 |
| News, media, publishing | 2 of 18 | 10 of 17 | 14 of 17 | 12 of 17 |
| Consumer and marketplaces | 3 of 20 | 6 of 19 | 6 of 19 | 3 of 19 |
| Education, reference, other | 4 of 17 | 2 of 16 | 2 of 16 | 2 of 16 |
llms.txt: common among developer tools, rare elsewhere
llms.txt is a proposal published at llmstxt.org by Jeremy Howard in 2024. It is a Markdown file at the site root that summarises a site for language models. We count presence only. We did not check the format or content, and we make no claim that having one changes how AI products treat a site.
Presence tracks category more than anything else. 25 of 35 developer-tool domains serve one, and so do 4 of 10 AI companies, against 2 of 18 news sites and 3 of 20 consumer sites. Nine domains answered our requests with 401, 403, 405 or no response, so we cannot say either way. File sizes among the 38 ranged from about 2.8 KB to about 2.5 MB (twilio.com, 2,523,347 bytes). Correction, 2026-11-18: this post first said "about 1 MB" for the largest file; the results CSV shows 2.5 MB.
Six domains returned HTTP 200 for /llms.txt with an HTML page (a redirect to another page, a branded not-found page or a bot challenge). We did not count those. This matters for audit tools, as the section below shows.
robots.txt: training crawlers and search crawlers are different decisions
The vendors document these crawlers separately. OpenAI says OAI-SearchBot and GPTBot settings are independent: a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot, which OpenAI describes as crawling content that may be used to train its foundation models. Anthropic's help article describes ClaudeBot as collecting web content that could contribute to training, and Perplexity's documentation describes PerplexityBot as surfacing sites in Perplexity search results and says it is not used to crawl for foundation models.
That is why the gap matters. 20 sites block GPTBot but only 10 block OAI-SearchBot. Of the 96 readable robots.txt files, 24 block at least one of GPTBot, ClaudeBot or PerplexityBot and 14 block all three. Most name the crawler in its own group: 18 of the 20 GPTBot blocks, 22 of 24 for ClaudeBot and 16 of 18 for PerplexityBot. The other 2 in each case come only from a User-agent: * rule with Disallow: /. We did not examine whether any block is deliberate.
One caution: robots.txt is a request. RFC 9309 states that its rules "are not a form of access authorization." A Disallow shows a site's stated preference, not that crawling is prevented.
Sitemaps, JSON-LD and Open Graph
85 sites declared a sitemap in robots.txt or served one at /sitemap.xml. Only 52 served a valid sitemap at that exact path, so a root-path check alone would under-count sites that use another location. JSON-LD appeared in the raw homepage HTML of 51 of 83 fetchable homepages; client-side-only injection would not show up in this method. Open Graph titles were the most common of the four signals at 73 of 83.
Where our own audit disagreed with the direct fetches
We called POST /api/free-audit once per domain. Every call returned HTTP 200, and scores ranged from 44 to 100 (median 93). Comparing it with the direct fetches turned up two problems:
- It could not read 27 of the 100 homepages. For those it reports "couldn't fetch the homepage" and skips structured-data and Open Graph checks. These are mostly large sites behind bot protection. 21 audits also reported
robots.txtorllms.txtas unverifiable. - It reported an
llms.txtfor 5 sites where we found none. For sendgrid.com, trello.com, booking.com, spotify.com and netflix.com the audit said anllms.txtexists but "needs more context". Our direct fetch returned an HTML page for each (HTTP 200 for four of them, a 404 page for booking.com). A check that trusts the HTTP status gets this wrong. It is a flaw in our audit, so until it is fixed treat anllms.txtaudit result as unconfirmed until you open the URL.
On ChatGPT-search crawler access the audit flagged 9 sites as blocked; the direct check found 11 for OAI-SearchBot or ChatGPT-User, and the two lists differ for 4 domains. That is why every figure in this post comes from the direct fetches.
What to do with this
- Open
https://yourdomain/robots.txtand read it. Decide separately about training crawlers (GPTBot, ClaudeBot) and search or user-fetch crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot). - Declare your sitemap in robots.txt.
- If you add an
llms.txt, check withcurl -ithat it returns a text file and not your homepage. - Check JSON-LD in the raw HTML (
curl -s https://yourdomain | grep ld+json), not in the browser inspector.
FAQ
Does blocking GPTBot keep me out of ChatGPT answers?
OpenAI documents GPTBot and OAI-SearchBot as independent settings and ties GPTBot to training. We did not test what happens to citations either way.
Does an llms.txt file help me get cited?
We don't know and did not test it. This study counts how many sites have one.
Is the sample representative of the web?
No. It is 100 well-known domains chosen by hand, skewed toward tech companies. Percentages describe this list only.
Why are some checks "unverifiable"?
Some sites return 401, 403 or 405 to automated requests. We retried once with a desktop browser user agent and, where it still failed, marked the check unverifiable instead of guessing.
Can I re-run this?
Yes. The domain list, the audit and direct-fetch scripts and the CSV are published with this post.
Methodology and data
- Date: 2026-11-17 (UTC). Audit requests ran from 09:38 to 11:20; direct fetches and their retry pass ran between 09:39 and 09:46.
- Domains: 100, fixed and hand-picked before running anything; none excluded afterwards.
- Audit:
POST https://seobolt.app/api/free-auditwith only{"domain": ...}, 5 requests then a 305-second pause to stay inside the documented limit of 5 per 5 minutes per IP. No request returned 429. This is the scoring version live after PR #167 ("honest audit scoring"); the audit code was unchanged between the live revisions522a570and1bdd25aseen during the run. - Direct fetches: GET of
/,/robots.txt,/llms.txt,/sitemap.xml, first asseobolt-research/1.0, then retried once with a desktop Chrome user agent for any answer other than 200, 404 or 410. 7 domains changed as a result. - Definitions: a valid
llms.txtis a 200 non-HTML non-empty response. Robots rules are evaluated with our own RFC 9309 implementation (specific group over*, longest match, Allow wins ties) for the path/only; partial blocks of sub-paths are not counted. We compared the results with the raw robots.txt of four sites by hand. - Limits: one day, one network location, hand-picked sample, JSON-LD and Open Graph read from raw HTML only, sitemaps checked at the root path only, 4 domains had no readable robots.txt.
- Data: results CSV, domain list and scripts.