AI companies read your site for three different reasons, and robots.txt lets you treat each one differently. Most "block AI bots" advice lumps them together, which can quietly remove you from AI answers you'd want to be in.

Three kinds of AI bots

  • Training crawlers collect pages to train models: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini), Applebot-Extended, meta-externalagent, Amazonbot, CCBot (Common Crawl) and Bytespider (ByteDance).
  • AI search crawlers index pages so assistants can cite you: OAI-SearchBot, Claude-SearchBot and PerplexityBot.
  • User-triggered fetchers open one page because someone asked about it: ChatGPT-User, Claude-User and Perplexity-User.

Google-Extended and Applebot-Extended aren't crawlers at all. They're labels Google and Apple check before using pages their normal bots already fetched for AI training.

The policy most small sites want

Block training, keep AI search:

User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot User-agent: CCBot User-agent: Bytespider Disallow: / User-agent: * Disallow: 

One trap: a bot that finds a group with its own name ignores your User-agent: * rules. If you hide /admin/ in the * group and then add a group for a named bot, repeat /admin/ there.

Google works differently

Blocking Google-Extended does not remove you from Google Search, AI Overviews or AI Mode. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". To limit what AI Overviews show from a page, Google points to nosnippet, max-snippet and noindex. Blocking Googlebot removes you from Google entirely, so never do it by accident.

What robots.txt can't do

It's a request, not a lock. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules", because a person asked for the page. Anthropic says all its bots respect it. For scrapers that ignore the file, you need your host's or CDN's bot blocking. And llms.txt blocks nothing.

What big sites do

On November 18, 2026, we checked the robots.txt files of 16 well-known sites. The New York Times, the BBC, Reddit and Amazon block every AI bot we tested. Meta's and ByteDance's crawlers were fully blocked on 12 sites and ClaudeBot on 11, but Google-Extended on only 6. Three Indian news sites, Times of India, Hindustan Times and Mint, fully blocked only Meta's crawler.

If you run a blog, a tool site or a small business, blocking training while keeping AI search is usually the best trade: you stay the source assistants link to, without donating your writing to the next model.

Sources

This is the short version. The full guide, with every step and source, is on GeekSpark.