How to measure how much of your crawl traffic is AI bots, from your own access logs (in Python)

Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com

Opinions about AI crawlers are everywhere. Numbers from your own site are rarer, and they're the only ones that should drive a robots.txt change.

The questions worth answering first:

What share of all bot requests come from AI-related crawlers?
How much bandwidth do they use?
Which sections of the site do they hit?
Are they training crawlers, search crawlers, or fetches triggered by a person asking an assistant to read your page?

Your access logs already have the answers. This post is a short Python script that classifies requests by user agent and reports exactly those numbers.

The categories

User agent strings change, and operators add new ones, so treat this as a starting list and check each operator's documentation before relying on it:

Search crawlers: Googlebot, bingbot, DuckDuckBot, YandexBot, Applebot (the plain one, used for Apple's search features).
AI training: GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent, Amazonbot, cohere-ai.
AI search: OAI-SearchBot, PerplexityBot, Claude-SearchBot.
AI user-triggered: ChatGPT-User, Claude-User, Perplexity-User.

Google-Extended and Applebot-Extended won't appear in logs: they're robots.txt tokens, not separate crawlers. Google's AI products fetch with the normal Googlebot.

Like Googlebot, all of these can be faked. A request claiming to be GPTBot from a residential IP is probably a scraper. The bigger operators publish their IP ranges; for numbers you'll act on, check requests against those ranges, the same way you'd verify Googlebot with reverse DNS.

The script

import gzip, glob, re, sys from collections import Counter, defaultdict CATEGORIES = { "search": ["Googlebot", "bingbot", "DuckDuckBot", "YandexBot", "Applebot/"], "ai-training": ["GPTBot", "ClaudeBot", "CCBot", "Bytespider", "meta-externalagent", "Amazonbot", "cohere-ai"], "ai-search": ["OAI-SearchBot", "PerplexityBot", "Claude-SearchBot"], "ai-user": ["ChatGPT-User", "Claude-User", "Perplexity-User"], } BOTLIKE = re.compile(r"bot|crawl|spider|slurp|fetch", re.I) # combined log format: ... "GET /path HTTP/1.1" 200 1234 "referer" "user agent" LINE = re.compile(r'"[A-Z]+ (\S+) [^"]*" (\d{3}) (\d+|-) "[^"]*" "([^"]*)"') def classify(ua): for cat, needles in CATEGORIES.items(): for n in needles: if n in ua: return cat, n if BOTLIKE.search(ua): return "other-bot", "other" return "human/other", "-" def lines(pattern): for p in sorted(glob.glob(pattern)): f = gzip.open(p, "rt", errors="replace") if p.endswith(".gz") else open(p, errors="replace") with f: yield from f reqs, bytes_, agents = Counter(), Counter(), Counter() paths = defaultdict(Counter) statuses = defaultdict(Counter) for line in lines(sys.argv[1]): m = LINE.search(line) if not m: continue path, status, size, ua = m.groups() cat, agent = classify(ua) reqs[cat] += 1 bytes_[cat] += 0 if size == "-" else int(size) agents[(cat, agent)] += 1 section = "/" + path.lstrip("/").split("/", 1)[0].split("?", 1)[0] paths[cat][section] += 1 statuses[cat][status[0] + "xx"] += 1 total_bot = sum(n for c, n in reqs.items() if c != "human/other") print(f"{'category':14s} {'requests':>10s} {'% of bots':>10s} {'MB':>10s} statuses") for cat, n in reqs.most_common(): share = "" if cat == "human/other" else f"{100 * n / max(total_bot, 1):.1f}%" print(f"{cat:14s} {n:10d} {share:>10s} {bytes_[cat] / 1e6:10.1f} {dict(statuses[cat])}") print("\nTop agents:") for (cat, agent), n in agents.most_common(15): if cat != "human/other": print(f"{n:8d} {cat:12s} {agent}") for cat in ("ai-training", "ai-search", "ai-user"): if paths[cat]: print(f"\nTop sections for {cat}:") for s, n in paths[cat].most_common(8): print(f"{n:8d} {s}")

Run it over a month of logs:

python3 ai_bots.py "/var/log/nginx/access.log*"

If you're behind a CDN, run it on the CDN's logs instead. Many bot requests are served from the edge cache and never reach your origin, so origin logs undercount.

Logs tell you what happened; a recurring check tells you when it changes. seotto.com flags indexing and crawler-access regressions so you don't have to re-run the script by hand.

Reading the numbers

The share of bot traffic

On many content sites, AI-related crawlers now make up a noticeable share of all bot requests, sometimes rivalling Googlebot. On a small site with little content, they may be a rounding error. Either way, this one number tells you whether the issue is worth more of your time.

Bandwidth versus requests

Compare the MB column with the request count. A crawler with many requests but small transfers is mostly hitting cached or small pages. One with large transfers per request may be downloading images, PDFs or large HTML pages. If server cost is your concern, bandwidth is the number to watch.

Status codes

A lot of 4xx for one agent usually means it's requesting URLs that don't exist: old links, guessed paths, or URLs from a stale dataset. A lot of 3xx means it's following your redirects. Neither is harmful, but both are wasted capacity on your side.

Sections

If training crawlers spend most of their requests on /blog and /docs, that's your published content being collected. If they're in /search, /tag or parameter URLs, they're stuck in the same crawl traps that waste Googlebot's time, and fixing those helps both.

User-triggered fetches

Requests from agents like ChatGPT-User are people asking an assistant about your page. The top paths here show which content people bring to AI assistants, which is useful product and content feedback in its own right.

Spot-check the claims with IP ranges

To check whether a busy "AI crawler" is genuine, take its top IP addresses from the logs and compare them with the operator's published ranges. OpenAI, for example, publishes JSON files of IP ranges for its crawlers, and other large operators document theirs. Python's ipaddress module makes the comparison a few lines: load the CIDR blocks, then test ipaddress.ip_address(ip) in network for each. Anything claiming to be a known crawler from outside its operator's ranges belongs in the "other-bot" bucket, and is often the real source of the load people blame on AI.

Turning it into a decision

With these numbers you can decide per category: allow, block in robots.txt, or rate limit at the CDN. Rerun the script a month after any change to confirm the crawlers you blocked actually stopped, and that search crawlers were unaffected. At seotto.com we like decisions like this to start from a log query rather than a headline.