Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com
You shipped a big release last Tuesday: a new navigation, a router upgrade, maybe a URL change in one section. A week later, someone asks whether it affected SEO.
Search Console's crawl stats report gives you a sitewide graph, a few days delayed, with limited breakdowns. What you actually want to know is more specific: did Googlebot crawl any section of the site less after the deploy, and what was it getting instead?
Your access logs have the answer, with every request Googlebot made, to the second. This script compares the days before a cutoff with the days after, per section, and shows where crawling dropped and which status codes changed.
What the script does
- Reads Nginx or Apache combined-format logs (plain or gzipped).
- Keeps only requests from Googlebot's user agent.
- Groups URLs into sections by their first path segment (
/blog/,/docs/,/products/). - Counts requests per day, per section and status class, before and after a cutoff time.
- Prints sections whose daily crawl rate changed the most, with the status mix for each period.
The script
#!/usr/bin/env python3 """Compare Googlebot crawling per section before and after a cutoff.""" import gzip, re, sys from collections import Counter, defaultdict from datetime import datetime, timezone LINE = re.compile(r'\[(?P<ts>[^\]]+)\] "(?:GET|HEAD) (?P<path>\S+) [^"]*" (?P<status>\d{3}) .*"(?P<ua>[^"]*)"$') cutoff = datetime.fromisoformat(sys.argv[1]).astimezone(timezone.utc) files = sys.argv[2:] counts = defaultdict(Counter) # (period, section) -> Counter(status class) days = {"before": set(), "after": set()} def section(path): path = path.split("?", 1)[0] parts = [p for p in path.split("/") if p] return "/" + parts[0] + "/" if parts else "/" for f in files: opener = gzip.open if f.endswith(".gz") else open with opener(f, "rt", errors="replace") as fh: for line in fh: m = LINE.search(line.rstrip()) if not m or "Googlebot" not in m["ua"]: continue ts = datetime.strptime(m["ts"], "%d/%b/%Y:%H:%M:%S %z").astimezone(timezone.utc) period = "before" if ts < cutoff else "after" days[period].add(ts.date()) counts[(period, section(m["path"]))][m["status"][0] + "xx"] += 1 nb, na = max(len(days["before"]), 1), max(len(days["after"]), 1) rows = [] for sec in {s for _, s in counts}: b, a = counts[("before", sec)], counts[("after", sec)] rb, ra = sum(b.values()) / nb, sum(a.values()) / na rows.append((ra - rb, sec, rb, ra, b, a)) print(f"{nb} day(s) before, {na} day(s) after {cutoff:%Y-%m-%d %H:%M} UTC") print(f"{'section':<24}{'before/day':>11}{'after/day':>11} status mix before -> after") for delta, sec, rb, ra, b, a in sorted(rows)[:25]: mix = lambda c: " ".join(f"{k}:{v}" for k, v in sorted(c.items())) or "-" print(f"{sec:<24}{rb:>11.1f}{ra:>11.1f} {mix(b)} -> {mix(a)}") Run it with the deploy time and the logs covering a week either side:
python3 crawl_diff.py 2026-11-10T14:00+02:00 /var/log/nginx/access.log* 7 day(s) before, 6 day(s) after 2026-11-10 12:00 UTC section before/day after/day status mix before -> after /docs/ 412.3 96.5 2xx:2801 3xx:85 -> 2xx:301 3xx:212 4xx:66 /products/ 188.0 141.2 2xx:1316 -> 2xx:802 5xx:45 /blog/ 120.4 131.0 2xx:843 -> 2xx:786 Sections are sorted by the biggest drop first. In this example, /docs/ lost most of its crawling, and what's left is now mostly redirects and 404s, which points straight at a URL change that wasn't fully redirected. /products/ has a new block of 5xx responses worth investigating.
Reading the results
Crawl rate down, status mix unchanged. Googlebot is simply asking for fewer URLs in that section. Often that means fewer links to them: the new navigation dropped the section, or internal links were removed. Check whether the section is still linked from the main menu and the sitemap.
More 3xx after the deploy. URLs changed and redirects were added. That's fine for a while, but crawl rate on the new URLs should pick up over the following weeks. If it doesn't, check that internal links and the sitemap point at the new URLs directly, not at the redirects.
New 4xx. URLs Googlebot knew about now return 404. Find the top ones and decide: redirect, restore, or let them go. Use the full paths for that:
zcat -f /var/log/nginx/access.log* | grep Googlebot | awk '$9 ~ /^4/ {print $7}' | sort | uniq -c | sort -rn | head New 5xx. These are the most urgent. Repeated server errors make Googlebot slow down crawling for the whole host, not just the failing section.
Logs tell you what happened after a deploy; the same check before the next deploy is better. seotto.com watches for indexing blocks and SEO regressions as you ship, so a section that started returning redirects or errors to crawlers gets flagged before its crawl rate falls.
Caveats
- User agents can be faked. For a quick comparison it's fine, since fake Googlebots are usually a small, steady share. For anything precise, verify IPs with reverse DNS or Google's published ranges first.
- Crawling varies naturally. Day-to-day swings of 20 to 30% are normal. Compare at least a week on each side, and look for big changes, not small ones.
- Weekly patterns. If possible, compare the same weekdays on both sides of the cutoff.
- Logs behind a CDN. If a CDN serves cached responses, your origin logs only show misses. Use the CDN's logs if you can, or the comparison will under-count.
Make it routine
Run the script a week after every significant release, and keep the output with the release notes. After a few releases you'll know what "normal" looks like for your site, and an unusual drop will stand out immediately.
Related reading
- How to fix 'Excluded by noindex tag' in Search Console when you're sure you never added noindex
- How to read your server logs like Googlebot's diary (with a short Python script)
- How long until Googlebot sees your change? Measure recrawl latency from your own access logs
- How to find the URLs Googlebot crawls most but that never get search traffic (access logs + Search Console, in Python)
- How to measure how much of your crawl traffic is AI bots, from your own access logs (in Python)
Do you check crawl behaviour after releases? I'm curious whether anyone has automated this as part of a deploy pipeline.