A broken link checker looks at status codes. A soft 404 hides from it, because the status code says everything is fine. The server answers 200, and the page says "Sorry, this product is no longer available," or shows the site's not-found template, or is nearly blank. Search Console reports these as soft 404s and, per Google, "Such pages are excluded from Search."

This post builds a small auditor in Python using only the standard library (urllib, urllib.robotparser, xml.etree, html.parser, difflib, uuid, csv, concurrent.futures). It reads your sitemap or sitemap index, first requests a URL that cannot exist to see how the server handles missing pages, then fetches every sitemap URL and flags:

  • 200 responses whose title, H1 or opening text carry an error message
  • 200 responses whose text matches the site's own not-found page
  • removed pages that redirect to the home page instead of a replacement
  • nearly empty pages
  • internal search result URLs listed in the sitemap
  • real 404 and 410 URLs still listed in the sitemap, and redirected URLs the sitemap should list by their final address

Google Search Central's Troubleshoot Google Search crawling errors page (showing "Last updated 2026-11-16 UTC") and its HTTP status codes and network errors page (showing "Last updated 2026-02-04 UTC") were opened on 21 Nov 2026, and every statement attributed to Google below comes from those pages. The output in this post comes from a small fictional shop ("Larkspur Outdoor Supply") served on localhost, built for this article with planted problems. Appie free-tool descriptions are as of 21 Nov 2026, checked on each tool's page that day.

Disclosure: Appie.run publishes this article. Two free tools mentioned near the end are ours. The script does not depend on them.

Where this fits

This is another script in a small standard-library series that reads a sitemap and checks one thing per page. An earlier post on finding broken links and redirect chains catches the URLs that fail loudly with a 4xx or 5xx. This one catches the URLs that fail quietly with a 200.

What Google actually says

Definition. "A soft 404 error is when a URL that returns a page telling the user that the page does not exist and also a 200 (success) status code. In some cases, it might be a page with no main content or empty page."

Common causes Google lists. "A missing server-side include file. A broken connection to the database. An empty internal search result page. An unloaded or otherwise missing JavaScript file."

What happens. On the status codes page: "If the content suggests an error for Google Search, an empty page or an error message, Search Console will show a soft 404 error."

How to fix it, by situation.

  • Page gone, no replacement: "return a 404 (not found) or 410 (gone) response (status) code for the page."
  • Page moved or replaced: "If your page has moved or has a clear replacement on your site, return a 301 (permanent redirect)."
  • Page still exists but was flagged: it is "likely it didn't load properly for Googlebot, it was missing critical resources, or it displayed a prominent error message during rendering." Google points you to the URL Inspection tool.

Two things the pages do not say, so treat them as my choices: Google does not publish the phrases or word counts its systems use, and it does not say a redirect to the home page is a soft 404. I flag home page redirects because the home page is rarely "a clear replacement" for a removed product or post.

How the audit works

  1. Read the sitemap (or every child sitemap in an index) and collect each <loc>.
  2. Read robots.txt and skip disallowed URLs.
  3. Request a random URL like /soft404-probe-3f9a1c2b7d4e. If the server answers 200, every missing URL on the site is a soft 404. Whatever it returns, keep its text as the site's not-found template.
  4. Fetch each sitemap URL with a redirect handler that records the hops.
  5. Extract the title, the H1 and the main text, skipping nav, header, footer, script and style, so menus do not count as content.
  6. Compare the page with the not-found template using difflib.SequenceMatcher, look for error phrases in the title, H1 and the first 600 characters of text, and count words.
  7. Write a CSV with one row per URL and print the flagged ones by severity.

Only the title, H1 and opening text are searched for error phrases. Searching the whole page would flag a blog post that happens to explain what a 404 is.

What the script flags

Issue Severity Meaning
matches_not_found_page high 200 response with 90%+ the same text as the probe's not-found page
error_message_200 high 200 response whose title, H1 or opening text says "not found", "no longer available", "no results found" and similar
redirect_to_home high A non-home URL that ends up on /
server_error / fetch_error high 5xx, or no response at all
nearly_empty medium Under 50 words of main text and no error phrase
search_url_in_sitemap medium A URL with q=, s= or search= in the query string
gone_in_sitemap medium 404 or 410: correct for a removed page, but it should leave the sitemap
redirected low Redirects somewhere other than home; list the final URL instead

The site-wide probe result is printed at the top rather than per row, since it describes the server, not one page.

The script

Save as soft404_audit.py. Python 3.9 or newer, no packages to install.

#!/usr/bin/env python3 """Find soft 404s in a sitemap, standard library only. Usage: python3 soft404_audit.py https://example.com/sitemap.xml [--out soft404_report.csv] """ import argparse, csv, difflib, gzip, re, sys, uuid import urllib.error, urllib.request, urllib.robotparser import xml.etree.ElementTree as ET from concurrent.futures import ThreadPoolExecutor from html.parser import HTMLParser from urllib.parse import urljoin, urlsplit UA = "soft404-audit/1.0 (+standard library script)" NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}" MIN_WORDS = 50 # main text shorter than this is "nearly empty" TEMPLATE_MATCH = 0.90 # similarity to the site's not-found page ERROR_PHRASES = [ "page not found", "not found", "can't find that page", "cannot find that page", "doesn't exist", "does not exist", "no longer available", "no results found", "0 results", "has been removed", "error establishing a database connection", "something went wrong", "internal server error", ] class Redirects(urllib.request.HTTPRedirectHandler): """Follow redirects but remember each hop.""" def redirect_request(self, req, fp, code, msg, headers, newurl): new = super().redirect_request(req, fp, code, msg, headers, newurl) if new is not None: new.hops = getattr(req, "hops", []) + [(code, newurl)] return new OPENER = urllib.request.build_opener(Redirects) def fetch(url, timeout=20): req = urllib.request.Request(url, headers={"User-Agent": UA}) req.hops = [] try: resp = OPENER.open(req, timeout=timeout) status, final, body = resp.status, resp.geturl(), resp.read(2_000_000) hops = getattr(resp, "hops", None) except urllib.error.HTTPError as e: status, final, body, hops = e.code, e.geturl() or url, e.read(2_000_000), None except Exception as e: # DNS, TLS, timeout return {"status": 0, "final": url, "body": "", "error": str(e)[:120], "redirected": False} if body[:2] == b"\x1f\x8b": body = gzip.decompress(body) return {"status": status, "final": final, "body": body.decode("utf-8", "replace"), "error": "", "redirected": final.rstrip("/") != url.rstrip("/")} class TextParser(HTMLParser): SKIP = {"script", "style", "noscript", "nav", "header", "footer", "template", "svg"} def __init__(self): super().__init__() self.title, self.h1, self.text = "", "", [] self._stack, self._in_title, self._in_h1 = [], False, False def handle_starttag(self, tag, attrs): if tag in self.SKIP: self._stack.append(tag) self._in_title |= tag == "title" self._in_h1 |= tag == "h1" def handle_endtag(self, tag): if self._stack and self._stack[-1] == tag: self._stack.pop() if tag == "title": self._in_title = False if tag == "h1": self._in_h1 = False def handle_data(self, data): if self._in_title: self.title += data elif not self._stack: if self._in_h1: self.h1 += data self.text.append(data) def parse(html): p = TextParser() p.feed(html) text = re.sub(r"\s+", " ", " ".join(p.text)).strip() return p.title.strip(), p.h1.strip(), text def read_sitemap(url, seen=None): seen = seen or set() if url in seen: return [] seen.add(url) r = fetch(url) if r["status"] != 200: sys.exit(f"Could not read sitemap {url}: HTTP {r['status']} {r['error']}") root = ET.fromstring(r["body"].encode()) locs = [e.text.strip() for e in root.iter(NS + "loc") if e.text] if root.tag == NS + "sitemapindex": out = [] for child in locs: out += read_sitemap(child, seen) return out return locs def probe_not_found(site_root): """Request a URL that cannot exist and see what the server does.""" url = urljoin(site_root, f"/soft404-probe-{uuid.uuid4().hex[:12]}") r = fetch(url) _, _, text = parse(r["body"]) if r["body"] else ("", "", "") return url, r["status"], text def check(url, robots, template_text): row = {"url": url, "status": "", "final_url": "", "title": "", "words": 0, "issue": "ok", "severity": "", "detail": ""} if robots and not robots.can_fetch(UA, url): row.update(issue="robots_blocked", severity="info", detail="Disallowed in robots.txt; not fetched") return row r = fetch(url) row.update(status=r["status"], final_url=r["final"]) if r["status"] == 0: row.update(issue="fetch_error", severity="high", detail=r["error"]) return row title, h1, text = parse(r["body"]) if r["body"] else ("", "", "") row.update(title=title[:80], words=len(text.split())) issues = [] if r["status"] in (404, 410): issues.append(("gone_in_sitemap", "medium", f"Returns {r['status']}; remove it from the sitemap")) elif r["status"] >= 500: issues.append(("server_error", "high", f"Returns {r['status']}")) elif r["status"] >= 400: issues.append(("client_error", "medium", f"Returns {r['status']}")) if r["redirected"]: if urlsplit(r["final"]).path in ("", "/") and urlsplit(url).path not in ("", "/"): issues.append(("redirect_to_home", "high", "Redirects to the home page, not to a replacement")) else: issues.append(("redirected", "low", f"Sitemap should list the final URL: {r['final']}")) if r["status"] == 200: haystack = f"{title} {h1} {text[:600]}".lower() hits = [p for p in ERROR_PHRASES if p in haystack] if template_text and text and difflib.SequenceMatcher(None, text, template_text).ratio() >= TEMPLATE_MATCH: issues.append(("matches_not_found_page", "high", "200 response with the same text as the site's not-found page")) elif hits: issues.append(("error_message_200", "high", f"200 response that says: {hits[0]!r}")) if row["words"] < MIN_WORDS and not hits: issues.append(("nearly_empty", "medium", f"Only {row['words']} words of main text")) if urlsplit(url).query and any(k in urlsplit(url).query for k in ("q=", "s=", "search=")): issues.append(("search_url_in_sitemap", "medium", "Internal search result URL listed in the sitemap")) if issues: order = {"high": 0, "medium": 1, "low": 2, "info": 3} issues.sort(key=lambda i: order[i[1]]) row.update(issue="; ".join(i[0] for i in issues), severity=issues[0][1], detail=" | ".join(i[2] for i in issues)) return row def main(): ap = argparse.ArgumentParser() ap.add_argument("sitemap") ap.add_argument("--out", default="soft404_report.csv") ap.add_argument("--workers", type=int, default=8) a = ap.parse_args() parts = urlsplit(a.sitemap) home = f"{parts.scheme}://{parts.netloc}/" robots = urllib.robotparser.RobotFileParser(urljoin(home, "/robots.txt")) try: robots.read() except Exception: robots = None urls = read_sitemap(a.sitemap) probe_url, probe_status, template_text = probe_not_found(home) print(f"Sitemap URLs: {len(urls)}") print(f"Not-found probe: {probe_url.replace(home, '/')} -> HTTP {probe_status}") if probe_status == 200: print(" WARNING: the server answers 200 for a URL that does not exist (site-wide soft 404).") with ThreadPoolExecutor(max_workers=a.workers) as pool: rows = list(pool.map(lambda u: check(u, robots, template_text), urls)) with open(a.out, "w", newline=Source:https://dev.to/appierun_66/find-soft-404-pages-in-your-sitemap-with-python-standard-library-only-error-messages-on-200-3cjg

Related posts