Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com
You submit https://www.example.com/sitemap.xml in Search Console, and the status column says Couldn't fetch. Or, a little later, Sitemap could not be read. You click the URL, it opens fine in your browser, and you're left wondering what Google is seeing that you aren't.
First, some reassurance: "Couldn't fetch" right after submitting is sometimes just Search Console's status not having caught up. Google may not have attempted the fetch yet. If the sitemap is new, wait a day before debugging.
If it's still there after a day or two, here's how to find the actual cause.
Step 1: fetch it the way Googlebot does
Your browser has cookies, a normal user agent and probably a session that passed a bot challenge hours ago. Googlebot has none of those. Request the sitemap with curl, using Googlebot's user agent, from a machine outside your network:
curl -sSL -D - -o /tmp/sitemap.xml \ -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \ https://www.example.com/sitemap.xml Look at the headers it prints. You want:
- A final status of
200, ideally with no redirects before it. - A
Content-Typeofapplication/xmlortext/xml(or a gzip type for.xml.gz). - A body that starts with
<?xmland contains<urlsetor<sitemapindex.
Then validate the XML itself:
xmllint --noout /tmp/sitemap.xml && echo "well-formed" Most problems show up in one of these three checks.
The usual causes
1. A firewall or bot protection is blocking Googlebot. A WAF rule, a "bot fight" mode or a rate limit may let browsers through and challenge everything else. If curl gets a 403, a 429, or an HTML challenge page instead of XML, that's it. Check your WAF logs for requests to /sitemap.xml with Googlebot's user agent. Allow verified Googlebot traffic rather than whitelisting the user agent string, which anyone can fake.
2. Robots.txt blocks the sitemap. It sounds silly, but a broad Disallow: /*.xml$ or a rule that blocks a /sitemaps/ folder will stop Google fetching the file. Check with Search Console's robots.txt report, or simply read the rules that match the sitemap's path.
3. Redirects. A sitemap at http:// that redirects to https://, or example.com to www.example.com, usually works, but submit the final URL anyway. Long chains, redirects to a login page, or a redirect to an HTML page are common causes of "could not be read".
4. It's not actually XML. A framework error page, a "maintenance" page or a soft 404 served with status 200 looks like a successful fetch but isn't a sitemap. So does a sitemap route that renders through the site's HTML layout by mistake, wrapping the XML in <html>. The body check above catches all of these.
5. Malformed XML. Unescaped ampersands in URLs are the classic: ?a=1&b=2 must be written as ?a=1&b=2 inside XML. Also watch for invalid characters copied from content, a byte order mark or whitespace before <?xml, and the wrong namespace on <urlset> (it must be http://www.sitemaps.org/schemas/sitemap/0.9).
6. Gzip confusion. A .xml.gz file served with Content-Encoding: gzip and gzipped content is double-compressed from the client's point of view. Either serve a gzipped file as a download (Content-Type: application/gzip, no Content-Encoding), or serve plain XML and let the server compress the transfer.
7. Too big. A single sitemap may contain at most 50,000 URLs and be at most 50 MB uncompressed. Bigger than that, split it and use a sitemap index.
8. It's slow or times out. Sitemaps generated on the fly from the database can take many seconds on big sites, especially uncached. Googlebot won't wait forever. Generate sitemaps ahead of time, or cache them aggressively.
A sitemap that worked last week can start failing after a WAF rule or a framework upgrade, and Search Console takes days to tell you. seotto.com checks for crawler-facing problems like blocked or broken sitemaps after you deploy, so you find out the same day.
Step 2: check the URLs inside
"Could not be read" is about the file. But once Google can read it, the URLs inside matter too. A quick check that every listed URL returns 200 and isn't redirected or noindexed:
grep -oE '<loc>[^<]+</loc>' /tmp/sitemap.xml | sed -E 's/<\/?loc>//g' | head -200 \ | while read -r u; do printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' "$u")" "$u"; done \ | grep -v '^200' Every URL in a sitemap should be the canonical, indexable, 200 version. A sitemap full of redirects and 404s doesn't break the fetch, but it teaches Google that your sitemap isn't reliable.
Step 3: resubmit and wait
Once the file passes all the curl checks, resubmit it in Search Console. The status usually updates within a day or two. If it doesn't, use the URL Inspection tool on the sitemap URL itself: the live test shows exactly what Googlebot received, including the HTTP status.
Sitemap index files
If you use a sitemap index (a sitemap of sitemaps), every child sitemap has to pass the same checks. A common failure is an index that lists child sitemaps on a different host or protocol (http:// children in an https:// index, or a CDN hostname), or children that were renamed in a deploy while the index still points at the old names. Fetch each <loc> in the index the same way you fetched the index itself.
Keeping it fixed
- Reference the sitemap in robots.txt (
Sitemap: https://www.example.com/sitemap.xml), so crawlers find it even if the Search Console submission is lost. - Test the sitemap in CI: well-formed XML, right content type, URL count under the limit.
- Check it from outside after deploys, with a crawler user agent, since WAF and CDN rules only exist in production. That's the part a tool like seotto.com automates.
Related reading
- How to check if robots.txt is blocking Google (and fix it without breaking anything)
- Postmortem (hypothetical): our robots.txt returned 503 for nine days, and crawling nearly stopped
- How to fix 'Discovered – currently not indexed' in Google Search Console (a developer's list of causes)
- How to fix 'Blocked due to access forbidden (403)' in Google Search Console (it's usually your WAF or bot rules)
- Should you noindex your tag and category pages? Let's argue about it
What was the cause the last time your sitemap showed "Couldn't fetch"? I've seen the ampersand one more times than I'd like to admit.

