Frontend
Our site served every URL the same 3,780 bytes, and Google believed it
Thedolceway DEV Community
2 views
Checked with a Googlebot user agent one morning: every single URL on our site returned the
same 3,780-byte shell. Same <title>, zero <h1>, zero body text. The homepage, a blog
post and a product page were byte-identical before JavaScript ran.
Search Console agreed with the crawler rather than with us. Of 741 URLs, 116 had earned a
single impression in 28 days, and a landing page that had been live for five months was
still reported as "URL is unknown to Google".
Here is what I actually learned fixing it, including the two things that cost us the most
time.
Google does render JavaScript. That is not the point.
The standard reply to this problem is "Googlebot executes JS now, you are fine." It does.
Several of our pages were indexed, so rendering clearly happened.
But rendering is a separate, budgeted queue. A domain with little authority does not
get much of that budget. So the practical question is not "can Google render our page", it
is "will Google spend its budget rendering this page, today, before it decides what the
page is about".
There is a second problem that has nothing to do with rendering: 741 URLs that are
byte-identical before render look like duplicates. You are handing a duplicate-content
signal to the crawler and hoping the render queue fixes your first impression.
What we built, and what we deliberately did not
We wrote a post-build script that injects a real <head> into each generated HTML file:
title, description, canonical, robots, Open Graph, Twitter.
Head only. The body stayed exactly as the SPA served it. That was deliberate:
No hydration flash.
No risk of a static copy drifting out of sync with what users see.
Nothing that could be read as cloaking, because the static markup is a subset of the
rendered markup, not a different page.
Every value is read from the same source the React page reads. Where a title is a literal
inside a component, the script extracts it from that component's source rather than having
anyone retype it. A number retyped in two places is a number that will disagree with itself
eventually.
Trap 1: react-helmet-async deletes tags you did not mark
Our shell had static <meta name="description"> in index.html. Helmet sets its own on
mount. The assumption was that the later one wins.
What actually happens: on its first commit, Helmet removes every tag carrying data-rh
and re-inserts its own. Tags without that attribute survive. So the static tag stayed,
Helmet's tag was added, and the page went out with two descriptions, with the static one
first in the head, which is the one scrapers and crawlers read.
Every blog post on the site was advertising the generic homepage copy.
The fix is to mark the tags Helmet actually emits:
<meta data-rh="true" name="description" content="..." />
With one sharp edge: only mark tags Helmet will re-add. A data-rh tag that Helmet does
not re-insert gets deleted on mount and never comes back. We marked og:image and lost our
share image everywhere until we worked out why.
The matching bug in our own script: the replace was written as a regex keyed on attribute
order, <meta name="description" ...>. The shell writes <meta data-rh="true"
name="description" ...>. It silently matched nothing. Match on a tag containing the
key, never on attribute order.
Trap 2: the script read the sitemap, and nothing regenerated the sitemap
This one cost the most time and had the least to do with SEO.
The prerender script takes its URL list from sitemap.xml. Reasonable: one source of truth
for what exists.
But npm run build did not include the sitemap generator. It ran vite, then redirects,
then the crawl index, then prerender. The sitemap was generated by a separate command
somebody had to remember to run.
So we added a new route, built, deployed, and the page was live and completely absent from
the prerendered set. No error. No warning. The build exited 0. The route simply was not in
the list, so it was never processed, and it went out with the generic shell head.
We burned two builds assuming the code was wrong before checking whether the input was
stale.
If a build step consumes a generated artifact, that artifact's generator belongs in the
same build command. Otherwise you have an ordering dependency that lives only in
somebody's memory, and it fails silently rather than loudly.
Trap 3: a dead ternary that read as intentional
We found this while adding static content for a specific route:
const key = m ? m[1] : route === '/some-route' ? null : null;
Both branches return null. It was presumably mid-refactor when someone got interrupted.
It looks purposeful enough to skim past in review: there is a route name in it, so it
reads like a handled special case.
The effect was that the route fell through to a generic fallback body, which contained
none of the terms the page existed to rank for. The page had been quietly pointless for
weeks.
Grep your codebase for ? null : null. I will wait.
What I would tell myself at the start
Verify with curl and a crawler user agent, not with your browser. Your browser runs
the JavaScript. That is exactly the thing you are trying to see past.
Static head, dynamic body, always from one source. Two hand-maintained copies of the
same string are a future contradiction.
When something silently does nothing, check the input before you debug the logic.
data-rh is not decoration. It decides which tags survive.
If you want to see the output, it is running on CVBooster, a
resume builder I run. Every page announces what it is before a line of JavaScript
executes, which is all this work was ever about.
Postscript, September 2026: head only was not enough
The approach above held for the pages the prerender step knew about. It did nothing for
the ones it did not. Every URL without a prerendered file fell through to the SPA fallback,
and on Cloudflare Pages that fallback was a copy of the homepage, complete with a canonical
tag pointing at /. Google indexed that fallback for 309 URLs and filed them under
"alternate page with proper canonical", which is the polite way of saying it believed the
tag and threw the pages away.
Three changes made it hold:
Render the body too. A server-side render step in the build writes the real page
content into each file, so the static markup is no longer a subset of the page but the
page itself.
Send the fallback to a noindex shell, never to a homepage clone. An unknown URL
should look like nothing, not like the front door.
Gate the build. A last step reads dist the way a crawler does and fails the build if
any sitemap URL lacks a static file, a self-referencing canonical or 150 words of text.
Then measure from outside, every deploy: fetch every sitemap URL with a Googlebot user agent
and count the failures. The only acceptable number is zero.
Read original: https://dev.to/thedolceway/our-site-served-every-url-the-same-3780-bytes-and-google-believed-it-1d9m
← Previous
What a Kubernetes controller actually does when you break something
Next →
Our regex found 199 records in a 1,723-record corpus and reported no errors
Related
Our regex found 199 records in a 1,723-record corpus and reported no errors
Frontend
1
DEV Community
How JavaScript Can Limit AI Crawler Access to Your Website Content
Frontend
3
Dev.to (EN Zone)
Measure a context plugin on your own repository before you install it
Frontend
2
Dev.to (EN Zone)
Zero-Budget Web Dev: Moving from Discord/Drive to Google Sites
Frontend
2
DEV Community
Comments0
No comments yet — be the first