Our pricing page renders per request, because it reads the country header to price in the visitor's own currency. I have written about that resolution before. What I had never checked is how many distinct documents that URL can produce, so I fetched it as 20 different clients and diffed the results.

The answer is four, and the reason is that two separate regular expressions are looking at the same User-Agent string for two unrelated purposes.

Regex one: ours, 12 terms, picks the currency

It lives in lib/http/bots.ts and it is the whole of our bot detection:

export const BOT_PATTERN = /vercel|bot|crawler|spider|googlebot|bingbot|slackbot|twitterbot|facebookexternalhit|linkedinbot|whatsapp|headless/i 

A match means the request is quoted the base currency, GBP, rather than the currency of whatever datacentre the request came from. The reason is indexing: the plans are defined in pence, the structured data on the page is in GBP, and a crawler geolocated to Virginia indexing dollar prices would disagree with both.

Twelve alternatives, of which five can never match anything on their own: googlebot, bingbot, slackbot, twitterbot and linkedinbot all contain bot, which is already an alternative. They document intent rather than adding coverage. The three that earn their place are crawler, spider and facebookexternalhit, plus two that do something slightly startling, which I will come back to.

Regex two: Next.js's, 27 terms, picks the shape of the HTML

This one I did not write and had never read. Next 16.2.7 ships it at next/dist/shared/lib/router/utils/html-bots.js:

const HTML_LIMITED_BOT_UA_RE = /[\w-]+-Google|Google-[\w-]+|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti|googleweblight/i 

Its own comment explains the purpose: these are clients that "can't safely stream the response due to how they parse the DOM", because they may check for metadata in the head before the stream has finished writing it. A match means Next does a blocking render instead of a streamed one. A sibling regex handles the one crawler that runs a real browser:

const HEADLESS_BROWSER_BOT_UA_RE = /Googlebot(?!-)|Googlebot$/i; 

Note the negative lookahead. Googlebot is a DOM bot and gets the stream. AdsBot-Google is not, and does not.

Twenty-seven alternatives, 25 of them literal names. Our regex matches 12 of those 25 and misses 13: Chrome-Lighthouse, Slurp, yandex, sogou, tumblr, vkShare, quora link preview, ia_archiver, BingPreview, facebookcatalog, SkypeUriPreview, Yeti and googleweblight. Every one of those is a client Next treats as a crawler and we treat as a customer.

The matrix

Twenty fetches of https://pub-trivia.app/pricing from one laptop in central Europe. CCY is the currency in the rendered price, SHELL is whether the response carries React streaming markers, BYTES is the whole document.

UA BYTES CCY SHELL FIRST PRICE Googlebot 101993 GBP streamed £30 HeadlessChrome 101994 GBP streamed £30 AdsBot-Google 101705 GBP inline £30 YandexBot 101705 GBP inline £30 bingbot 101705 GBP inline £30 facebookexternalhit 101705 GBP inline £30 Slackbot 101705 GBP inline £30 Twitterbot 101741 GBP inline £30 WhatsApp 101705 GBP inline £30 Applebot 101705 GBP inline £30 Mediapartners-Google 101713 EUR inline €36.99 Google-InspectionTool 101713 EUR inline €36.99 Chrome-Lighthouse 101713 EUR inline €36.99 Slurp 101713 EUR inline €36.99 quora-link-preview 101713 EUR inline €36.99 SkypeUriPreview 101712 EUR inline €36.99 ia_archiver 101713 EUR inline €36.99 vkShare 101713 EUR inline €36.99 my-own-scraper 102002 EUR streamed €36.99 Chrome 102003 EUR streamed €36.99 

Ten of the 20 are quoted pounds and ten euros, and the split has nothing to do with which of them is really a crawler. Four combinations, all of them populated:

  • GBP, streamed. Googlebot, and also headless Chrome, because headless is one of our 12 terms. A colleague automating a check in Puppeteer sees a different price from the same script run headful. This is the cheapest way for a reader to reproduce the whole post.
  • GBP, blocking. The link preview fetchers and the non-Google search crawlers. Both regexes fire.
  • EUR, blocking. Next thinks you are a crawler, we think you are a shopper.
  • EUR, streamed. Anything with a plain user agent, including a browser and including a scraper that names itself honestly.

The third row is the one worth staring at. Google-InspectionTool is the user agent behind the URL Inspection tool in Search Console, the thing you reach for when you want to see what Google sees. It is not matched by our regex, so it is quoted euros, while Googlebot a second later is quoted pounds. The tool for checking the crawler's view of the page is the one client guaranteed not to get it. Same for Mediapartners-Google, and for Chrome-Lighthouse, which means a PageSpeed Insights run measures a document that no indexing crawler will ever be served.

What "blocking" actually changes

I expected the blocking render to put the <title> earlier in the byte stream. It does not:

$ python3 -c "print(open('bing.html').read().find('<title>'))" 2730 $ python3 -c "print(open('chrome.html').read().find('<title>'))" 2730 

Identical offset. Both documents carry the full metadata block inside <head>, in the same place. What the streamed version adds is React's suspense bookkeeping:

bingbot: bytes=101703 suspense markers=0 hidden divs=0 __next_f.push=18 chrome: bytes=102002 suspense markers=2 hidden divs=1 __next_f.push=18 

One <div hidden> wrapper, two <!--$--> boundary comments, and a flight payload that references the metadata as a late chunk instead of writing it in place. That is the whole 299 byte difference, and it is 0.3 per cent of the document.

The timing cost is not measurable here either. Ten fetches each, time to first byte:

bingbot median 235 ms chrome median 234 ms 

Which makes sense: this page has nothing to wait for. The data it renders is two integers and a header read, so there is no slow boundary for the stream to defer, and the blocking path gives up nothing. On a page that awaited something slow, the same mechanism would be the difference between a crawler seeing a shell and a crawler seeing content, which is presumably why the framework ships the list rather than a config flag.

The part I got wrong first

My first pass grouped the 20 responses by byte count, saw three clusters, and concluded that the small cluster was a caching artefact, because nothing in our code branches on anything except currency. Then I diffed two responses that agreed on currency and differed by 289 bytes, found the hidden div, and went looking in node_modules. The lesson is dull and keeps being true: when a response varies and your own code cannot explain it, the framework is also reading the request.

Worth adding: neither of these two pages is ever served from the CDN cache. Both answer x-vercel-cache: MISS with cache-control: public, max-age=0, must-revalidate, every time. No response here declares Vary: User-Agent, which would matter a great deal if any of them were cacheable.

Check it yourself

No account needed, and every number above comes out of three commands:

curl -s https://pub-trivia.app/pricing | grep -o '£30\|€36.99' | head -1 curl -s -A "Googlebot/2.1" https://pub-trivia.app/pricing | grep -o '£30\|€36.99' | head -1 curl -s -A "HeadlessChrome/141" https://pub-trivia.app/pricing | grep -c '<!--\$-->' 

The first shows your own currency, which depends on where you are, so a reader in the UK will see pounds both times and should pick any non-UK exit node to see the switch. The second shows the base currency whoever you are. The third counts the streaming markers that the bot path removes.

Then open the pricing page in a browser and compare it to what curl just printed, or go and use one of the free tools, which are the pages on the site with no price on them at all.