At 22:04 UTC on Wednesday our error inbox lit up: a request that does nothing but read from the database had returned 500. Then another. Then someone's studio page crashed into a bare white "Application error" screen.

I opened our health endpoint. It said {"db":"ok"}, in 3 milliseconds.

This post is about the 40 minutes in which our monitoring was confidently wrong, why, and the small change that makes it tell the truth.

TL;DR
Our /api/health ran select 1 through a connection pooler that caches query results, so it kept answering "ok" while real reads took 15–31 seconds. The probe now runs inside a transaction, which the pooler never caches. We still don't know why the database stalled.

The symptom: some reads took 31 seconds, the health check took 3 ms

I build RefVid, a small tool where you upload a few photos of a product or a character once, then make short videos that keep it looking the same in every shot. It runs on Cloudflare Workers and talks to Postgres through Hyperdrive, Cloudflare's connection pooler.

Between 22:31 and 22:33 I fired 8 parallel requests at an endpoint that reads the default model settings. 5 of them took between 15 and 31 seconds. In the same two minutes, /api/health answered "ok" every time, in 3 to 24 ms.

Meanwhile, for real people:

  • Starting a video timed out at our 15-second limit and came back as a 500.
  • The studio page, which reads the user's account on the server, failed to render and showed Next.js's default crash screen: no sidebar, no words in their language, no retry button.
  • Someone signed up from an iPhone at 22:25. They never made a video. Their first minute on our site was that crash screen.

Rolling back didn't help

My first guess was the deploy from earlier that evening. We had shipped several times that day, and the last one went out at 20:51.

So I rolled the Worker back to the 20:55 version. Nothing changed. Reads still stalled.

Then I checked our other app, which lives on the same database cluster. It was fine the whole time. And nobody on our side had touched the production database. So: not our code, not the cluster as a whole, not a migration. Something between this one app and its database.

By 22:47 it had recovered on its own: 5 of 5 test video starts went through. I don't like that sentence, and it's the honest one.

Why the health check lied: select 1 was a cached answer

Here is the part I should have known. Hyperdrive caches the results of read queries that run outside a transaction. That's a feature, and most of our reads are happy with it. We even have a helper, readFresh, that wraps reads in a transaction for the places where a stale answer would be wrong.

Our health probe didn't use it. It had looked like this since the day we added it:

// before await db.execute(sql`select 1`); 

select 1 is the most cacheable query there is. Its answer never changes. So for 40 minutes the pooler kept handing our probe a remembered "1" without ever asking the database, and our monitoring reported a perfect, fast, healthy database.

The fix is one line, plus the 3-second timeout that was already there:

// after await readFresh(db, (tx) => tx.execute(sql`select 1`)); 

A transaction always goes to the database. If the database is stuck, the probe is stuck, the timeout fires, and health says degraded. There's a test now that builds a fake database whose plain read succeeds but whose transaction fails, and checks that health says degraded. It fails on the old probe.

A crash page that says "dev" when it means "unknown"

The crash report from the studio page had one more lie in it. It said the page was running build dev, which made it look like a local server, not production.

It wasn't. No route in our app had its own error boundary, so the failed render fell all the way through to the global one, which draws its own <html> without the build stamp we put on every page. Our reporter read the missing stamp as "dev". It now says unknown.

And the person on the other end now gets something better than "Application error": a page that keeps the studio sidebar around it, says "This page couldn't load" in their language (we ship 7), and has a "Try again" button that refreshes the data and re-renders. It's the same size in both states, so nothing jumps when it decides.

The same evening, a second cache fooled us on purpose

Earlier that evening I had been looking at a different number: how often ChatGPT's fetcher gives up on our pages before they load. Over a week it was 43 of 788 loads, 5.5%. Every single one of those 43 was a cache miss.

The reason: our edge HTML cache keys each page by the Worker version. Every deploy starts a new, empty cache in every data centre. We deployed 10 times between 13:55 and 19:17 that day. A data centre that ChatGPT reaches three times a day is almost always asked first after a deploy, the page has to be built from storage in another region, and the fetcher hangs up.

The fix runs the other way from the health check: we now want an old answer. On a miss, if the same page was stored in the last 48 hours, the Worker serves that copy right away, marked STALE, and builds the new one in the background. The next visitor in that data centre gets the new build. After the next deploy, all 12 of our warm-up probes got STALE instead of waiting.

It fooled us once too: our own post-deploy browser check loaded a STALE page, saw its prefetches refused by the newer build, and failed the deploy. The check now throws away any load that isn't the build it just shipped and tries again.

Same evening, same lesson from both directions: a cached answer is fine for a visitor and wrong for a probe. Every check has to know which one it's getting.

What we haven't fixed yet

  • We don't know why the database stalled. Not our deploy (rollback didn't help), not the whole cluster (the other app was fine), not a migration (nobody touched it). The next step is reading the database provider's query insights for 22:00–22:45. Until then, the honest root cause is "unknown".
  • We found out from users, not from monitoring. The first signal was error reports from real requests; the health endpoint stayed green the whole time. The probe is honest now, but it still only speaks when something asks it.
  • The person who signed up at 22:25 made no video that night. Their first minute was our crash page. The new error page would have given them a "Try again" button, but it shipped at 22:42, 17 minutes too late for them.
  • STALE has a cost. The first visitor of a page in each data centre after a deploy sees the previous build's HTML. For a page whose text just changed, that's one visitor per data centre who reads the old version.

If you want to see the page that crashed that night working again, the studio is where it lives, and one of the pages ChatGPT sends people to is the free video generator.

A question for you

What does your health check actually touch? If you ran it right now with your database frozen, would it say "ok"?