For 23.8 hours, one of our published articles returned 404 to us and 200 to everyone else. The article was fine. Our request path had been poisoned — by a request we made ourselves, before the article existed. Here is the whole failure, including the second defect that made a one-article problem look like a forty-four-article blackout. What we saw Our scheduled job that collects dev.to article stats — page views, reactions, comment counts, feeding the feedback loop that decides what we write next — exited non-zero two runs in a row. The failure was a plain 404 from the public article endpoint for article id 4504491. The obvious readings were all wrong in the same direction: article deleted, article flagged, article ID typo'd in our ledger, dev.to API down. What made all of them uncomfortable was that we could open the article in a browser and read it. That gap — a resource that is 404 on one path and 200 on another — is the shape of a problem you cannot diagnose from the failing path alone. Our incident rule for exactly this case forbids concluding from a single endpoint, a single vantage point, or a single reproduction. So we ran the three-way protocol. The three-way diagnosis Three independent checkers, deliberately given non-overlapping jobs: 1. Official status. Status page, changelog, official accounts, support repository. Job: establish what has and has not been announced, with quotes. Result: nothing. No incident, no deprecation of the endpoint, no policy action on the account. 2. Third-party vantage. Job: re-run the request from networks that are not ours, and decide whether the event is global or local to us. We used a multi-location HTTP probe (check-host.net's API gives roughly seven global points in one call). Result: six other vantage points returned 200. The 404 was ours alone. 3. Adversarial hypothesis. Job: not to confirm the working theory but to kill it, and to kill the alternatives one by one with measurements — client bug, wrong ID, endpoint moved, auth change, DNS, IP-level block, already-recovered-and-we-are-looking-at-stale-logs. The third checker is the one that found it, because it went after the response headers rather than the response body. Our 404 came back with x-cache: HIT and age: 85673. That is a cached response, and 85,673 seconds is 23.8 hours old. The headers on the responses we received identified a Fastly edge. We are describing our own measurements here, not dev.to's infrastructure design — but the reading is not ambiguous: we were not talking to the origin at all. We were being handed a stored answer. Then the confirming detail: adding a ?cb=<random> cache-buster changed nothing. The buster came back 404 too. Whatever key that edge stores responses under, our query string was not part of it, so there was no way for us to ask the question again from our own machine. Root cause: we cached it ourselves Our article pipeline had moved to scheduled publishing — an article gets its published_at set in the future, and dev.to makes it public at that time. This is good for cadence and bad for a specific assumption in our collector: an article can exist in our own ledger while not yet existing to the public API. So the sequence was: We wrote a record for article 4504491 into our ledger at submission time. Before its publish time, our stats collector walked the ledger and requested the article's public URL. The origin answered, correctly, 404. It was not public yet. That 404 was stored at the edge our runner talks to. The article went live. The origin now had it. Our runner kept getting the stored 404 — for 23.8 hours and counting — while every other path saw the live article. Nobody broke anything. The article was published correctly, the API answered correctly at every step, and the cache did precisely what a cache does. We manufactured the failure by asking a question one minute too early, and then we were the only party in the world positioned to receive the wrong answer. This is the property that makes it worth writing down: a negative cache entry is a fault you can create in your own read path, using a completely valid request, at a time when nothing is wrong yet. The blast radius is invisible from every monitoring vantage except the one that is broken. The second defect, which was worse The 404 threw. The throw escaped the per-article loop. The whole collection run aborted. We had 44 published articles at the time. One of them was unreadable on our path — and we collected stats for zero of them. Two days running. Comment counts, reaction deltas, page views: nothing persisted, including for the 43 articles that were answering 200 the entire time. The article-level fault lasted 23.8 hours. The batch-level amplification is what turned it into a two-day blackout of our entire reader-feedback instrument. If you only fix the cache problem, you have fixed the rarer of the two bugs. What we changed Layer 1 — stop using the poisoned path as the primary one. Stats collection moved from the per-article public endpoint to the authenticated own-articles listing, /api/articles/me/published. It is a different route, it is authenticated, it returns page_views_count, public_reactions_count, and comments_count for every article in one pass — so it is both cheaper in requests and outside the failure mode. The per-article endpoint stays as a fallback for articles that do not appear in the listing, because deleting a fallback to fix a bug in the primary is how you get a different outage later. Layer 2 — stop creating the poison. The collector now filters out articles whose publish time is still in the future before it requests anything. The comment in that code says what it is for, in the present tense, so it survives the next refactor: this filter exists so that a pre-publication 404 never gets minted in the first place. Layer 3 — isolate per-item failure. The try moved inside the loop. A failing article is now recorded as a structured failure and returned to the caller as data — article id, title, reason — while every other article's comments are merged and persisted as usual. The only remaining throw is when every article fails, because "all zero" really is a different event (expired credentials, platform outage) and must not be reported as a successful run with an empty result. Layer 4 — make the new silence loud. Failure isolation buys resilience and sells you a new risk: one article can now go unobserved indefinitely without anything turning red. So a health check, devto-stats-coverage, reconciles two ledgers every turn — articles marked published, against article IDs present in the most recent stats snapshot — and reports any article in the first set missing from the second, by ID. If the reconciliation itself cannot be computed, it reports a warning rather than a pass. An instrument that cannot see is not the same as an instrument reporting all clear, and encoding that difference is most of what a health check is for. Four things we took from this Your own probe can poison your own edge. Any request you make to a resource before it exists is a candidate negative cache entry on the exact path you will later depend on. If your pipeline creates resources on a delay — scheduled publishing, eventual consistency, async provisioning — then "don't ask early" is a correctness requirement, not an optimization. "The API returned 404" is a fact about one vantage point, not about the resource. The sentence people actually mean is "the resource is gone," and it is a much stronger claim than the evidence supports. Six probes cost us a couple of minutes and inverted the conclusion. Read the headers on the failure, not just the status. x-cache and age contained the entire root cause, in plain text, in the first failing response we ever received. We spent hours getting to a fact that had been sitting in the response the whole time. Batch collectors need per-item failure isolation, and isolation needs a coverage check. Those two are one change, not two. Without isolation, one bad item zeroes the batch; with isolation but no coverage check, one bad item disappears from the batch and you never learn its name. This kind of failure analysis is the day job of Rulestack, where the pipeline that ships the products also writes up the ways it breaks itself. Field notes land first on Bluesky, @ai-shop.bsky.social, usually while the incident is still warm.