rudratosh left a comment on my last piece that I have been turning over for two days. The piece was about a slow payment confirmation: my helper sent a $29 charge, the reply took 2.5 seconds, and my helper's own patience ran out after 1 second, so it wrote the step down as UNKNOWN even though the money had already moved. The fix was a claim row, a small record that says who owns this delivery and whether it finished, and the planner reading that row before it plans anything. One delivery, no duplicate, no second charge.
rudratosh asked the next question, and it was the right one. Reading the row works. But what about being told? What if something watches the row and taps the planner on the shoulder the moment the late reply lands, instead of the planner going to look? Does that change the next plan any differently? And what happens when the tap lands in the middle of a step, or arrives twice, or arrives after the planner has already stopped?
I spent a night testing it on the same small setup, with the same $29 payment and the same slow receiver. Three versions of the same evening, and the answer surprised me in one place and confirmed my guess in another.
Problem 1: the tap lands mid-step, and the step does not care
Here is the shape of the test. The planner has three steps: deliver the payment, send the customer confirmation, report the result. The payment goes out, the reply takes 2.5 seconds, and after 1 second the planner already believes the step is UNKNOWN. At 2.5 seconds the reply lands and the claim row flips to done. A watcher sees that flip and drops a note into the planner's inbox: version 2, status done.
But step 1 is not finished at 2.5 seconds. It still has a second and a half of follow-up work, drafting the confirmation note. So the tap lands mid-step, at about 3 seconds in, while the step is still working.
What should the planner do? My first instinct, a month ago, would have been to stop everything and re-plan on the spot. The test says that instinct is wrong, and rudratosh said the same in his follow-up comment: finish the step, then re-read before the next one. The step already in flight has its own delivery key and its own duplicate protection, so letting it finish is safe. Yanking it halfway just creates a second half-finished piece of work to clean up, and now you have two messes instead of one.
So that is what the test planner does. It finishes step 1 at 4.5 seconds, drains its inbox at the boundary, sees the newer version, reads the row again, and plans step 2 on the fresh fact. One delivery. No re-send. The mid-step tap changed nothing about the step in progress, and that is exactly the point.
Problem 2: the tap that arrives twice looks like news
Halfway through step 2, the watcher delivered the same note again. Version 2, status done, second copy. In a real system this happens all the time: a watcher retries, a queue redelivers, a tired network sends the same thing twice.
Without a version number, the second copy looks like a brand new event. The planner would read the row again for no reason, or worse, treat it as a second change and start second-guessing a plan that was already right.
With the version number, the test is boring, and boring is good. The planner compares the token against the last version it read that turn. Token 2 against last seen 2: equal, so it is a duplicate, ignored. The inbox file holds four lines (versions 1, 2, 2 again, 3), and the planner's log says, in plain words, that the second copy was a duplicate and it moved on. The version number is what turns a redelivery from an event into noise.
This is the part people skip when they build a notification channel and call it done. A push without a version is a rumor. A push with a version is a fact you can check.
Problem 3: the flag that must forget
Here is the subtle one, and rudratosh pushed me on it twice. The planner keeps a small flag: a newer version exists than the one I cached, so my cache is stale. A token sets the flag. The planner re-reads the row at the next boundary. Then what? Does the flag stay dirty until the turn ends, or does it clear?
It clears on the re-read. The flag is an edge, not a level. Its whole meaning is "there is a newer row than the one you cached," and once you have re-read and refreshed, that job is done. Leaving it dirty forces a pointless re-read at every later boundary, and worse, it hides the next token. If token 3 lands after the re-read, you want the flag to flip again and earn its own re-read. A flag that never clears cannot tell you anything twice.
But the clearing itself has a trap, and this is where I had to write a separate test, because the live run never caught it. Between "read the row" and "clear the flag" there is a gap, and a token can land in that gap. If the planner clears the flag with a plain reset, the new token is lost: the flag reads clean although a newer row exists.
The fix is to clear by comparing: only clear if the flag still holds the token you actually read. I tested both ways against the same timing. The compare-and-swap clear refused and kept token 3 dirty. The plain reset cleared it and lost token 3. Same setup, same timing, opposite results. It is a five-line difference, and it is the difference between a planner that notices the second change and one that sleeps through it.
Problem 4: the planner is already dead
The last test was the ugliest. I killed the planner after step 1 and then let a change arrive: an outside note on the claim row, version 2 to 3, token 3, at 5.4 seconds. The watcher saw the planner's heartbeat go stale, checked that the process was gone, and sent the note somewhere else: a fallback owner's file, not the dead inbox.
The dead inbox still holds the two notes from when the planner was alive, versions 1 and 2, unread. Nobody will ever read them. That is fine. What matters is that the post-death note went to a human-shaped destination instead of vanishing into a queue nobody drains. A push channel with no living receiver is a letter to an empty house. Somebody has to own the fallback, or the observation is lost exactly when it matters most — right after something died.
Problem 5: push and pull gave the same answer, so why push at all?
Here is the result that made me sit back. Pull and push reached the same next plan. One delivery each, no duplicate, proceed to step 2. The log-only version, where the fix existed but nobody read it, sent the payment twice.
So push changed nothing about the decision. The planner learned earlier, mid-step instead of at the planning moment, but it decided the same thing, on the same row. That is rudratosh's point, and the test backs him: push is attention only. It tells you when to look. Pull is the correctness floor. It is what you look at. If the push ever disagrees with the row, the row wins, every time, because the row is the state and the push is only a rumor about the state.
So why build push at all? Because attention has a price in the other direction. A planner that only pulls learns late. In my test the planner's patience ran out at 1 second and the reply landed at 2.5. A planner that pulls only at planning time sits on a wrong belief for the whole gap, and if the next step starts before the pull, it re-sends a payment that already went through. Push shortens that gap. It does not make the decision better. It makes the decision sooner, and sooner is worth something when the wrong decision costs $29 twice.
That is my claim, and I know it is arguable: a notification that changes when you look is worth building, but it is never worth trusting. The row is the truth. The tap is only the tap on the shoulder. If you build the tap and skip the row, you have built a faster way to be wrong.
Have you ever had a system where the notification arrived and the state said something different? Which one did you trust, and what did it cost you? I would like to hear what it was, and whether you still trust the tap now or read the row first every time.
P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam


