Yesterday's fix was on the sending side. A slow receiver took 2.5 seconds to answer, the sweeper checked every second, heard silence, and sent the same order three times. The fix was a claim row: one row per delivery key, claimed once, so the second and third ticks had nothing to send. Three deliveries became one.
Then itsvicmoreno asked the question that undoes that fix, politely, in the comments. What if the acknowledgement is lost AFTER the write? The receiver has already taken the order, already charged the card, already sent the email. The sender never hears back, decides the delivery failed, and sends it again. The claim row cannot help you there. The sender did its job, the row says done or it says nothing useful, and a second copy is already on its way. Does the fix live only in the sweeper, or does the receiving end have a say?
I ran that exact case on my own machine. One event, evt-1042, a $29.00 payment confirmation for order ORD-1042. The same event delivered twice, the second send waiting for the first reply, so the second copy arrives after the first write has fully landed. Two receivers, side by side, with the same side effects: a charge line, a confirmation email line, and one stock decrement each time a delivery is accepted. The only difference is that one of them keeps a table of keys it has already processed, and checks it before doing anything else.
My claim is simple, and I expect some of you to disagree with it. A smarter retry loop will never be enough. The receiver has to be able to say no to the second copy, because the receiver is the last place where a duplicate can still be stopped before it becomes a second charge.
Problem 1: the plain inbox charged twice for one order
The plain receiver has no duplicate check at all. Every delivery it accepts is a real delivery, as far as it knows. Here is what two copies of evt-1042 did to it:
| Plain inbox | Receiver with a processed-keys table | |
|---|---|---|
| Deliveries received | 2 | 2 (1 accepted, 1 refused) |
| Charges | 2, $29.00 twice, $58.00 in total | 1, $29.00 |
| Confirmation emails | 2 | 1 |
| Stock decrements | 2, stock 10 down to 8 | 1, stock 10 down to 9 |
| Second copy's reply | 200, processed | 200, duplicate, refused |
Both runs gave the same numbers, twice each, at 00:07:04 and 00:07:08 for the plain receiver, 00:07:06 and 00:07:11 for the one with the table. The plain receiver logged two deliveries for one order, charged $58.00 for a $29.00 order, sent the customer the same confirmation twice, and took two items out of stock for one sale. Nothing about the second copy looked wrong when it arrived. It had the same key, the same order, the same amount, and it arrived after a perfectly normal gap. A receiver that cannot recognise a key it has already seen has no way to know it has been paid twice.
That is the shape of the bug itsvicmoreno pointed at. The sender-side claim is a good fix for the slow case, and I am not taking that back. But the lost-acknowledgement case walks straight past it. The sender believes the first delivery failed. From the sender's side, resending is the correct, careful thing to do. The duplicate is not a mistake in the retry logic. It is the retry logic working as designed, on wrong information.
Problem 2: the table works, and the order of the two steps is the whole trick
The second receiver keeps a small table, one row per delivery key. When a delivery arrives, it checks the table first. If the key is already there, it refuses the copy before any side effect runs, and still answers 200, so the sender stops retrying. If the key is new, it writes the key into the table first, and only then charges, emails, and decrements stock.
The reply matters as much as the refusal. The second copy got {"status":"duplicate","refused":true} with a 200 behind it. A refused copy that answers with an error would look like another failed delivery, and the sender would send it a third time, then a fourth. A refusal has to sound like success to the sender, because from the sender's point of view the order did go through. It went through the first time.
And the order of the two steps inside the receiver is the whole trick. Key first, then side effects. If you charge first and write the key second, there is a gap between the two, and a crash in that gap leaves a charge with no record of it. The next copy arrives, the table says it is new, and the customer is charged again. Writing the key first closes that gap in one direction: a crash after the write leaves a recorded key with no charge, which is a missing charge you can reconcile, not a double charge you have to refund. One of those is a bookkeeping problem. The other is a customer with two charges on their statement.
The check itself is small. In the receiver, before anything else happens:
if (keys.includes(evt.delivery_key)) { // refuse: already processed, no side effects res.end(JSON.stringify({ status: 'duplicate', refused: true })); return; } // record the key FIRST, then charge, email, decrement stock writeKeys([...keys, evt.delivery_key]); That is the entire defence. A lookup, a refusal, and a write that happens before the money moves.
Problem 3: the table does not remove the duplicate, only its cost
Be honest about what the table buys you. The second copy still arrives. It still costs a delivery, a lookup, and a reply. On a bad day, when every acknowledgement is being lost, the table does not reduce the traffic at all. It reduces the damage the traffic can do. Two copies still cross the wire. Only one of them touches money, email, or stock.
That is a narrower win than the headline version, and it is worth saying plainly, because it decides where this fix belongs. The table is not a replacement for the sender-side claim. It is the second layer, the one that catches the copy the claim could not stop. The claim stops the sweeper resending a delivery that is still in flight. The table stops the receiver acting on a delivery that already landed. You need both, because they guard different moments, and the lost-acknowledgement case is the one that slips between them.
Problem 4: the honest limits of this test
Three limits, because the test was small. First, my two copies arrived one after the other, in order, never at the same instant. Two copies landing at the exact same moment, both checking the table before either has written, were not tested, and a race like that is exactly where a simple lookup breaks. A real table wants the check and the write to happen as one step, not two, and I did not test that step.
Second, the crash window I described above, between writing the key and running the side effects, was not tested either. I know the order matters. I did not prove what happens when the receiver dies inside it. A real table wants the key write and the side effects in one transaction, or a reconciliation pass that finds recorded keys with no charge. My test had neither.
Third, I reset the table before each run. Whether the keys survive a receiver restart, and for how long they should be kept before they are cleared, are both open questions my test does not answer. A table that forgets is a table that charges twice, just later.
So the claim stands, narrowed to what the numbers support. The sender-side claim fixed the slow receiver. It does not fix the lost acknowledgement, because in that case the sender is right to resend. The receiver-side table does fix it: the second evt-1042 was refused, one charge instead of two, one email instead of two, stock 9 instead of 8. The fix belongs at the receiving end, in a table checked before the money moves, with the key written before the side effects run.
If you run webhooks, you have met this one. What did your duplicate cost you, a double charge, a double email, a customer asking why they got two copies of the same thing? And what caught it in the end, a table like this one, a claim on the sending side, or something else entirely?
P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam

