Preventing Duplicate Node.js Webhooks: Queue Retry, Delayed Jobs, and DLQ Design

Short answer: for a Node.js edtech service sending outbound webhooks, use a durable queue with delayed retries, an idempotency key, and a dead letter queue (DLQ). Treat delivery as at-least-once, then make duplicate business effects harmless in the application database.

That is the useful answer when a callback can be late, the receiving school system can be slow, and a worker can disappear between an HTTP response and a database commit. A queue is not a promise of exactly-once delivery. It is a place to hold work while the application makes the result auditable.

I care about the boring part: the delivery ledger. A missed webhook becomes an incident; a duplicate webhook becomes a support ticket if the receiver enrolls a student twice. The pattern below makes both visible.

What records and retention rules belong in an edtech delivery ledger?

Imagine an assessment service that emits grade.updated to a district integration. The worker sends the request, gets a successful response, and then the process stops before it records that response. The queue retries. The district receives the same event again.

The reverse ordering is no better. If the worker writes “delivered” first and stops before sending the request, the callback can be lost while the ledger says it succeeded. The remote HTTP request and the local database transaction cannot share one atomic commit, so the system must choose which evidence to trust and expose the remaining uncertainty.

The invariant is narrower than “exactly once”: one delivery ID may produce one local business effect. Put a unique constraint on that ID. Send the same ID as an idempotency key when the receiving contract supports it. Store the event ID, destination, attempt number, response class, and final state. A retry then becomes a new attempt against the same delivery identity, rather than a new event.

Keep the message small. It needs an identifier and enough routing data for a worker to load canonical state; it does not need to carry an unlimited copy of a student record. That separation makes retention, redaction, and replay easier to reason about.

How should a queue pattern protect webhook retry after failed callbacks?

The worker path should be explicit:

1. Create a delivery record with a stable delivery ID.
2. Enqueue a task containing that ID, the callback destination, and the current attempt.
3. Claim the task and load the current payload from durable storage.
4. Send the callback with the delivery ID.
5. Record the outcome before acknowledging the queue message.
6. Delay a retry for a retryable outcome, or move the task to the DLQ after the policy is exhausted.

There is a sharp edge here. An acknowledgement only tells the queue that the worker has finished with its message. It does not prove that the remote system applied the event. The receiver needs its own deduplication rule, and the sender needs a reconciliation view that can distinguish “not attempted” from “attempted, response unknown.”

Use exponential backoff with jitter when the destination is overloaded, but set a maximum attempt count and an overall deadline. A retry policy that is gentle to a receiver can be too slow for a learner-facing workflow. A policy that is fast can turn a temporary outage into a request storm. Your mileage will vary because the right delay depends on the receiver's contract and the business meaning of a late callback.

The DLQ is an operator queue, not a graveyard. Its record should include the last response class, the error reason, and the original delivery ID. Redrive only after someone has checked the destination and source state. Never mint a new identity during redrive; doing so defeats the duplicate guard.

How can a team evaluate cron recovery for overdue work?

Start with the smallest failure boundary that matches the work.

Cron is useful for a recovery sweep: find deliveries that are eligible but have no active task, then enqueue them. It should not be the only record of the job. The public description of cron is a time-based command scheduler, and scheduled workflow systems use the same basic idea of triggering work on a schedule. That makes a scheduler a good clock, but a poor substitute for per-delivery state.

The catch is scope. A queue pattern is not suitable when the process needs durable joins, compensation across many services, or independent replay by several consumer groups. In those cases, use a workflow or event-stream design and keep the same idempotency ledger. Stick with a simple queue when each callback stands alone and an operator can inspect the DLQ.

What does the worker implementation commit before acknowledgement?

The code should show the state transition, not pretend that an HTTP client can provide transactional delivery. This Go example uses a mutex-backed store to make duplicate suppression visible. A production implementation needs a database transaction and a uniqueness constraint on DeliveryID.

package main import ( "context" "fmt" "sync" ) type Delivery struct { ID string } type Ledger struct { mu sync.Mutex completed map[string]bool } func (l *Ledger) Apply(_ context.Context, d Delivery) (bool, error) { l.mu.Lock() defer l.mu.Unlock() if l.completed[d.ID] { return false, nil } // In production, the domain write and this claim share one transaction. l.completed[d.ID] = true return true, nil } func main() { ledger := &Ledger{completed: make(map[string]bool)} ctx := context.Background() for range 2 { applied, err := ledger.Apply(ctx, Delivery{ID: "grade-1042"}) if err != nil { panic(err) } fmt.Println("applied:", applied) } } 

The output is one true and one false. The important ordering is outside this toy store: load the event, apply the local effect under the delivery key, record the attempt outcome, and acknowledge only after those writes are durable. If the remote response is retryable, persist the next eligible time before enqueueing the delayed replacement. If the response is terminal, persist the DLQ decision with the same identity.

Don't acknowledge early.

How does a team plan rollout for safe redrive?

Test the boundary, not just the happy-path HTTP request. Kill the worker after the remote response but before the ledger write. Kill it after the ledger write but before acknowledgement. Deliver the same task twice concurrently. Return a retryable response until the attempt limit is reached. Then verify that the DLQ record can be inspected and redriven without changing the delivery ID.

Observability should follow the identifier across the queue, worker, HTTP request, ledger, and DLQ. Useful fields are delivery ID, event ID, attempt, scheduled time, completion time, response class, and redrive actor. Keep payloads out of ordinary logs when they contain learner data. A counter saying “12 retries” is less useful than a trace that explains which destination and which delivery ID consumed those retries.

Measure outcomes that affect the decision: age of the oldest pending delivery, retry rate by destination, DLQ growth, redrive success, and the count of duplicate claims rejected by the ledger. The exact alert thresholds belong to the service-level objective; there is no honest universal number for an edtech integration serving every school district.

For a Node.js service, the framework or queue library is secondary. BullMQ, a cloud queue, a workflow engine, and a cron-plus-database design can all be made to follow the invariant. They differ in deployment burden, visibility, retention, and replay semantics. Pick the mechanism that your team can operate at 03:00, then document the failure ordering in the runbook.

References

• https://en.wikipedia.org/wiki/Cron
• https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows