Webhook Follow-Up Scheduling: Delayed Messages vs Per-Event Timers Explained

Short answer: for a logistics webhook that must run about an hour later, put a durable due time in a queue-backed table and let a bounded worker claim it; use a cron sweep as the recovery path, not one timer per event. This keeps latency visible while a rate-limited pool drains at a controlled cost.

Imagine a carrier sends package.delivered while your label service is already busy. The follow-up checks for a missing proof-of-delivery image 60 minutes later. A process-local timer is easy to write, yet a deploy, crash, or horizontal scale-out can erase it. A per-event cron entry survives longer, but thousands of entries turn operations into bookkeeping. The useful design is boring: one record, one due timestamp, one idempotent handler.

The field guide

For a rate-limited worker pool, the first two options can share the same handler. The difference is where waiting happens. A delayed message waits in the broker; a due-time row waits in storage. In both cases, the worker should acquire capacity only when the task is ready.

How should a webhook schedule its follow-up task an hour later?

Start with a durable state transition. Store the webhook event ID, the intended action, run_at, attempt count, and a deduplication key in one transaction with the business update. Return a success response to the carrier before doing the slow follow-up. The one-hour value is a scheduling target, not a promise of millisecond precision.

A small table can be enough:

type FollowUp = { eventId: string; kind: "pod_check"; runAt: Date; attempts: number; dedupeKey: string; }; 

A poller claims ready rows in short batches. PostgreSQL's FOR UPDATE SKIP LOCKED lets concurrent workers skip rows already claimed by another worker, which is useful for queue-like table access. Keep the transaction short, mark a lease or status, then perform the external call outside that transaction.

async function claimReady(limit: number): Promise<FollowUp[]> { return db.transaction(async (tx) => tx.query<FollowUp>( `SELECT event_id AS "eventId", kind, run_at AS "runAt", attempts, dedupe_key AS "dedupeKey" FROM follow_ups WHERE status = 'ready' AND run_at <= now() ORDER BY run_at FOR UPDATE SKIP LOCKED LIMIT $1`, [limit], )); } 

The claim size is your first latency-versus-cost knob. A batch of 20 may keep a small pool full; a batch of 500 may create a burst that trips a carrier limit. Measure queue age, claim duration, downstream response time, retry count, and the number of ready rows. Logs should carry the webhook ID and dedupe key so one delivery can be followed across retries.

Delayed message or cron sweep: how do failure modes differ?

A delayed queue gives the broker responsibility for wake-up and often carries retry and visibility settings alongside the message. That is convenient when your team already monitors the broker. It also means retention, redelivery, and ordering rules become operational dependencies; read those rules before promising exactly-once effects.

A cron sweep over a due-time table is more inspectable. An operator can query late rows, replay one event, or change a lease without learning a broker-specific console. The cost is polling. A five-minute sweep can add nearly five minutes of scheduling latency, while a 15-second sweep consumes more database reads. Your mileage may vary because row volume, indexes, and pool size change the balance.

Per-event timers are the smallest code path, and the least durable. They are suitable for a single-process utility where a missed follow-up is harmless. They are not suitable when a delivery must be audited, when deployments are frequent, or when multiple replicas can receive the same webhook.

An observable drain loop

Treat the worker as a control loop: discover ready work, admit only as much as the downstream limit allows, execute idempotently, and record the next state. A simple semaphore around the carrier call is clearer than letting an unbounded promise list decide your concurrency.

async function drain(pool: FollowUp[], maxInFlight: number) { const queue = [...pool]; const active = new Set<Promise<void>>(); while (queue.length || active.size) { while (queue.length && active.size < maxInFlight) { const task = queue.shift()!; const run = handleOnce(task).finally(() => active.delete(run)); active.add(run); } if (active.size) await Promise.race(active); } } 

handleOnce must check the dedupe key before changing shipment state, and it must classify failures. A timeout may be retried with backoff; a permanent validation error should be recorded for review. Emit a metric for age at execution, not only age at claim. That catches a pool that is technically busy but failing to make progress.

Measure twice.

For example, suppose 1,200 delivery events become due during a regional outage. If the carrier allows 40 calls per minute, releasing every row at once creates a retry storm. Instead, the scheduler can keep 40 permits available, replenish them as calls finish, and leave the remaining rows in ready state. An alert on the oldest run_at value tells the on-call engineer whether the backlog is draining. A second alert on the ratio of retries to completions distinguishes a slow downstream from a poisoned payload. These signals also make cost legible: idle capacity shows up as low utilization, while excessive retries show up as extra calls without completed work. The exact thresholds belong to your service-level objective and carrier contract, so I would not copy a number from another team.

I once assumed a larger batch would lower cost because it reduced polling overhead. The surprise was the latency spike: a carrier's 429 responses caused retries to occupy every slot, so fresh due tasks waited even longer. The correction was to cap retries separately, add jitter, and alert on the oldest ready task. Small change. Big signal.

Limits and a decision rule

The catch is that no scheduler removes the need for idempotency or capacity planning. A delayed queue is a poor fit if your organization cannot operate its retention and redelivery settings. A database sweep is a poor fit when the table is already under heavy write pressure or the deadline is sub-second. Stick with a per-event timer only when loss on restart is an accepted business outcome.

For this logistics case, choose the durable delayed queue when predictable wake-up and built-in retry controls matter more than broker operations. Choose the due-time table when auditability and SQL-level recovery matter more. In either choice, keep the follow-up handler independent of the waiting mechanism; then a later migration changes the dispatcher, not shipment logic.

References

• https://docs.celeryq.dev/en/stable/getting-started/introduction.html
• https://www.postgresql.org/docs/current/sql-select.html