Short answer: let the cron trigger enqueue a bounded Node.js task, checkpoint each shipment webhook, and let a worker retry from that checkpoint. Do not make the scheduler own a long-running delivery loop.

In a logistics system, a nightly carrier scan can find 80,000 shipment updates. A 15-minute execution limit is not a small inconvenience at that volume. It is a signal that scheduling and delivery have different lifetimes. The scheduler should create durable work; a queue worker should own timeout handling, retries, and idempotency.

I care about the first successful call and the first useful failure message. Configuration that exists only to connect two control planes is a smell. The design below keeps the application ledger authoritative and makes a timeout recoverable instead of mysterious.

A governance ledger for outbound shipment webhook retry policy

Shape Use it when Recovery rule Trade-off
Direct cron loop The batch is bounded and finishes well inside 15 minutes One transaction covers the whole run A slow carrier can consume the scheduler budget and leave no resume point
Cron plus queue worker The backlog grows or recipients have uneven latency Each message resumes from a stable delivery record Requires a queue, a ledger, and worker observability
Workflow engine The job has branching, timers, and human approval Workflow state records every step More concepts and operational surface than a retry scan needs

For this scenario, choose cron plus a queue worker. The runner-up is valid when the workload is genuinely tiny; the workflow engine is better for a multi-step customs process. The choice is about the recovery invariant, not a vendor badge.

How can background queue workers split long-running cron tasks?

Treat the scheduled endpoint as a transaction boundary. It validates the requested scan window, inserts one scan record, and enqueues the first chunk. It returns immediately. A worker claims, say, 200 deliveries, sends them, records the result, then publishes the next cursor. The chunk size is a measured setting, not a magic constant; carrier latency and the desired retry delay decide it.

The message should contain a scan ID, a cursor, and a limit. Keep the shipment payload in the database. That prevents a stale queue message from becoming a second source of truth and keeps messages comfortably below common queue size limits.

The acknowledgement edge matters most. A worker acknowledges only after the delivery row records an accepted response. If the carrier accepted the request and the process died before acknowledgement, the queue will deliver the message again. The stable delivery key makes that replay harmless.

No magic.

Here is the core loop in TypeScript. The interfaces are intentionally boring so the same state machine can sit over a hosted queue, Redis-backed queue, or a self-hosted broker.

type DeliveryStatus = "pending" | "accepted" | "retry" | "dead"; interface Delivery { id: string; shipmentId: string; endpoint: string; payload: unknown; status: DeliveryStatus; attempts: number; nextAttemptAt: Date; } interface Ledger { claim(id: string): Promise<Delivery | null>; markAccepted(id: string, responseCode: number): Promise<void>; markRetry(id: string, nextAttemptAt: Date, reason: string): Promise<void>; } interface Queue { publish(message: { scanId: string; cursor: string; limit: number }): Promise<void>; ack(): Promise<void>; retry(delayMs: number): Promise<void>; } async function processDelivery( message: { deliveryId: string }, ledger: Ledger, queue: Queue, fetcher: typeof fetch, ): Promise<void> { const delivery = await ledger.claim(message.deliveryId); if (!delivery || delivery.status === "accepted") { await queue.ack(); return; } const response = await fetcher(delivery.endpoint, { method: "POST", headers: { "content-type": "application/json", // The recipient uses this value to collapse a replay. "idempotency-key": delivery.id, }, body: JSON.stringify(delivery.payload), }); if (response.ok) { await ledger.markAccepted(delivery.id, response.status); await queue.ack(); return; } const retryable = response.status === 408 || response.status === 429 || response.status >= 500; if (!retryable || delivery.attempts >= 8) { await ledger.markRetry(delivery.id, new Date(8640000000000000), `terminal ${response.status}`); await queue.ack(); return; } const delayMs = Math.min(30 * 60_000, 2 ** delivery.attempts * 1_000); await ledger.markRetry(delivery.id, new Date(Date.now() + delayMs), `retryable ${response.status}`); await queue.retry(delayMs); } 

The code makes one deliberate assumption: the recipient honors the idempotency key. If it does not, put a deduplication table in front of the side effect. HMAC authentication can prove who sent a payload, but it cannot stop a duplicate; those are separate controls.

Measuring timeout budgets with a failure-injection harness

There are three different clocks. The scheduler clock limits the enqueue request. The worker clock limits one HTTP attempt. The business clock limits how long a shipment update remains useful. Mixing them produces bad alerts: a worker timeout may be a retryable network failure, while a successful carrier response followed by a process crash is an acknowledgement failure.

Write those states explicitly. Store attempts, nextAttemptAt, the last response code, and a monotonic scan cursor. A 429 should honor Retry-After when supplied; otherwise use bounded exponential backoff. Never sleep inside the scheduler request. Release the worker and schedule another message.

Test the awkward sequence. Stop the worker after the carrier has accepted shipment-4821 but before ack(). Restart it. The ledger should show one accepted delivery, and the replay should be a no-op. Then pause the schedule, add 10,000 rows, and run a reconciliation command from the stored cursor. Logs are evidence; the ledger is the decision. When I exercise this path, I inject an HTTP 429, a dropped connection, and a process kill at that exact acknowledgement boundary, then check that attempts increments once and the next cursor never moves backward. I also enforce a 900-second scheduler deadline in the test harness, even though the enqueue handler should finish in seconds, because a passing happy-path benchmark can hide a retry storm that only appears near the limit.

Provider boundaries for queues and schedulers

BullMQ is a practical Node.js choice when the team already operates Redis and wants queue primitives close to application code. Temporal is a better fit for durable, branching workflows with timers. Airflow models scheduled DAGs well, but a latency-sensitive webhook retry path is not its natural unit. Cloudflare Cron Triggers can provide the scheduled entry point, yet a trigger alone is neither a queue nor a delivery ledger.

The catch is scope. A queue worker is not suitable when the process needs a native fan-out/join graph, independent consumer groups, or replayable event streams; use a workflow or streaming system for those requirements. Stick with direct cron execution when the maximum batch is proven to complete far below the limit and the side effect is idempotent by construction.

Your mileage may vary on chunk size. Measure p95 carrier latency, kill workers during acknowledgement, and inspect the age of the oldest pending delivery. Those three numbers tell you more than a dashboard showing only successful cron invocations.

References