Property Webhook Recovery with Queue Retries, Exponential Backoff, and Delayed Redrive

Short answer: make recovery the design constraint: keep one delivery ID across every attempt, classify failures before retrying, use capped exponential backoff with jitter, and send exhausted work to a dead-letter queue that operators can inspect and redrive safely.

Consider a property-management SaaS sending a lease.payment_recorded webhook to an accounting integration. The useful data flow is event store to queue to delivery worker to recipient, with a delivery ledger recording each transition. A timeout leaves the sender with an uncomfortable fact: the recipient may have applied the payment even though the worker never saw the response. A queue retry policy controls when another request runs. It cannot, by itself, prevent that request from repeating the business effect.

That distinction drives the whole design.

What should a simple queue retry policy preserve for SaaS background jobs?

Start with the recovery record, not the delay formula. For each delivery, preserve the immutable event ID, tenant and property scope, destination, payload version, attempt number, first enqueue time, next eligible time, and final disposition. Keep credentials and sensitive request headers out of this record. The goal is to answer three operational questions without piecing together scattered logs: what should have happened, why did automated delivery stop, and can an operator replay it without creating a second payment entry?

The delivery ID is the anchor. Generate it once and send the same value as an idempotency key on every attempt, including manual redrive. For property prop_204, lease lease_881, and payment pay_7319, delivery whd_01JQ7K6N9M2A remains whd_01JQ7K6N9M2A on attempt two and on a later DLQ replay. Don't replace it with a new job ID when moving between queues. A new audit ID for the replay action is useful, but it is a different identifier with a different job.

There is a hard limit here. If the recipient neither honors an idempotency key nor provides a way to query whether it applied an event, the sender cannot guarantee one business effect after an ambiguous timeout. Local deduplication can stop the queue from processing the same completed delivery twice, but it cannot reverse an unknown remote side effect. In that situation, stick with documented at-least-once delivery and require the recipient to deduplicate by immutable event ID. This pattern is not suitable when the business operation must be exactly-once but the receiving contract offers no deduplication mechanism.

How should the worker choose between retrying and dead-lettering?

Recovery needs an explicit classification before it needs exponential backoff. A retryable result says another attempt may succeed later. A permanent result says more attempts will waste capacity or repeat an invalid request. Delivered means the recipient acknowledged the request according to the integration contract. Keep that protocol-specific mapping in the HTTP adapter rather than spreading status checks through queue code, because authentication rejection, payload rejection, rate limiting, and an ambiguous network outcome have different operational meanings.

Worker result	Queue transition	Operator meaning
Delivered	Record delivery, then acknowledge	No redrive is needed for this delivery ID
Retryable with budget remaining	Record the next attempt, then schedule it	Automated recovery is still active
Permanent or attempts exhausted	Record the failure, then dead-letter	Human review is required before redrive

This ordering prevents a poison message from spending the retry budget while healthy lease events wait behind it. It also makes the DLQ meaningful: entries arrive because automation made an explicit terminal decision, not because every error followed the same timer until a counter happened to expire.

Implementation starts with an auditable delayed-message state machine

Use a deliberately small policy first: seven total attempts, a one-second base delay, a 15-minute cap, and full jitter. These numbers are operating choices for the example, not universal defaults. A rent-accounting update that remains useful for an hour has a different retry horizon from a door-access event. I'm not sure a fixed schedule can fit every recipient; observed recovery time and delivery age should decide later tuning.

The queue interface below exposes business transitions rather than a particular library. That keeps the worker testable and makes transport changes less invasive. More important, it forces retryAt and deadLetter to carry the original delivery record instead of creating an unrelated message.

type WebhookJob = { deliveryId: string; propertyId: string; destination: string; eventType: "lease.payment_recorded"; payload: { leaseId: string; paymentId: string; amountCents: number }; attempt: number; firstQueuedAt: string; }; type SendResult = | { kind: "delivered" } | { kind: "retryable"; reason: string } | { kind: "permanent"; reason: string }; interface QueuePort { acknowledge(deliveryId: string): Promise<void>; retryAt(job: WebhookJob, availableAt: Date): Promise<void>; deadLetter(job: WebhookJob, reason: string): Promise<void>; } interface DeliveryLedger { wasDelivered(deliveryId: string): Promise<boolean>; markDelivered(deliveryId: string, deliveredAt: Date): Promise<void>; recordRetry( deliveryId: string, attempt: number, availableAt: Date, ): Promise<void>; recordDeadLetter( deliveryId: string, reason: string, failedAt: Date, ): Promise<void>; } const policy = { maxAttempts: 7, baseDelayMs: 1_000, maxDelayMs: 15 * 60_000, }; function nextDelayMs(attempt: number, random: () => number): number { const exponential = policy.baseDelayMs * 2 ** (attempt - 1); const cap = Math.min(exponential, policy.maxDelayMs); return Math.floor(random() * cap); } async function processWebhook( job: WebhookJob, queue: QueuePort, ledger: DeliveryLedger, send: (job: WebhookJob) => Promise<SendResult>, now: () => Date = () => new Date(), random: () => number = Math.random, ): Promise<void> { if (await ledger.wasDelivered(job.deliveryId)) { await queue.acknowledge(job.deliveryId); return; } const result = await send(job); if (result.kind === "delivered") { await ledger.markDelivered(job.deliveryId, now()); await queue.acknowledge(job.deliveryId); return; } if (result.kind === "permanent" || job.attempt >= policy.maxAttempts) { const reason = result.kind === "permanent" ? result.reason : "attempts_exhausted"; await ledger.recordDeadLetter(job.deliveryId, reason, now()); await queue.deadLetter(job, reason); return; } const availableAt = new Date( now().getTime() + nextDelayMs(job.attempt, random), ); const nextJob = { ...job, attempt: job.attempt + 1 }; await ledger.recordRetry(job.deliveryId, nextJob.attempt, availableAt); await queue.retryAt(nextJob, availableAt); }

Full jitter selects a delay from zero to the current capped exponential value. Injecting the random function and clock is not ceremony — it makes the schedule deterministic under test. With random fixed at 0.5, the first delay is 500 ms and the second is 1,000 ms. Also test the attempt-seven boundary, a permanent rejection that schedules no retry, and the ambiguous outcome where the recipient applies the payment but the sender does not observe the response. Every execution must retain the same delivery ID.

One ordering detail deserves scrutiny: recording delivery and acknowledging the queue are separate operations. If the worker stops between them, the message can appear again, which is why wasDelivered runs before the send. If it stops after the remote side effect but before markDelivered, only recipient-side idempotency can close the gap. Exactly-once language tends to hide this boundary. Don't use it unless the entire end-to-end contract earns it.

Governance turns dead-letter redrive into a controlled production change

A DLQ is an isolation boundary, not permanent event history. AWS documents that dead-letter queues hold messages that were not successfully consumed, that their retention needs deliberate configuration, and that a redrive policy can restrict which source queues may use them. The portable lesson is to separate environments and event families whenever shared dead-letter storage would blur authorization, retention, or ownership.

Redrive should preserve evidence. Leave the failed delivery record intact, write a separate audit entry for who initiated replay and which selection criteria they used, then enqueue the original delivery identity. Before release, show a dry count grouped by destination and failure reason. Apply a destination rate limit and provide a stop control. Otherwise a recovery action can recreate the pressure that caused the backlog — only faster.

Be conservative.

For the property webhook, an operator might select exhausted deliveries for one accounting destination after its credentials were corrected. The tool should report how many lease.payment_recorded events match, reject a mixed-tenant selection unless authorization permits it, and redrive at a bounded rate. It should not edit the payload, reset the immutable event ID, or erase the old attempts. If the payload itself needs correction, create a new business event through the normal application path; replay is not a hidden data editor.

Dead-letter storage is also the wrong place for compliance history or long-term tenant support records. Keep the authoritative business event and delivery audit in durable application storage when those requirements outlive queue retention. The catch is extra state and another consistency boundary. For very low-volume, noncritical jobs where duplicates have no business impact, a ledger may cost more operational effort than it returns; a basic retry counter and failure log can be the better choice. Property payment webhooks do have a duplicate side effect, so the ledger earns its keep here.

Migration tests belong in the operational recovery rehearsal

Exponential backoff spaces attempts, but many workers can still contend for the same unhealthy recipient. Set a per-destination concurrency budget so one accounting endpoint cannot occupy the whole worker pool, then put a separate limit on global concurrency. Keep queue polling independent from outbound request timeouts. Delay is not recipient protection.

Measure outcomes rather than queue depth alone. Split delivered, retry-scheduled, permanently rejected, and dead-lettered work by destination. Track the age of the oldest eligible message and oldest dead-letter entry, because a stable total depth can conceal one tenant that is no longer making progress. Alerts should point to an operator action such as pausing one destination or reviewing one failure class; a generic “queue high” page creates noise but gives no recovery decision. Deployment belongs in the same runbook. Delayed messages may be decoded by a newer worker, so version the envelope and retain compatible decoders for the maximum delivery window, or explicitly migrate queued records. During rollback, verify that both worker versions derive the same delivery identity, since one small difference in that calculation defeats deduplication even if the backoff math is perfect. Before launch, run one complete recovery rehearsal: enqueue a payment webhook, produce a retryable result, verify a delayed copy with the unchanged ID, exhaust the attempt budget, inspect the dead-letter record, correct the destination configuration, redrive at a limited rate, and confirm one recipient-side effect. Repeat with a permanent rejection and confirm that it never returns to the active queue. Check that logs omit secrets, dashboards expose property and destination scope, and an operator can halt redrive without stopping unrelated tenants. This is the acceptance test for operational recovery; a unit test of nextDelayMs cannot replace it.

The final choice is modest: use the simplest queue in your environment that provides delayed availability, explicit acknowledgement, dead-letter isolation, and controlled redrive. A library-backed queue can fit a team already operating its data store and wanting retry controls in application code. A managed queue can fit a team that would rather hand off queue operations. Neither option supplies idempotency or a recovery contract for you. Keep correctness in delivery identity, keep evidence in the ledger, and tune backoff only after actual destination behavior gives you a reason.