Reliable webhook delivery at scale is an exercise in bounded failure and controlled recovery. Dispatching an HTTP POST to a customer endpoint looks trivial until you run millions of events across thousands of multi-tenant endpoints. Each endpoint has its own latency profile, rate limits, outage durations, and consumer bugs.

Three failure modes repeatedly take down event-driven platforms:

  1. Tenant starvation during rate limiting: A single noisy or rate-limited tenant forces naive workers or shared backoff loops to stall, degrading delivery SLAs for unrelated tenants.
  2. Poison-pill worker crashes: Malformed, schema-mismatched, or oversized payloads crash workers or poison replay batches if they aren’t isolated before execution.
  3. Self-inflicted replay DDoS: Replaying a large backlog or DLQ without rate controls floods recovering consumers, triggering another outage or a cascade of 429 / 5xx errors.

This guide covers the architectural patterns for solving all three in a production webhook pipeline.

What Industry Practice Looks Like

Retry policies vary widely across webhook providers, which is one reason your own pipeline needs explicit, tunable semantics rather than copied defaults.

  • Stripe retries live-mode deliveries with exponential backoff for up to three days. Sandbox deliveries are retried only three times over a few hours. Stripe does not publish the exact intervals in its docs, though third-party write-ups describe an immediate retry followed by delays of roughly 5 minutes, 30 minutes, 2 hours, 5 hours, and 10 hours, then every 12 hours. Treat that schedule as community-reported, not official. Stripe treats any non-2xx response as a failure, including 4xx, and disables an endpoint after about three days of continuous failures.
  • Svix, which powers webhooks for many SaaS products, documents a schedule of immediately, then 5 seconds, 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours, and a final 10 hours. An endpoint that fails continuously for 5 days is disabled, and operators can manually retry or “recover failed” messages from a given date.
  • GitHub does not automatically redeliver failed webhook deliveries. It records a delivery as failed if your server is down or exceeds its response timeout, and expects you to redeliver manually or through the REST API. The documented timeout is 10 seconds on GitHub.com, while some GitHub Enterprise Server versions document 30 seconds. Retry budgets differ enormously between providers, so don’t assume a sender will retry at all.
  • Standard Webhooks, an open specification, defines webhook-id, webhook-timestamp, and webhook-signature headers. The spec says the unique identifier stays the same across retries of a failed webhook and is often used as an idempotency key. The timestamp reflects the attempt, so it helps prevent replay attacks.

1. Handling Retry-After Headers Without Starving Other Tenants

Why Blind Exponential Backoff Ignores Provider Rate Windows

Exponential backoff with jitter smooths over transient network blips, DNS hiccups, and brief 5xx outages without creating a thundering herd. A typical delay is min(cap, base_delay * 2^attempt) with randomization applied (for example, full jitter picks a random value between 0 and that bound).

However, blind exponential backoff treats a 429 Too Many Requests (or a 503 with Retry-After) as just another generic failure. That creates two problems:

  • Premature retries during explicit cooldowns: If a receiver responds with 429 and Retry-After: 120, retrying after 2 or 4 seconds wastes worker cycles and pollutes logs. Some APIs keep counting requests made while you are throttled, so retrying too early can lengthen the wait.
  • Overly long delays when a short window is specified: If a receiver says Retry-After: 5 on attempt 5, a blind schedule that jumps to 30 minutes waits far longer than necessary.

How a worker handles the wait matters just as much:

  • Blocking a shared worker: If a worker sleeps for Retry-After seconds while holding a connection or slot from a global pool, a handful of throttled endpoints can tie up concurrency for the entire platform.
  • Head-of-line blocking in a FIFO queue: If deliveries for many tenants share one FIFO queue and a failing message stalls the head, or retries are requeued naively, healthy tenants behind the bottleneck see unnecessary latency.

Core rule: Treat Retry-After as scheduling metadata for that tenant/endpoint, not a reason to stall global workers or delay unrelated tenants.

Parsing and Validating Retry-After Safely

Retry-After is defined in RFC 9110 (section 10.2.3). It is commonly sent with 429 (RFC 6585) and 503 (RFC 9110), and the RFCs let servers include it on other responses too, such as 413 and 3xx redirects. For 429 and 503 the server may include it, so its absence is normal. Parse it defensively before scheduling the next attempt:

  1. Support both formats. The value is either a non-negative integer number of seconds (Retry-After: 30) or an HTTP-date in GMT (Retry-After: Sun, 06 Dec 2026 07:28:00 GMT). For a date, subtract the current time. If the result is zero or negative, you may retry immediately, subject to your own floor.
  2. Clamp to safe bounds. Never trust an endpoint blindly. Enforce a minimum floor (for example, 1 second) and a maximum cap (for example, 15 minutes to 1 hour). If the header is missing, unparseable, or absurdly large, fall back to your capped exponential backoff schedule.
  3. Add jitter. Something like min(max_delay, retry_after_delay) + random(0, jitter_ms) prevents many events for the same endpoint from waking on the same millisecond when a rate window ends.

Scheduling Tenant-Aware Retry Delays in Priority Queues

To keep a throttled tenant from starving others, decouple eligibility time (next_attempt_at) from worker execution and isolate per-tenant and per-endpoint pacing:

  • Use a time-ordered delayed queue. Instead of letting a throttled message block a consumer, compute next_attempt_at = now + effective_delay and store the job in a structure keyed by that time. Options include Redis sorted sets, a PostgreSQL table polled with FOR UPDATE SKIP LOCKED on due times, or a managed delay or scheduling feature. Workers pull only jobs whose next_attempt_at <= now.
  • Track per-endpoint rate state. Keep lightweight state per (tenant_id, endpoint_id), such as a rate_limited_until timestamp plus a token bucket or concurrency permit. If endpoint A is limited until T + 60s, its queued and retrying jobs are scheduled for no earlier than that, while workers keep processing tenants B, C, and D.
  • Enforce fair scheduling across tenants. Even without 429s, a tenant bursting 100,000 events can starve smaller tenants in a strict global FIFO queue. Combine weighted fair queuing or round-robin tenant selection with priority tiers (interactive vs. bulk) and per-endpoint concurrency caps.
  • Pair with circuit breaking. If an endpoint repeatedly returns 5xx, times out, or keeps returning 429, open a per-endpoint circuit breaker for a short cooldown. Failing fast avoids wasting connections on a dead or saturated target and sends events straight to the delayed retry tier.
  • Set explicit delivery timeouts. Providers commonly allow roughly 10 to 30 seconds for a response (GitHub documents 10 seconds on GitHub.com, and Svix-based platforms commonly document 15). Bound your own outbound timeouts so slow endpoints can’t hold connections indefinitely, and tell customers to acknowledge quickly and process asynchronously.
Failure / Signal Naive Backoff Behavior Tenant-Aware Priority Queue Behavior
429 with Retry-After: 45 Retries too soon or blocks a worker Sets next_attempt_at = now + 45s + jitter; frees the worker for other tenants
429 with malformed or huge Retry-After Crashes the parser or sleeps for hours Falls back to capped exponential backoff with jitter; logs a warning
High burst from Tenant X Floods the shared queue, delaying Tenant Y Per-tenant fairness and per-endpoint concurrency caps protect Tenant Y
Repeated 503 / timeouts on Endpoint X Hammers a struggling endpoint from many workers Trips the per-endpoint circuit breaker; parks retries in the delayed tier

2. Poison-Pill Dead-Letter Queues: Quarantining Malformed Payloads Before Worker Crashes

What Makes a Payload a “Poison Pill” in Webhook Systems?

A poison pill is a message that repeatedly causes processing or delivery logic to fail, or crashes or hangs a worker, so retrying it is pointless and dangerous. In webhook systems, poison pills typically appear in two places:

  1. Ingestion, transformation, and dispatch workers:
    • Invalid JSON or UTF-8, missing required metadata (tenant_id, event_id, endpoint_url), or schema version mismatches.
    • Oversized payloads that exceed memory buffers, serialization limits, or downstream HTTP size caps, causing OOM kills or repeated serialization exceptions.
    • Destination rejections you have decided are terminal, such as 400, 404, or 410 Gone. Whether these are terminal is a policy choice. Some senders, including Stripe, retry any non-2xx response. A 404 may mean a misconfigured URL that a customer will fix tomorrow, and a 410 is a stronger signal that the endpoint is gone.
  2. Consumer and replay workers:
    • An event that passed earlier stages but triggers an unhandled null dereference, type error, or schema bug in a handler.

If a worker crashes before acknowledging or moving a poison pill and the broker redelivers it immediately, every worker in the consumer group can crash in a loop, stalling the queue or partition.

Quarantine Before Execution: Validation and Guardrail Gates

The cleanest approach is to catch and quarantine poison pills before they reach crash-prone code paths:

  • Pre-enqueue validation: At ingestion, validate envelope structure, content type, signature, and byte size (both Content-Length and the streamed byte count). Quarantine structurally invalid envelopes immediately.
  • Payload size limits and externalized bodies: Enforce hard size ceilings at the edge. If legitimate events are occasionally large, store the body in object storage (such as S3 or GCS) and pass a size-checked reference through the queue, or route large events to a dedicated lane with higher memory limits and streaming HTTP clients.
  • Defensive deserialization and schema validation: Wrap parsing and validation in explicit error boundaries, and classify errors:
    • Transient (network timeout, 429, 502/503/504, temporary DB lock): send to the retry queue.
    • Terminal (corrupt encoding, schema violation, payload over limit, a 4xx you’ve classified as permanent, or retry budget exhausted): send to the dead-letter queue (DLQ) without burning further attempts.
  • Bounded delivery counts: Even for unexpected runtime exceptions, count attempts and enforce a maximum so a crashing message is quarantined after N failures. Managed queues provide this natively. In Amazon SQS, a redrive policy’s maxReceiveCount is the number of times a consumer can receive a message before SQS moves it to the DLQ. Set it high enough to allow genuine retries, and pair it with sensible visibility timeouts.

Designing DLQ Routing Rules That Don’t Break Healthy Replay Batches

A common mistake is dumping every failed message into one flat DLQ with no metadata. When an incident happens and operators run a bulk replay, one unquarantined poison pill can poison the replay job itself, aborting scripts or looping bad events back into active queues.

  • Tag every DLQ record with structured metadata:
    • dlq_reason (for example schema_validation_failed, payload_too_large, retries_exhausted_5xx, endpoint_410_gone, handler_exception)
    • tenant_id, endpoint_id, event_id, original_queue, attempt_count, last_error, first_failed_at, last_failed_at
  • Never mix uninspected poison pills into bulk replay. Filter explicitly by dlq_reason and time window. For instance, replay retries_exhausted_5xx from a known 20-minute outage while leaving schema_validation_failed quarantined until the schema is fixed.
  • Isolate replay workers from live traffic. Use a dedicated replay worker pool or separate queue so a buggy or slow replay can’t starve real-time deliveries.
  • Cap re-DLQ loops. Track a replay_count. If an event fails again during replay, move it to a failed-replay state or alert rather than bouncing it between the DLQ and the main queue forever.
  • Set retention deliberately. Amazon SQS recommends that a DLQ’s retention period always be longer than the source queue’s. For standard queues, a message’s expiry is based on its original enqueue time, which is unchanged when it moves to the DLQ. A message that spent a day in the source queue has less time left in the DLQ than its configured retention suggests. Archive resolved or expiring records to cold storage for auditability.
  • Mind platform constraints. On SQS, a DLQ must be in the same account and Region as its source queue. AWS also cautions against using a DLQ with a FIFO queue if you need to preserve exact ordering.

3. Safe Webhook Replay at Scale: Rate-Limiting Re-Dispatches So You Don’t DDoS Your Own Consumers

Why Naive Replay Overwhelms Downstream Consumers

Suppose a downstream service, yours or a customer’s, suffers a two-hour outage and accumulates 500,000 pending or dead-lettered events. Once the root cause is fixed, someone triggers “Replay All.” If replay is unthrottled:

  1. Instant traffic spike: Instead of the steady trickle the consumer normally handles, it receives a burst of hundreds or thousands of requests per second, often alongside live traffic.
  2. Resource exhaustion: Web server threads, database connection pools, and rate limiters saturate within seconds and begin returning 429 or 503.
  3. Retry amplification: If replayed deliveries trigger aggressive automatic retries, or operators click replay again, the system enters a retry storm that keeps the consumer down.

Engineering Controlled Replay Throttles and Priority Lanes

Treat backlog recovery as a rate-controlled background workload:

  • Per-endpoint and per-tenant throttles: Apply token-bucket or leaky-bucket limits per destination (max R requests/second or C concurrent in-flight deliveries), not just globally. A modest global rate can still overwhelm one small customer endpoint.
  • Honor 429 and Retry-After dynamically: If an endpoint starts returning 429 or 503 during replay, reduce its replay rate using AIMD (additive increase, multiplicative decrease) and honor Retry-After.
  • Separate lanes for live and replay traffic: Give live traffic priority or reserved concurrency so catch-up work never delays new events. Within replay, prioritize recent or high-value events when SLAs demand it.
  • Use managed velocity controls where available: SQS’s DLQ redrive lets you set a custom maximum rate (up to 500 messages per second) and recommends gradually ramping it up while monitoring the destination queue. A redrive task can run for at most 36 hours. Redriven messages are treated as new, with a new message ID and enqueue time, and the retention period resets.
  • Batch with pause/resume controls: Replay in bounded chunks with live visibility into 2xx vs. 429/5xx rates and consumer latency. If errors cross a threshold you define (for example, more than 10% non-2xx), automatically pause or slow the job and alert operators.
  • Treat idempotency as mandatory: Replays mean consumers will see duplicates, since webhook delivery is at-least-once in practice. Consumers must deduplicate on a stable event identifier such as event_id or webhook-id. Standard Webhooks senders keep the same webhook-id across retries while regenerating the timestamp and signature for each attempt, so a late retry still passes timestamp-tolerance checks.
  • Don’t promise ordering: Stripe explicitly states it does not guarantee delivery in the order events are generated, and retries and replays make reordering more likely. Consumers should handle out-of-order and stale events, for example by comparing event versions or re-fetching current state.

Dry-Run Previews and Pre-Flight Checks Before Replaying

A dry run turns high-stakes recovery from guesswork into a predictable operation. Before sending a single live replay request, it should report:

  • Exact event count and byte volume matching the filter (tenant_id, endpoint_id, event_type, dlq_reason, time range).
  • A breakdown by dlq_reason and event type, highlighting poison pills that should be fixed or excluded first.
  • Sample payloads and target endpoints, with secrets and PII redacted, so engineers can verify payload shape, signature headers, and target URLs.
  • Estimated duration at safe rates. For example, 120,000 events sent to a single endpoint at 25 requests/second takes about 80 minutes, before accounting for concurrent live traffic.
  • A pre-flight health check confirming the root cause is resolved, consumer error rates are healthy, and target endpoints are active.

Pre-Replay Operational Checklist

  • Root cause verified: The bug, schema issue, or outage is fixed and confirmed in production.
  • Dry run inspected: Event counts, dlq_reason mix, and estimated completion time have been reviewed.
  • Poison pills filtered out: Malformed or non-retriable events are excluded from automated batches.
  • Per-endpoint rate and concurrency set: Throughput starts conservative, with automatic slowdown on 429/5xx.
  • Live traffic protected: Live dispatch lanes are isolated from, or prioritized above, replay.
  • Kill switch and metrics ready: 2xx rate, 429/5xx rate, queue depth, and consumer latency are monitored, with a one-click pause or abort.

Putting It All Together: A Resilient Webhook Pipeline Flow

When these mechanisms work together, failures stay localized and recovery stays predictable:

  1. Ingress and validation: Verify signatures, envelope integrity, and size limits. Route structurally invalid or oversized poison pills straight to the DLQ with structured reason codes.
  2. Fair, priority dispatch: Schedule deliveries with tenant-aware fairness and per-endpoint concurrency limits, backed by a time-ordered delayed queue keyed on next_attempt_at.
  3. Adaptive retries: On transient failures, honor parsed and clamped Retry-After windows (plus jitter), or fall back to capped exponential backoff with jitter. Trip per-endpoint circuit breakers on sustained failures without blocking other tenants.
  4. Quarantined DLQ and throttled replay: Exhausted or non-retriable events land in a tagged DLQ. After root-cause fixes and a dry-run preview, operators replay filtered batches through rate-limited, monitored replay lanes that protect both live traffic and recovering consumers.

Originally published at https://instawebhook.com/blog/webhook-delivery-retries-queue-mechanics-tenant-aware-scheduling-poison-pill-dlq