For failed job retries in Node.js, the AWS SQS versus RabbitMQ choice is secondary to the recovery boundary: use a durable work queue for shipment-update fanout, but make retry ownership explicit. The queue transports attempts, while the application records subscriber state and decides when recovery is complete. That boundary matters more than choosing a familiar broker.
For a customer-support system, one tracking event may need to reach email, SMS, chat, and an internal case timeline. A retry that republishes the whole event can duplicate the channels that already succeeded. I would model one delivery record per subscriber, carry an idempotency key through every attempt, and make a failed delivery recoverable without replaying successful work.
The invariants are plain:
- A shipment event is immutable after acceptance.
- Each subscriber delivery has a bounded attempt history and a visible next-attempt time.
- A worker can crash after the provider accepts a message without corrupting the delivery record.
- Operators can find, pause, and replay one delivery without releasing the entire backlog.
The recovery boundary is the delivery record, not the original shipment event.
How should failed job retries cross queue boundaries?
There are three useful places to put retry responsibility. A broker can redeliver a message after a visibility or acknowledgement deadline. A worker can schedule a delayed attempt. Or a durable table can hold the next attempt and a dispatcher can claim due rows. Each option has a different recovery story.
| Boundary | What it does well | Failure to design around | Good fit |
|---|---|---|---|
| Queue redelivery | Keeps transient transport failures off the application path | A crash can make an accepted side effect look unacknowledged | Short, idempotent attempts |
| Worker delay | Gives code precise backoff and classification | A process restart can erase in-memory timers | Small, disposable delays |
| Durable delivery record | Makes inspection, replay, and per-subscriber recovery explicit | Requires claiming, indexes, and cleanup | Customer-visible fanout |
For shipment notifications, I use the queue as the wake-up mechanism and the delivery record as the source of truth. A message says “try this delivery”; it does not say “this delivery succeeded.” That distinction is the difference between operational recovery and hopeful logging.
The reader’s familiar comparison is often a managed queue service versus a broker with richer routing. Those are real differences, but they are downstream of the recovery boundary. A managed queue commonly supplies visibility timeouts and dead-letter behavior. RabbitMQ supplies acknowledgements, exchanges, queues, and routing controls. Kafka supplies partitioned ordered logs and replay semantics. Redis Streams supplies consumer groups and pending-entry inspection. None of those facts alone records that subscriber 7 received the SMS while subscriber 8 still needs a retry. Your application still has to own that state.
The critical path: claim, attempt, settle
The worker should claim one delivery with a 2-minute lease, perform one side effect, and settle the result. The lease is deliberately separate from the provider idempotency key: a lease prevents two workers from doing normal work at once, while the key makes an ambiguous repeat safe when a crash occurs after acceptance. That is a real trade-off: a short lease limits duplicate work during slow calls, while a long lease delays recovery after a dead worker.
from dataclasses import dataclass from datetime import datetime, timedelta, timezone @dataclass class Delivery: delivery_id: str event_id: str channel: str attempt: int lease_until: datetime | None def next_delay(attempt: int) -> timedelta: seconds = min(900, 2 ** min(attempt, 9)) return timedelta(seconds=seconds) def process(delivery: Delivery, store, provider, now: datetime) -> None: if delivery.lease_until and delivery.lease_until > now: return lease_until = now + timedelta(minutes=2) store.claim(delivery.delivery_id, lease_until=lease_until) key = f"shipment:{delivery.event_id}:delivery:{delivery.delivery_id}" try: provider.send(channel=delivery.channel, idempotency_key=key) except provider.TransientError as error: store.reschedule( delivery.delivery_id, next_at=now + next_delay(delivery.attempt + 1), reason=str(error), ) except provider.PermanentError as error: store.dead_letter(delivery.delivery_id, reason=str(error)) else: store.mark_delivered(delivery.delivery_id, delivered_at=now) This code leaves an uncomfortable but necessary case visible: the worker can die after provider.send returns and before mark_delivered. The next worker must be allowed to try again, and the provider-facing operation must tolerate that repeat. If the downstream API has no idempotency facility, store a deterministic request token where that provider can check it, or accept that the system can provide at-least-once delivery rather than pretending it has exactly-once behavior.
The ambiguity is the design.
Do not classify every exception as transient. A malformed phone number, revoked consent, or unsupported channel will not become healthy because the clock advanced. Those should move to a terminal state with enough context for an operator or policy process to act. Timeouts, connection resets, and temporary capacity responses are candidates for retry, subject to a cap and a provider-specific policy.
What I would reject, and when it is still valid
I would reject a single timer job that wakes every five minutes, loads all failed shipment events, and republishes each event wholesale. It creates a thundering herd at the boundary, obscures which channel failed, and turns one bad address into repeated work for every healthy subscriber. It also makes a restart question unnecessarily large: should the process replay everything it loaded before it stopped?
That design is still valid for a low-volume batch where the unit of work is genuinely the whole batch, the side effect is idempotent, and an operator can replay the batch safely. It is a poor fit for selective customer notification recovery, but a nightly reconciliation task has those properties more often than customer-facing fanout does. The decision is about the recovery unit, not whether a timer is elegant.
A second tempting shortcut is to rely only on dead-letter messages. A dead-letter queue is useful evidence and a pressure valve, but it is not a complete repair workflow. Retain the original event ID, delivery ID, attempt count, classification, and last error. Give operators a replay action that creates a new attempt under the same logical delivery, rather than copying opaque payloads into a new stream.
Operating the comparison after deployment
Measure recovery, not just throughput. I would watch age of the oldest due delivery, retry rate by channel and error class, lease-expiry count, dead-letter growth, and the time from terminal failure to human disposition. A queue that reports low latency while the delivery table accumulates abandoned leases is not healthy.
Test the ugly transitions directly: kill a worker after the provider call, delay acknowledgements, return a permanent error on attempt one, and replay a delivery while another worker owns its lease. The expected result is boring: no duplicate customer-visible notification, no lost delivery, and an audit trail that explains the decision.
The practical selection rule is therefore narrow. Choose the transport whose acknowledgement, delay, ordering, and inspection behavior matches your workload, then put subscriber-level state outside that transport when recovery must be selective. Compare managed queues, AMQP brokers, logs, and stream systems by their failure boundaries and operator workflow. Product vocabulary changes; these invariants do not.
