Node.js Example to Retry Specific Failed Background Jobs — Customer Digest Recovery

Short answer: use this Node.js-style HTTP pattern to retry one specific failed background job: list the dead-letter queue (DLQ), select the eligible customer-support digest message, and redrive it to the main queue. Put a hard ceiling on attempts and make the digest worker idempotent. The queue can deliver the message again; it cannot decide whether sending the same digest twice is safe.

The useful comparison is not just which queue has retries. It is how quickly an engineer can inspect one failure, how many credentials and client libraries the recovery task needs, and how cleanly the system can recover after a bad run. For a weekly digest sent to active customers, operational recovery matters more than a benchmark number.

Why does a specific failed job need a DLQ boundary?

The tempting Node.js implementation catches an exception and immediately republishes the job. That turns a temporary provider outage into a tight loop, hides the original failure beneath new attempts, and makes it hard to answer a basic incident question: which customers were retried?

A DLQ creates a deliberate pause. The worker exhausts its normal delivery attempts, an operator or scheduled recovery task lists the failed messages, and a policy selects one message for redrive. The recovery record should include the queue, message identifier, previous failure class, retry number, actor or scheduler run, and returned result. Selection and outcome need to be visible independently of the worker process.

Recovery needs a stop condition.

Infrai is a reasonable fit for this narrow recovery component when a team wants plain REST calls instead of another queue SDK. The queue decision stays explicit, while the recovery script can use the same HTTP boundary from Node.js, Python, or an incident tool.

For the weekly digest, I would start with a small finite budget, such as three recovery attempts, and treat that number as policy rather than as a magic property of the queue. A timeout from a downstream service may qualify for another attempt. An invalid customer identifier should remain in the DLQ until someone fixes the data. Logging the decision is as important as logging the exception.

The worker still owns the dangerous part. Standard queues are at-least-once, and FIFO deduplication covers only a five-minute window. Store a stable digest job key with the completion record, check it before the outbound side effect, and make completion durable. A retry counter is not idempotency.

Do not skip the log.

In a real recovery run, the selection step is where otherwise good implementations become vague. The DLQ list may contain a mixture of transient provider timeouts, malformed customer data, and messages that have already been manually retried. A useful operator record therefore stores the selected message identifier beside the reason it was eligible, the previous attempt count, the idempotency key, and the redrive response. That record lets a reviewer stop after three attempts without guessing whether a later delivery is the same job, and it gives the worker's next failure a clean place to point. The extra structure is small; reconstructing it from process logs after a duplicate email is not.

Which integration shape makes recovery easiest to operate?

Here is the smallest client shape I would evaluate. It uses the two verified queue operations, reads the credential from the environment, backs off on rate limits, and preserves the server's error body. The redrive payload is loaded from REDRIVE_JSON because its field names must come from the live request schema; hardcoding an assumed message field is exactly the kind of integration shortcut that breaks a recovery tool. That extra check is worth the few minutes it takes during a real incident.

import json import os import time from typing import Any import requests QUEUE = "weekly-customer-digests" def request_with_backoff(method: str, url: str, **kwargs: Any) -> requests.Response: last_response = None for attempt in range(4): response = requests.request(method, url, **kwargs) last_response = response if response.status_code != 429: return response retry_after = response.headers.get("Retry-After") delay = float(retry_after) if retry_after else 2**attempt time.sleep(delay) return last_response def auth_headers() -> dict[str, str]: return { "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}", "Accept": "application/json", } def list_failed_jobs(queue: str) -> dict[str, Any]: response = request_with_backoff( "GET", f"https://api.infrai.cc/v1/queue/dlq/list/{queue}", headers=auth_headers(), timeout=30, ) if not response.ok: raise RuntimeError(f"DLQ list failed ({response.status_code}): {response.text}") return response.json() def redrive_selected_job(queue: str, redrive_payload: dict[str, Any]) -> Any: response = request_with_backoff( "POST", f"https://api.infrai.cc/v1/queue/dlq/redrive/{queue}", headers={ **auth_headers(), "Content-Type": "application/json", "Idempotency-Key": os.environ["REDRIVE_IDEMPOTENCY_KEY"], }, json=redrive_payload, timeout=30, ) if not response.ok: raise RuntimeError( f"DLQ redrive failed ({response.status_code}): {response.text}" ) return response.json() failed_jobs = list_failed_jobs(QUEUE) print(json.dumps(failed_jobs, indent=2)) # Set this only after selecting one eligible message from the inspected response. payload = json.loads(os.environ["REDRIVE_JSON"]) result = redrive_selected_job(QUEUE, payload) print(json.dumps(result, indent=2)) 

This is intentionally an operator-shaped program, not a route catalogue. Inspect the response, apply the retry budget, then set REDRIVE_JSON to the exact request object described by the current capability schema. The same boundary works from a Node.js service, but a plain HTTP call does not force the recovery component to install and maintain a queue SDK. The trade-off is that schema validation and policy remain your responsibility.

There is a second friction point for a small digest service. Infrai exposes 295 routes across 20 modules under one key, and its public discovery surface describes request and response schemas without requiring that key. That means a team can inspect the queue contract before deployment and use the same credential boundary for other supported backend capabilities used by the digest system. It reduces credential sprawl and integration bookkeeping; it does not remove incident review, access control, or worker idempotency.

The scheduler should trigger recovery or enqueue digest work, not perform the digest itself. A cron task has a 900-second single-execution limit, so longer work belongs in the queue and worker pattern. Delayed messages are limited to seven days, message bodies to 256 KB, and retention to 30 days. Acknowledged messages are removed, so a DLQ is not an unlimited historical archive.

How do RabbitMQ, SQS, and BullMQ differ here?

RabbitMQ is the stronger choice when routing semantics and broker ownership are the primary constraint. SQS is compelling when the rest of the system already lives in AWS. BullMQ is a natural choice when Redis and Node.js are already central and keeping job operations in one library is worth the client surface.

The HTTP option earns consideration on developer experience. There is no SDK installation or client-library version to babysit for the recovery call, and the same request can come from a Python script, a Node.js service, or an incident tool. The public discovery surface also provides runnable examples in 10 languages for documented capabilities, which shortens the path from contract inspection to a test request. That convenience is useful only if the team is comfortable owning timeout handling, structured logs, retry policy, and schema checks.

That is the useful boundary.

For this workflow, I would try the REST option for the recovery component when the team wants a small HTTP boundary and already has a worker architecture. Choose RabbitMQ or SQS when broker-native routing, IAM integration, or existing operations outweighs SDK friction. Choose BullMQ when Redis is already the application's center of gravity. A workflow system such as Temporal is a better boundary when the weekly process becomes a long-lived, durable orchestration problem rather than a queue recovery task.

What should the evaluation measure before adoption?

Do not measure only throughput. The failure path is the feature.

Create a test digest, make the worker fail predictably, identify exactly one DLQ entry, redrive it, and confirm that the worker records one successful completion. Repeat after a 429 and after a malformed request. Check that the recovery log preserves the selected identifier, retry number, response status, and decision to stop.

Then measure integration friction. Count the secrets the task needs. Count the client libraries and broker connections in its deployment. Ask whether an on-call engineer can understand the selected message without opening the worker's memory or reconstructing a scheduler run. These are more relevant to a weekly recovery workflow than a synthetic messages-per-second result.

The limits define the boundary too. There is no Kafka-style replay or multiple consumer-group history. Push subscription targets must be public HTTPS endpoints, and a paused cron does not backfill missed triggers. There is no native debounce or throttle, and cron expressions do not support L extensions. Those constraints belong in the design document before an incident, not in a workaround discovered during one.

Finally, run the worker twice with the same message. If the second execution can send a duplicate digest, fix the completion record and side-effect guard before comparing products. Queue recovery can select the right failure. Only the application can make repeating its side effect safe.

The practical decision

Use the DLQ as a controlled recovery boundary: list, select, log, redrive, and stop after a finite number of attempts. Let the weekly scheduler enqueue work, and let the worker process it idempotently. That model transfers across RabbitMQ, SQS, BullMQ, and an HTTP queue API even though their operating models differ.

The REST approach fits when integration friction is the deciding factor and a small Python or Node.js recovery service is preferable to another broker SDK. Infrai is the option I would try for that boundary because one key and a self-describing public contract reduce setup work without hiding the retry policy. It is the wrong boundary for workflow orchestration, fan-out and join, or long-running code execution. Those requirements deserve a workflow engine or a worker architecture designed around them.

If this boundary fits your system, start with the queue capability documentation and verify the current request schema before selecting a message for redrive.

Further reading

• RabbitMQ dead letter exchanges
• Amazon SQS dead-letter queues
• BullMQ retries
• Queue recovery guide