Nightly payment reconciliation forces a latency-versus-cost choice: create each background job in a queue that a bounded Node.js worker pool can drain without hammering the provider.
Short answer: create a durable background job queue, let a Node.js worker consume one bounded reconciliation slice at a time, ack only after an atomic idempotency record is committed, nack retryable failures with delay, and send exhausted or permanent failures to a dead letter queue.
The queue library matters less than those state transitions. I would benchmark end-to-end completion time, provider calls per settled account, and retry volume before tuning concurrency. Otherwise, “faster” is just a guess with a larger bill attached.
Freeze the business key before choosing a queue
The unit of work should be a stable slice, not “reconcile everything.” In this example, one job represents one merchant account for one settlement date. Its identity is merchantId + settlementDate; its payload contains no credentials and no mutable retry counter supplied by a caller. That gives the worker a useful idempotency boundary and keeps a single bad account from replaying the entire nightly run.
Latency and cost pull in opposite directions. More workers can drain a backlog faster, yet the payment provider may answer with HTTP 429 Too Many Requests. MDN notes that a 429 response may include Retry-After, so a consumer should honor that signal rather than immediately creating more traffic. A fixed worker count is still a starting assumption — not a law. The right count depends on observed service time and the provider's limits, and I'm not sure a static value remains right through month-end volume without load data.
Keep the state machine small:
- A scheduler creates one deterministic job per account and settlement date.
- A worker consumes a job and checks the durable idempotency record.
- The worker fetches the provider settlement and commits the reconciliation result plus idempotency key atomically.
- It acknowledges success. A retryable response is negatively acknowledged for delayed retry; a permanent failure or exhausted attempt goes to the dead letter queue.
No config maze.
How should a Node.js background job queue worker consume, ack, and retry?
Here is the smallest working model I use to inspect the transitions. It has no dependency and runs with a TypeScript runner. The in-memory arrays are deliberately educational, not durable infrastructure; the QueuePort boundary is where a production broker or database-backed queue belongs.
type ReconcilePayload = { merchantId: string; settlementDate: string; }; type Job = { id: string; payload: ReconcilePayload; attempts: number; maxAttempts: number; }; type Failure = { retryable: boolean; retryAfterMs?: number; reason: string; }; const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms)); class QueuePort { private ready: Job[] = []; private inFlight = new Map<string, Job>(); readonly deadLetters: Array<{ job: Job; reason: string }> = []; create(job: Job): void { this.ready.push(job); } consume(): Job | undefined { const job = this.ready.shift(); if (job) this.inFlight.set(job.id, job); return job; } ack(job: Job): void { this.inFlight.delete(job.id); } async nack(job: Job, failure: Failure): Promise<void> { this.inFlight.delete(job.id); const next = { ...job, attempts: job.attempts + 1 }; if (failure.retryable && next.attempts < next.maxAttempts) { await sleep(failure.retryAfterMs ?? 250); this.ready.push(next); return; } this.deadLetters.push({ job: next, reason: failure.reason }); } get size(): number { return this.ready.length + this.inFlight.size; } } class IdempotencyStore { private completed = new Set<string>(); has(key: string): boolean { return this.completed.has(key); } commitWithResult(key: string): void { // In production, persist the result and unique key in one transaction. this.completed.add(key); } } const queue = new QueuePort(); const completed = new IdempotencyStore(); const providerCalls = new Map<string, number>(); async function reconcile(job: Job): Promise<Failure | undefined> { const calls = (providerCalls.get(job.id) ?? 0) + 1; providerCalls.set(job.id, calls); // The demo makes one account rate-limited once, then successful. if (job.payload.merchantId === "merchant-b" && calls === 1) { return { retryable: true, retryAfterMs: 25, reason: "HTTP 429" }; } return undefined; } async function runWorker(): Promise<void> { while (queue.size > 0) { const job = queue.consume(); if (!job) continue; const key = `${job.payload.merchantId}:${job.payload.settlementDate}`; if (completed.has(key)) { queue.ack(job); continue; } const failure = await reconcile(job); if (failure) { await queue.nack(job, failure); continue; } completed.commitWithResult(key); queue.ack(job); } } for (const merchantId of ["merchant-a", "merchant-b"]) { const settlementDate = "2026-08-18"; queue.create({ id: `${merchantId}:${settlementDate}`, payload: { merchantId, settlementDate }, attempts: 0, maxAttempts: 3, }); } await runWorker(); console.log({ deadLetters: queue.deadLetters.length, providerCalls: [...providerCalls] }); The interesting line isn't consume(). It is commitWithResult(). A real implementation needs a unique constraint on the idempotency key and a transaction that writes both the reconciliation result and completion marker. If the process disappears after the provider call but before that commit, the broker can deliver the job again. RabbitMQ's acknowledgement guide explicitly describes automatic requeueing of unacknowledged deliveries after a connection or channel closes, and warns consumers to handle redelivery. Acking before the commit flips the failure mode: now a crash can lose work.
This sample retries a synthetic 429 once so the behavior is visible. In production, classify failures narrowly. Rate limiting and a temporary network interruption can be retryable; invalid account input is not repaired by a fourth attempt. Keep the provider's response details in restricted logs, while the dead-letter record carries the job identity, attempt count, reason category, and timestamps needed for triage.
Run the duplicate and 429 drills
Before tuning concurrency, run two boring failure tests. First, create the same merchant/date job twice. The second delivery should observe the committed idempotency key, skip the provider call, and ack. Then interrupt a worker after it consumes a job but before it acknowledges; when the queue redelivers, the final stored result must still be singular. That test catches the expensive mistake of treating queue deduplication as a durable business invariant.
Second, force the provider adapter to return a 429 with a retry delay. The attempt count should rise once, the job should become eligible no earlier than the delay, and unrelated merchants should continue. Force enough retryable responses to reach maxAttempts; exactly one dead-letter entry should remain available for inspection. Then feed invalid account input and verify that it goes there without burning the retry budget. These are state-transition tests, not vendor integration tests, so they should run quickly and deterministically.
The scheduler gets its own duplicate test. It should calculate the settlement date explicitly, enumerate eligible merchant accounts from a consistent snapshot, and create the same deterministic job ID when invoked twice. Time zones deserve suspicion — define the business cutoff in the merchant's settlement zone, put the chosen date in the payload, and never let a worker recalculate “yesterday” when it eventually consumes the job.
Measure the night, then turn one knob
Capture one row per run: first enqueue time, final ack time, oldest-job age, provider calls, 429 responses, retry attempts, permanent failures, and dead-letter count. Segment service time by account size if that input varies. I care about the ratio of provider calls to completed accounts because it makes retry waste visible; total duration alone can reward a noisy configuration that merely pushes harder against a rate limit.
Change only worker concurrency for the first comparison. Keep the account snapshot, enqueue pacing, retry policy, and provider test conditions fixed. Raise concurrency until completion latency stops improving or calls per completed account begin climbing, then back off. Your mileage may vary because the payment provider's limits and account distributions set the curve; the experiment, rather than a copied worker count, resolves that uncertainty.
At larger volume, sleeping inside nack() distorts the measurement by holding consumer capacity. Move delayed retries into a durable delivery schedule and free the worker immediately. Partition by merchant ID, limit global and per-provider concurrency separately, and emit created, started, retried, acked, and dead_lettered events. Alert on backlog age and final failure. Raw queue length by itself cannot tell a healthy scheduled burst from a stalled run.
Where this build stops fitting
The in-memory adapter is only a state-machine demonstration. It is not suitable for production because process exit erases jobs, idempotency keys, and dead letters. Replace it with durable storage, enforce a unique business key, commit the reconciliation result and completion marker in one transaction, and make replay an explicit, audited operation.
At-least-once delivery plus idempotent processing is a practical fit when completing every account matters more than avoiding duplicate delivery. The catch is the transaction boundary. If reconciliation changes two systems that cannot share a transaction, a local idempotency table cannot promise an atomic cross-system outcome; each remote write needs idempotent semantics where its contract permits, plus a documented recovery process.
Stick with a database-backed job table when volume is modest, the team already operates that database, and transactional enqueue with local business data matters more than sub-second dispatch. A dedicated broker fits independent scaling, richer delivery controls, or sustained queue traffic, but adds an operational component. A scheduler alone is the wrong tool when each job needs acknowledgement, retry state, and dead-letter inspection; it can trigger creation, while the queue owns delivery.
For this nightly workload, start with low bounded concurrency and provider-directed retry delays. Raise one knob only while the scorecard improves. Fast enough wins.
References
- RabbitMQ, “Consumer Acknowledgements and Publisher Confirms”: https://www.rabbitmq.com/docs/confirms
- MDN, “429 Too Many Requests”: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
