To configure a queued cleanup task for a weekly customer-support digest, put a run ID in the message and recover work through an explicit retry policy. The deciding constraint is operational recovery: a cleanup failure must remain visible and replayable without sending the same digest twice or deleting a record that a later run still needs.

Short answer: put only a time window and a run identifier in the scheduled message. Let the worker claim work in small batches, make each mutation idempotent, retry transient failures with bounded backoff, and move exhausted messages to a quarantine queue for inspection. A scheduler is a clock. It is not a recovery system.

The constraint that changed the design

The concrete job is easy to describe: every week, send a digest to customers who are active, then remove temporary data that has passed its retention point. The dangerous part is the boundary between those two actions. A process can send the digest and crash before cleanup. It can clean a page of records and lose its lease before acknowledging the message. It can retry after a network timeout even though the remote side already accepted the request.

I initially treated the weekly trigger as the main design decision. It was not. The useful unit was a recovery record: one digest period, one deterministic run ID, and a set of cleanup batches that can be safely attempted again. That makes a missed worker run an operational state to repair, instead of a mystery hidden in scheduler logs.

A cron entry can express a periodic command, but the crontab manual does not provide the queue semantics, acknowledgement state, or application-level idempotency needed here. Keep the schedule boring. Put recovery in the application.

The run contract can be this small:

type DigestRun = { runId: string; periodStart: string; periodEnd: string; customerIds: string[]; }; function cleanupMessage(run: DigestRun, batchNumber: number, recordIds: string[]) { return { kind: "support-digest-cleanup", messageId: `${run.runId}:${batchNumber}`, runId: run.runId, periodEnd: run.periodEnd, recordIds, }; }

The message carries intent, not a snapshot of every mutable field. The worker can validate the period, look up current state, and record the result under messageId. A stable ID also gives an operator something precise to replay.

What should a failed execution do?

A retry is useful only when the failure class makes another attempt plausible. A timeout talking to a dependency is different from malformed input. A permission failure is different again. Treating all failures as “try five times” creates noise and delays the real repair.

I use three recovery boundaries:

  1. The worker retries transient failures with bounded exponential backoff and a small random delay.
  2. The queue stops automatic retries after a maximum attempt count and preserves the original error.
  3. A quarantine consumer records the message, run ID, attempt history, and next action for an operator.

The exact count is a policy choice, not a magic constant. I would cap automatic attempts at 5 for this weekly job, then preserve the message for review. A retry window measured in hours may be reasonable; for a cleanup that protects a retention boundary, blindly retrying for days may be worse than stopping and asking for review. The important rule is that the schedule must not create a second independent cleanup run while the first one is still recoverable.

Here is the worker shape. The queue adapter is deliberately generic so the retry policy stays testable.

type CleanupMessage = { kind: "support-digest-cleanup"; messageId: string; runId: string; periodEnd: string; recordIds: string[]; }; type RetryDecision = | { action: "retry"; delayMs: number } | { action: "quarantine"; reason: string }; function retryDecision(error: unknown, attempt: number): RetryDecision { if (!isTransient(error) || attempt >= 5) { return { action: "quarantine", reason: describe(error) }; } const base = Math.min(60 * 60 * 1000, 2 ** attempt * 1_000); const jitter = Math.floor(Math.random() * 500); return { action: "retry", delayMs: base + jitter }; } async function handleCleanup(message: CleanupMessage, attempt: number) { if (await alreadyCompleted(message.messageId)) return; try { await markStarted(message.messageId, attempt); await deleteEligibleRecords(message.recordIds, message.periodEnd); await markCompleted(message.messageId); } catch (error) { const decision = retryDecision(error, attempt); await recordFailure(message.messageId, attempt, error, decision); throw decision; } }

The alreadyCompleted check is not enough by itself. It needs a durable uniqueness rule, and the delete operation must be safe to repeat. For example, filtering by the retention timestamp on every attempt means a second attempt sees fewer eligible rows rather than inventing a new side effect. Use a transaction for the state change and completion marker when the storage system supports that boundary.

How should I configure queued cleanup task message retries?

Cleanup and delivery should not share an optimistic assumption that one acknowledgement covers both. The digest needs its own idempotency key, such as customerId:periodEnd, stored with the delivery result. The cleanup message can then be retried independently without re-sending a completed digest.

This is the awkward but useful split:

  • The scheduler creates or finds the weekly run.
  • The digest stage records delivery intent and outcome per customer.
  • The cleanup stage removes only data eligible for that run.
  • The acknowledgement happens after the stage’s durable result is recorded.

That ordering makes a crash boring. A worker may execute a message twice, but a repeated delivery key returns the recorded result, and a repeated cleanup query finds no newly eligible copy of an already removed record. At-least-once delivery is common in queue systems; application idempotency is what makes it survivable.

Do not put a giant customer list into one message just because the weekly run is logically one job. Batch it. A failed batch should be replayable without re-running every customer, and the batch size should be chosen from observed processing time and dependency limits. I would start with a deliberately modest batch of 100 records, measure it, then change one variable at a time.

The smallest working implementation

The producer needs a duplicate guard. It should derive the run ID from the digest period rather than from the scheduler’s invocation timestamp, because a restarted scheduler must resolve to the same logical run. The following is the important shape, with persistence and queue calls left behind interfaces:

async function enqueueWeeklyCleanup(clock: () => Date) { const periodEnd = clock().toISOString().slice(0, 10); const runId = `support-digest:${periodEnd}`; const run = await createRunIfMissing(runId, periodEnd); if (run.cleanupEnqueued) return runId; const recordIds = await findEligibleRecordIds(periodEnd); for (let offset = 0, batch = 0; offset < recordIds.length; offset += 100, batch += 1) { const ids = recordIds.slice(offset, offset + 100); await publish(cleanupMessage( { runId, periodStart: run.periodStart, periodEnd, customerIds: [] }, batch, ids, )); } await markCleanupEnqueued(runId); return runId; }

The 100 here is an example batch size, not a universal answer. It is also a useful test seam. A fake clock can produce the same runId twice; the test should prove that the second invocation does not publish a second set of messages. Another test should inject a timeout after the deletion and before acknowledgement, then run the same message again and verify the final state is unchanged.

Observe four things: schedule lag, oldest queued message age, retry count by failure class, and quarantine depth. A green scheduler process with a growing quarantine is not healthy. Add a run-level view that answers three questions quickly: which period is incomplete, which batch stopped, and whether replay is safe.

What I would change at scale

At higher volume, I would separate digest delivery from cleanup capacity. They have different dependency limits and different operator urgency. I would also add leases or a claim token for batch ownership, with an expiry long enough for normal work and short enough to recover a dead worker. The lease does not replace idempotency; it reduces unnecessary concurrent work.

I would keep retention decisions server-side. A message can carry a cutoff, but the worker should validate it against the run record before deleting anything. This protects against stale messages and makes a replay an explicit operation rather than a hidden consequence of an old scheduler invocation.

The trade-off is state. Run records, delivery keys, attempt history, and quarantine entries add tables and queries. That is config and code I would accept. A dozen queue settings cannot compensate for missing recovery state.

The decision rule is simple: choose the smallest queue and scheduler arrangement that can prove duplicate runs are harmless, transient failures are delayed, permanent failures are visible, and an operator can replay one failed batch. If it cannot show those four properties, it is only moving work around.

References