A renewal reminder in an education product should wait for the business deadline, then run once. The least complex design that holds up across three worker nodes is a durable work table plus a lease: claim a row atomically, process a small batch, and let an expired lease return the row to the pool.
The lease is the boundary.
That choice gives a solo SaaS team a useful balance. A queue can improve wake-up latency, but it adds delivery semantics and another operational surface. A database poller costs a few seconds of latency in exchange for fewer moving parts. For a reminder that is due at 09:00 on a school administrator's renewal date, that trade usually favors the poller unless the product has a strict sub-second requirement.
| Option | Latency | Main failure to design for | Operational shape |
|---|---|---|---|
| Database lease and polling | Seconds to a minute | Lease expiry during slow work | One durable table and indexed query |
| Queue with delayed delivery | Usually lower | Visibility timeout and duplicate delivery | Queue plus workers plus retry policy |
| One scheduler assigning jobs | Low at dispatch | Scheduler becomes a bottleneck | Leader election or a single active scheduler |
My rule is blunt: use the database lease when the cleanup is already database-shaped and a few seconds do not change the outcome. Move to a queue when the work is bursty, the database should not wake workers, or latency has a product-level consequence. This is a revenue-per-hour decision. Every extra service deserves a reason that survives a weekly shipping cycle.
How do I implement distributed data cleanup across multiple workers?
They should see the same due set, but only one should own each row. A normal read followed by an update has a race window: worker A reads reminder 42, worker B reads reminder 42, and both send it. A process-local mutex does not help because the processes live on different nodes.
Use a conditional claim instead. The claim needs a stable identifier, a due timestamp, a status, an attempt counter, and a lease deadline. The important invariant is not “the job runs exactly once.” That promise is usually too strong once a process can send a message and die before recording success. The practical invariant is “a job has one active owner at a time, and the side effect is idempotent.”
The reminder row can represent a delayed business action without making the scheduler understand the education domain. For example, a row might contain due_at = 2026-10-24T09:00:00Z, kind = renewal-reminder, and a deterministic key such as school_184:term_2026:renewal. The date is data. The worker only decides what is due and what is safely claimable.
That is enough for the first implementation.
A five-second lease is too short for an API call with an ordinary retry. An hour is unnecessarily sticky when a node disappears. Pick a lease from the longest expected unit of work plus a margin, then measure lease expirations. The exact number belongs to the workload, not to a tutorial.
The failure sequence is easy to miss. Worker A claims a reminder, calls the notification system, and loses its connection before it can mark the row done. The lease expires. Worker B claims the same row and calls the notification system again. No lock can undo a side effect that happened outside the database. The durable fix is to carry the deterministic reminder key into the side-effect boundary, so the notification operation can recognize a repeated request, or to write an outbox record inside the same transaction that claims the business action and let a separate sender own delivery. The outbox adds storage and another loop, so I would not add it for every cleanup job. I would add it when duplicate reminders have a material support cost or when the downstream system provides an idempotency key. This is the kind of boundary that changes the implementation decision: ownership protects the row, while idempotency protects the outside world.
How do latency and cost change the design?
Polling every second gives a tighter deadline but creates repeated reads on every idle worker. Polling every minute reduces that load and makes the reminder less precise. Neither number is universally correct. Start from the business tolerance: if a reminder can arrive within 30 seconds of the deadline, a 10-second poll interval leaves room for claim and delivery time. If the deadline is a legal or contractual boundary, record the actual dispatch timestamp and define what late means.
The queue alternative has a different clock. A worker receives a message, processes it, and the message becomes visible again when its visibility timeout expires unless the worker deletes or extends it. That is a recovery mechanism, not a duplicate-prevention guarantee. A timeout shorter than the work can produce concurrent attempts; a timeout much longer than the work delays recovery after a crashed node.
The limitation is deliberate: a database lease is a poor fit for high-volume bursts that should not wake every worker. In that case, the queue's extra moving parts are the better trade-off because they move the wake-up cost away from the database.
I initially treated “more workers” as a latency fix. The queue and database still had to agree on ownership, and the slowest external call became the real limit. The better unit to optimize was the claim batch. A batch of 20 keeps transactions short and gives a worker room to finish before its lease needs renewal. Larger batches increase throughput until one stuck record holds too much work.
A minimal TypeScript lease
The application code should make the ownership boundary visible. The storage-specific transaction is represented by a small interface here; the same contract can sit over a relational database or another durable store with conditional updates. The worker does not fetch a row and hope it still owns it. It claims only rows that are due and unleased, or whose previous lease has expired.
type CleanupJob = { id: string; dueAt: Date; leaseUntil: Date | null; attempts: number; }; type JobStore = { claimDue(limit: number, leaseUntil: Date): Promise<CleanupJob[]>; markDone(id: string, workerId: string): Promise<void>; release(id: string, workerId: string, reason: string): Promise<void>; }; async function runBatch(store: JobStore, workerId: string): Promise<void> { const now = new Date(); const leaseUntil = new Date(now.getTime() + 60_000); const jobs = await store.claimDue(20, leaseUntil); for (const job of jobs) { try { await performIdempotentCleanup(job); await store.markDone(job.id, workerId); } catch (error) { await store.release(job.id, workerId, String(error)); } } } async function performIdempotentCleanup(job: CleanupJob): Promise<void> { // Use job.id or a domain key as the idempotency key for the side effect. void job; } claimDue must be atomic from the store's point of view. In a relational implementation, that commonly means a transaction that locks eligible rows, skips rows already locked by another worker, and updates the lease before returning the rows. The exact SQL is less important than the contract: two successful claim calls cannot return the same active lease.
A worker should renew a lease only while it is still the owner. It should also stop work when renewal fails. Continuing after losing ownership is how a clean concurrency model turns into duplicate reminders. After the side effect succeeds, markDone should verify the worker identity and lease state; a late worker must not mark another worker's replacement attempt as complete.
What does a failure-safe cleanup workflow measure?
Count due rows, successful claims, lease expirations, attempts, completed jobs, and dispatch delay. Keep the fields tied to decisions. A growing due count says capacity or upstream latency is wrong. Lease expirations say the work duration or lease policy needs attention. A high attempt count points to a poison record or an external dependency, not automatically to a need for more nodes.
Keep cleanup bounded. Delete or archive in chunks, commit between chunks, and make each chunk restartable. A single transaction that removes millions of records creates a long lease, lock pressure, and a painful recovery path. A partial cleanup is acceptable if the next run can safely continue; an ambiguous cleanup is not.
Test the events that matter: two workers claiming the same due row, a worker dying after the side effect, a lease expiring during a slow batch, a clock being slightly ahead, and a retry after a transient storage error. Use a fake clock for scheduling decisions, then run one integration test against the real locking behavior. The test suite should prove ownership; it should not merely prove that a loop calls a function.
When is the runner-up the better choice?
Choose a queue when the due work arrives in bursts, polling would repeatedly hit a hot database, or the product requires a tighter dispatch window than polling can provide. Set the visibility timeout from observed processing time, extend it for legitimately long work, and make the handler idempotent because delivery can repeat. The public visibility-timeout model documents this as temporary hiding, not deletion: a message can become available again when the timeout ends.
Choose a single active scheduler when there are many kinds of jobs but only one small dispatch decision. It still needs a takeover path. A lease on the scheduler itself is the same idea at a different level, and a stalled leader must not block renewal reminders indefinitely.
For a one-person SaaS, I would ship the leased database worker first when the deadline tolerance is measured in seconds or minutes. It keeps the failure model close to the data and leaves room to outsource undifferentiated queue plumbing later. The decision should be revisited when measurements show that polling, not the business workflow, is consuming the team's revenue-per-hour budget.
The useful design is therefore modest: durable due records, atomic ownership, bounded work, idempotent effects, and metrics that expose lateness. Three workers are enough to exercise the failure modes. More nodes do not remove them.
Further reading
- AWS SQS visibility timeout documentation: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
- PostgreSQL transaction isolation documentation: https://www.postgresql.org/docs/current/transaction-iso.html
- MDN Date documentation: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date
