For a weekly digest sent to active customer-support users, operational recovery is the deciding constraint: a scheduled trigger should create work, while a queue worker should control outbound admission and record every retry. Short answer: use a shared token bucket before each external request, honor Retry-After on HTTP 429, and requeue the same idempotent operation with a bounded delayed retry.

The system must preserve four invariants: one digest operation has one stable identity, an external attempt is paced before it leaves the process, a retry is eligible at a recorded time, and acknowledgement follows durable completion of the represented transition. Delivery can repeat. The business action must not.

A weekly support digest is a state machine

The Express route should accept or schedule the digest job; it should not sit on an HTTP request while an external provider recovers. A cron trigger can wake the scheduler, but the scheduler should enqueue one operation per active customer and return. A worker then takes responsibility for pacing, calling the external API, classifying the result, and acknowledging the queue message.

There are two distinct clocks. Token-bucket admission handles ordinary quota consumption: tokens refill at a fixed rate up to a bounded capacity, which allows a controlled burst and then makes the worker wait. Backoff handles a rejection after admission. On 429, use the provider's Retry-After value when available; otherwise calculate exponential delay with a cap and some jitter. The delay is part of the operation state, not an undocumented sleep.

Do not sleep while holding the queue message. Publish a delayed successor with the same operation ID, then acknowledge the current delivery only after that publication is durable. The ordering still has a duplicate window: a crash after publication and before acknowledgement can redeliver the original. That is why the receiver or a durable operation ledger must treat the operation ID as an idempotency key.

How does a token bucket rate limit protect a queue worker?

The failure boundary is deliberately narrow. A malformed request, an authentication failure, or another permanent 4xx becomes a terminal record; retrying it only repeats an invalid instruction. A 429 and a temporary 5xx become retryable states. A network error needs the same classification policy, with an attempt cap and an operator-visible terminal state when the cap is reached.

The queue is not the audit ledger. Store the operation ID, customer ID, digest period, attempt number, next eligible timestamp, response class, and final state in durable application storage. Keep response bodies bounded because a queue message may be limited to 256KB and sensitive support data should not be retained merely because an HTTP client returned it. A seven-day delayed-message limit also means a longer recovery horizon needs a fresh scheduling decision based on the stored timestamp.

That boundary matters.

For example, suppose a weekly digest for customer c-1842 is admitted at 09:00, receives a 429 with Retry-After: 120, and the worker crashes after publishing the delayed successor. The redelivered original must not create a second business operation; both deliveries point to the same ledger row, where the operation ID, attempt number, next eligible time, and provider result can be compared atomically. If the successor is later admitted and the provider times out after accepting the request, the worker cannot infer failure from the missing response. It must ask the receiver's idempotency boundary or consult a durable provider result, then record a terminal or retryable state according to that evidence. This is why a queue is a recovery mechanism, not proof of delivery, and why an audit record needs state transitions rather than a single sent=true flag.

Option Good fit Choose another boundary when
Shared queue workers Independently paced digest calls with delayed retry The work is a replayable stream or a multi-step dependency graph
A cron-only job Small, bounded batches with no per-customer recovery state One customer can be throttled independently or must be retried later
A durable workflow engine Timers, branches, compensation, and human approval The operation is one outbound call with a simple state machine
A stream platform Replay, fan-out, and independent consumer groups A successful work item should leave the active queue

This comparison is intentionally about failure semantics, not product preference. A Node.js team may use a Redis-backed queue, a managed HTTP queue, or a self-hosted broker; the token policy and idempotency record still belong to the application contract. I'm not sure which quota a provider applies when several credentials share an account, so that question must be answered from its published contract before replicas are sized.

The critical path in Go

The adapter below leaves queue-specific mechanics behind two interfaces. It shows the important transition: admission, request, classification, delayed publication, and acknowledgement. The operation ID stays unchanged across attempts, and the outbound request carries it as an idempotency key when the receiving API supports that standard pattern.

package main import ( "context" "fmt" "io" "net/http" "strconv" "time" ) const maxDelay = 7 * 24 * time.Hour type DigestJob struct { OperationID string CustomerID string Endpoint string Attempt int } type RetryPublisher interface { PublishDelayed(context.Context, DigestJob, time.Duration) error } type Acker interface { Ack(context.Context, string) error } type Worker struct { client *http.Client tokens <-chan time.Time retry RetryPublisher acker Acker } func retryDelay(response *http.Response, attempt int) time.Duration { if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil && seconds > 0 { return minDuration(time.Duration(seconds)*time.Second, maxDelay) } if attempt > 10 { attempt = 10 } return minDuration(time.Second*time.Duration(1<<attempt), maxDelay) } func (w *Worker) Process(ctx context.Context, job DigestJob) error { select { case <-ctx.Done(): return ctx.Err() case <-w.tokens: } request, err := http.NewRequestWithContext(ctx, http.MethodPost, job.Endpoint, nil) if err != nil { return fmt.Errorf("build digest request: %w", err) } request.Header.Set("Idempotency-Key", job.OperationID) response, err := w.client.Do(request) if err != nil { return fmt.Errorf("send digest request: %w", err) } defer response.Body.Close() body, err := io.ReadAll(io.LimitReader(response.Body, 64<<10)) if err != nil { return fmt.Errorf("read digest response: %w", err) } if response.StatusCode == http.StatusTooManyRequests || response.StatusCode >= 500 { job.Attempt++ if err := w.retry.PublishDelayed(ctx, job, retryDelay(response, job.Attempt)); err != nil { return fmt.Errorf("publish digest retry: %w", err) } return w.acker.Ack(ctx, job.OperationID) } if response.StatusCode < 200 || response.StatusCode >= 300 { return fmt.Errorf("terminal digest status %d: %s", response.StatusCode, body) } return w.acker.Ack(ctx, job.OperationID) } func minDuration(a, b time.Duration) time.Duration { if a < b { return a } return b } 

The token channel in this example is an injection point, not a claim that an in-memory ticker solves a distributed quota. With three worker replicas, three private buckets can multiply the outbound rate. Use coordinated state, or assign and document a per-replica share. The exact implementation depends on the queue and deployment, but the decision must be observable: record granted admission, request start, response class, retry eligibility, and acknowledgement outcome. A trace without the operation ID is decoration.

Cron's valid boundary for digest recovery

The rejected design is a cron-only loop that fetches all active customers, sends digests inline, sleeps after a 429, and marks the run successful when the loop exits. The catch is that it is only suitable for a small, bounded batch whose only requirement is best-effort completion. It is not suitable when one slow provider response can delay every customer, when a customer needs an independent retry deadline, or when an operator must prove which digest was sent; choose a queue worker with durable operation state in those cases.

The same caution applies to treating a successful queue acknowledgement as proof of an external side effect. It proves only that the worker completed the transition represented by that message. The ledger must distinguish queued, admitted, sent, acknowledged-by-provider, retry-scheduled, and terminal states. For payment-adjacent support systems, that distinction is the difference between an explainable reconciliation and a guess.

Cron itself has limits: a run can last at most 900 seconds, scheduled timing can have second-level jitter, and missed triggers are not replayed after a schedule is paused. Treat it as a wake-up signal. The queue and operation record are what make recovery explicit.

References