For a cron trigger that runs every minute, the least complex dependable Node.js design is to let a public webhook endpoint admit durable work to a queue, then return; a separate worker consumes that work at the downstream rate. The endpoint must be public enough for the scheduler to call, but it must not be a public work executor.
Short answer: for minute-based, rate-limited processing in Node.js, give every scheduled interval a deterministic batch key, enqueue stable operation IDs, and acknowledge queue messages only after a durable idempotent outcome exists.
A timer is not a transaction coordinator. It can arrive late, arrive twice, or overlap a retry; a queue can redeliver; and a downstream webhook can accept an effect just before the client loses its connection. Treating any one of those events as proof of completion creates the kind of reconciliation gap that payment and ledger systems cannot wave away later.
How should a Node.js public webhook endpoint queue rate-limited work every minute?
Make the HTTP handler an admission boundary. It accepts only a configured task identity over TLS, authenticates the scheduler, derives the UTC minute being admitted, and attempts to insert one durable batch record with a uniqueness constraint such as (task_name, scheduled_minute). A duplicate request finds the existing row and succeeds without creating a second batch. The handler then records, or transactionally publishes, operation references belonging to that batch and returns an accepted response. It does not wait for each operation to call a downstream service.
That separation sounds fussy until a scheduler retries while the first request is still finishing. The two requests must converge on the same batch record; the body should not contain a caller-selected rate, arbitrary destination URL, or an unbounded list of work. Those are configuration choices owned by the service, where they can be reviewed and audited.
Keep the identities separate:
- A batch key identifies one logical scheduled interval, for example
ledger-close:20260807T1432Z. - An operation key identifies one intended business effect, such as
ledger-close:account-184:2026-08-07. - A delivery attempt identifies a queue handoff and is never a business idempotency key.
The worker controls pacing because it observes real completion and throttling. One process can use a local token bucket. Multiple replicas need a shared limiter or an intentionally partitioned budget; giving each replica the full limit quietly multiplies traffic during a scale-out. A bounded queue depth and oldest-message age turn overload into an explicit policy decision rather than an invisible backlog.
The durable contract matters more than the minute schedule
For a financial workflow, exactly-once transport is an aspiration, not an audit record. The useful guarantee is that repeated delivery produces one business result, or repeatedly observes the already-recorded result. Persist the operation key before the external call when the workflow permits it, send that key to a downstream service that supports idempotency, and store its returned correlation identifier with the outcome. If the connection breaks after the remote service may have acted, retry with the same key or query that key; generating a fresh key converts uncertainty into a duplicate effect. This distinction matters during reconciliation: a worker can truthfully report that it has no response while the business system has already accepted the operation. The operator's next action is then evidence gathering, not blind replay. Search the durable operation row, the downstream correlation identifier, and the request trace under the same key. If those records cannot establish an outcome, route the item into an explicit exception state whose owner and review deadline are visible; a queue retry counter cannot decide whether a charge, posting, or notification happened.
No random keys.
Here is a compact Go model of the public admission boundary. The maps and channel are deliberately local so the important limitation remains visible: a deployed service replaces them with a database uniqueness constraint and durable queue or transactional outbox. The example is a protocol sketch, not a storage recommendation.
package main import ( "crypto/subtle" "fmt" "net/http" "sync" "time" ) type admission struct { mu sync.Mutex batches map[string]bool work chan string taskName string secret string } func (a *admission) handle(w http.ResponseWriter, r *http.Request) { if r.Method != http.MethodPost { http.Error(w, "method not allowed", http.StatusMethodNotAllowed) return } got := []byte(r.Header.Get("Authorization")) want := []byte("Bearer " + a.secret) if len(got) != len(want) || subtle.ConstantTimeCompare(got, want) != 1 { http.Error(w, "unauthorized", http.StatusUnauthorized) return } minute := time.Now().UTC().Truncate(time.Minute).Format("20060102T1504Z") batchID := a.taskName + ":" + minute a.mu.Lock() if a.batches[batchID] { a.mu.Unlock() w.WriteHeader(http.StatusNoContent) return } a.batches[batchID] = true a.mu.Unlock() for account := 1; account <= 3; account++ { a.work <- fmt.Sprintf("%s:account:%d", batchID, account) } w.WriteHeader(http.StatusAccepted) } In production, the batch insert and work publication must share a durable atomic boundary. Otherwise a successful batch insert followed by a process exit before publication creates an admitted batch with no work, while publishing before the insert can leave orphan work. An outbox table is a common answer: one transaction writes the batch, operations, and unpublished outbox rows; a relay publishes rows and marks them published; reconciliation detects rows that remain pending. The ledger-like rule is simple: every accepted interval must be explainable from durable evidence.
Where retries, dead letters, and rate limits change the design
Retries require a taxonomy. Bad authentication, malformed input, and an operation that violates a permanent business rule should terminate with an inspectable reason. Temporary rate pressure and transport uncertainty can use bounded backoff and jitter while retaining the same operation key. The queue acknowledgement follows the durable outcome, not the start of the attempt.
Dead-letter queues are quarantine, not a success state. AWS documents them as targets for messages that remain unprocessed after a configured number of receives. That pattern helps isolate poison work, but it also creates a reconciliation obligation: record the batch key, operation key, failure category, attempt metadata, and a safe diagnostic, then make replay a deliberate decision. Do not place credentials or regulated payload fields into a diagnostic merely because a dead-letter record is convenient. For ordered workflows, AWS also cautions that a dead-letter queue can disrupt FIFO ordering, so a sequence-dependent posting stream needs a stop-partition or compensating policy rather than automatic continuation.
The catch is that a per-minute trigger is not suitable when the work has a strict end-to-end deadline shorter than ordinary queue delay, or when every item must preserve a single global order. In those cases, use a design with an explicit partition owner and a documented latency budget; a timer plus independent consumers does not supply those semantics.
Kubernetes makes a related point in its CronJob documentation: schedule creation is approximate, and Jobs should be idempotent because duplicate or missed execution can occur. A concurrency policy can reduce overlap among Jobs created by one CronJob, yet it cannot establish business uniqueness across retries, deploys, or downstream requests. Database keys and idempotency records establish that truth.
A small operational rollout is safer than a heroic cutover
Start by running admission in observation mode: create the deterministic batch and audit record without dispatching external effects, then compare its expected operation count with the current path. Next, enable a small partition with the same idempotency keys and retain the old path only as a reconciliation reference, not as a second writer. Finally, move ownership once the difference set is empty and the rollback procedure is understood.
The dashboard should connect the layers: admitted batches by interval, duplicate admissions, oldest queued age, limiter wait time, claim-to-outcome latency, terminal failure count, quarantine depth, and unmatched operations. Alert on age and missing terminal outcomes. Raw retry volume alone is a poor proxy for correctness.
Compliance constraints belong in this rollout as well. Audit records need retention and access policies, and correlation identifiers should be sufficient for traceability without becoming a second store of sensitive payloads. Your storage and regulatory obligations may vary by jurisdiction; the evidence needed to resolve the question is the applicable retention schedule and data-classification policy, not a queue metric.
