Short answer: choose a queue that exposes enough delivery state to build a feedback loop, then enforce the consumption limit in the workers. For an e-commerce reservation that expires after a fixed hold window, the queue is a transport and retry boundary; it is not the rate limiter.

Start with a modest worker concurrency, measure completion and failure rates, and change the allowance from observed demand. That keeps expiration work from competing blindly with checkout traffic. The important choice is latency versus cost: tighter expiry latency needs more continuously available capacity, while a cheaper design can tolerate a bounded sweep delay.

I start with three questions: how old may the oldest expiry message be, how much downstream capacity is reserved for it, and what happens when that capacity disappears? Those answers set the control interval and the ceiling more reliably than a queue feature checklist.

What fails when expiration is treated like a timer?

A reservation has two clocks. The business clock says that a hold is no longer valid after its fixed window. The delivery clock says when a worker gets around to marking it expired. Those clocks are related, but they are not the same thing.

A common first design scans a database, publishes every stale reservation, and lets workers consume as fast as possible. It looks efficient during a quiet test. During a promotion, the same scan can create a burst that consumes connection pools, competes with payment work, and produces a second burst when retries become eligible. The result is a reservation that is logically stale but operationally late. A 2-minute controller interval is a starting example, not a promise: the right interval depends on the hold window and the cost of each downstream call.

That is the boundary.

The opposite failure is subtler. A very small fixed consumer rate protects the rest of the system, but the backlog grows past the hold window. Increasing the rate forever is also wrong: the queue drains, but downstream dependencies see a short, sharp overload. A rate limit must respond to both backlog age and the capacity of the dependency being protected.

At-least-once delivery makes the data transition idempotent. A worker should attempt an update whose predicate still says the reservation is held and whose expiry timestamp has passed. If another delivery already changed it, the second attempt is a no-op. Acknowledging a message only after that durable result is recorded prevents a process restart from silently losing work.

Keep the retry path boring.

Which queue shape fits a dynamic limit?

Do not begin with a brand name. Begin with the state the scheduler needs to observe and control:

Requirement Queue capability to look for Why it matters
Bound worker pressure Explicit consumer concurrency or a pull operation The limiter needs a control point before work reaches the dependency
Detect late holds Approximate depth plus oldest-message age Count alone hides a small but old backlog
Recover transient failures Delayed retry or a separate retry schedule Immediate redelivery turns one failure into a tight loop
Isolate poison messages Dead-letter path with reason and attempt metadata A bad record must not block ordinary expiry work
Preserve correctness Acknowledgement after the database transition Delivery state must follow the business result

These capabilities can be implemented around many queue models. A pull-based queue makes the control loop explicit. A push-based queue can work when the worker has a local semaphore and the broker's prefetch or in-flight setting is bounded. A timer-oriented scheduler can also feed a queue, provided the queue remains the place where retries and backpressure are visible.

The selection rule is therefore operational: pick the simplest model that gives the team a bounded in-flight set, observable age, controlled retry behavior, and a quarantine path. A richer feature set is not automatically better. Every hidden retry or timeout is another state that has to appear in the runbook.

There is a real limitation here. A queue with weak age visibility is a poor fit when the business objective is tight expiry latency, even if it is inexpensive and easy to operate. A timer-heavy scheduler is a poor fit when the dependency needs fast backpressure and explicit retry inspection. In those cases, choose the model whose observability matches the objective; accepting more operational machinery is the trade-off for a tighter deadline.

How should the control loop change consumption?

Use a rate limiter as a governor, not as a promise about exact completion time. Recompute its allowance at a fixed cadence from signals such as oldest age, queue depth, downstream error rate, and available database capacity. Keep a floor so a quiet but stuck queue is still noticed, and keep a ceiling so a bad metric cannot unleash every worker.

For a small Go worker, the boundary can stay plain. The queue client is behind an interface; the limiter decides when the next receive is allowed; the handler owns idempotency and acknowledgement.

package expiry import ( "context" "time" ) type Message struct { ReservationID string Attempt int } type Queue interface { Receive(context.Context) (Message, error) Ack(context.Context, Message) error Retry(context.Context, Message, time.Duration) error } type Reservations interface { ExpireIfHeld(context.Context, string, time.Time) (bool, error) } type Limiter interface { Wait(context.Context) error } func Run(ctx context.Context, q Queue, store Reservations, limit Limiter, retryDelay time.Duration) error { for { if err := limit.Wait(ctx); err != nil { return err } msg, err := q.Receive(ctx) if err != nil { return err } _, err = store.ExpireIfHeld(ctx, msg.ReservationID, time.Now().UTC()) if err == nil { if err = q.Ack(ctx, msg); err == nil { continue } } if retryErr := q.Retry(ctx, msg, retryDelay); retryErr != nil { return retryErr } } }

The example deliberately does not use queue depth as a direct worker count. Depth is a backlog signal, not a capacity measurement. A controller can raise the allowance when the oldest age approaches the service objective and downstream errors remain below their threshold. It should lower the allowance quickly on timeouts or database saturation, with a slower recovery to avoid oscillation. That choice trades freshness against load: raising the limit reduces expiry lag but increases the number of concurrent calls that can fail together.

The fixed hold window still needs a source of truth. Store the reservation expiry timestamp with the reservation, and make the conditional state transition authoritative. A message can arrive early, late, or twice. None of those delivery properties should change the business rule.

What should verification and rollback prove?

Test the failure modes, not only the happy path. The useful cases are a duplicate delivery, a worker restart after the database commit but before acknowledgement, a retry that arrives after the reservation was already released, and a dependency that returns errors while the backlog grows. Assert that each case leaves one valid business state and that the message eventually becomes acknowledged or quarantined.

For a deployment, canary the new limiter with a bounded worker set. Watch oldest-message age, expiry lag, in-flight messages, conditional-update outcomes, retry counts, dead-letter count, database saturation, and downstream error rate. A dashboard that shows only queue depth will miss the incident that matters: a short queue containing very old reservations.

Rollback should change the consumer configuration before it changes the data model. Lower concurrency, pause the expiry consumer, or restore the previous controller parameters while preserving the queue and its attempt metadata. Do not delete a backlog to make a graph look healthy. Once the dependency is stable, resume below the previous ceiling and let age, not panic, determine the next increase.

The practical decision is simple to state. Select the queue that makes in-flight work, age, retry state, and dead-letter state inspectable to the operators who will be paged. Put dynamic rate limiting at the worker boundary, make expiry idempotent, and choose the latency target before choosing how much always-on capacity to pay for.

References