This is article 6 of 10 in Beyond Kubernetes Defaults. Read article 5: A Running Pod Is Not an Admitted Workload first.

Single-process correctness can hide distributed-system bugs. Replay protection is a particularly clear example.

One replica, then two

A signed request arrives with a timestamp and a nonce. The application verifies the signature and records the nonce so it cannot be accepted twice. With one process and local state, this can look correct.

Now run two replicas behind a Kubernetes Service. The first request lands on replica A, which records nonce N locally. A replay lands on B, which has never seen N. B accepts the same signed request.

A Deployment or Service does not turn local security state into shared state. Scaling is not state synchronization.

Sticky routing is not a security boundary

Consistent hashing or sticky sessions can reduce the chance of a replay reaching another replica. But Pods restart, pools resize and endpoints change. If membership movement makes a consumed nonce fresh again, routing affinity reduces probability; it does not enforce the guarantee.

Define the invariant

After the first successful use of a nonce, every later use inside the replay window must be denied, regardless of which live replica receives it.

A shared store, safely replicated state, or correctly managed partition ownership may help, but each design needs explicit answers about:

  • Scope: include the appropriate tenant or source in the replay key.
  • Atomicity: checking and recording must be one atomic operation; two replicas must not both accept the same nonce.
  • Lifetime: retain the record for the full replay window, accounting for clock skew and membership changes.
  • Failure: decide what happens when the replay authority is unavailable. For security-sensitive signed ingress, a retryable refusal is often safer than silently accepting without the check.

Acknowledgement has a similar boundary

If a service returns HTTP 200 after putting an event only in process memory, a crash can lose the event after the sender believes it was accepted. A durable acknowledgement should correspond to the durability guarantee the service actually offers.

Test the seams

A useful HA test is not just killing a Pod and checking that another stays up:

  1. Send a signed request with nonce N and confirm acceptance.
  2. Replay N to a different replica and confirm rejection.
  3. Replace a replica and replay N again.
  4. Scale the pool up and down and repeat within the replay window.
  5. Make the replay authority unavailable and verify the documented failure behavior.

Also test the difference between recording durable acceptance and marking a downstream effect complete: suppressing a retry before an effect succeeds can lose an event.

This is not exactly-once delivery

Fencing, replay protection and idempotency protect different boundaries. At-least-once transport with correctly scoped idempotent effects can be useful, but exactly-once claims need precise boundaries and evidence.

A second replica is an availability feature and a correctness question: do your guarantees survive when state is distributed?

Originally published on the Zen Mesh blog.