Originally Published in https://builder.aws.com/content/3MDnGtpMukH2JgVoEYV8dpiaBkv/designing-safe-kubernetes-rollbacks-when-stateful-changes-make-rollback-dangerous

Kubernetes makes rollback look deceptively simple:
kubectl rollout undo deployment/orders-api -n production
That command can restore a previous Deployment revision. It cannot reverse a database migration, recall a message already published, undo a payment, or restore a contract that another service has started using.

The distinction matters because production releases rarely change only container images. They change state, contracts, and behavior across independently deployed components.

A rollback strategy that considers only the Deployment is not a recovery strategy. It is an image-reversion procedure.

  1. The rollback illusion Consider an order-processing application deployed as version 1: orders-api v1 | +--> orders database schema v1 | +--> order-created event v1 | +--> fulfillment-worker v1 A new release introduces version 2, adds a non-null database field, changes the event payload, and deploys a new worker. Everything looks healthy for several minutes. Then the API starts failing under a production-only workload.

The on-call engineer executes kubectl rollout undo. Kubernetes replaces the new application Pods with the previous template. But the database still contains the new schema and data. Some messages already use the new format. The new worker may already have produced external side effects.
ROLLBACK REQUEST
|
+----------+----------+
| | |
v v v
API v1 DB schema v2 Events v2
| | |
+----------+----------+
|
v
Compatibility failure
The Deployment rollback succeeded. The service recovery failed.

The architectural question is therefore not, "Can we roll back the Pods?" It is, "Will the previous version remain compatible with every state transition that occurred after deployment?"

  1. Classify changes by reversibility Before discussing tools, classify the change. Change Typical recovery property Primary risk Stateless application code Often reversible Changed external contracts Additive database column Usually backward-compatible if optional Old writers and readers Column rename or removal Potentially breaking Old code expects removed schema Data rewrite Often conditionally reversible Information loss or ambiguity Event schema change Depends on consumer compatibility Messages outlive deployments External API side effect Frequently irreversible Payment, email, or third-party mutation Infrastructure identity change Depends on resource semantics Replacement and data retention

Do not label an entire release "rollback safe" because the container image is reversible. The release inherits the risk of its least reversible dependency.

A practical classification is:

  • Revertible: the old version can run safely against current state.
  • Conditionally revertible: recovery requires a documented compatibility or repair action.
  • Roll-forward only: old behavior cannot safely interpret or operate on the new state.
  • Manual recovery: an operator must reconcile external effects or data. These are policy inputs, not marketing labels. Each classification needs evidence.
  • Expand and contract instead of mutate and hope The most reliable database migration pattern separates structural expansion from destructive contraction.

Suppose version 1 stores a customer name in full_name. Version 2 wants given_name and family_name.

A dangerous migration immediately removes full_name and deploys the new code. If the new release fails, version 1 has lost the field it requires.

A safer sequence is:
Phase A: EXPAND
Add given_name and family_name
Preserve full_name
|
v
Phase B: COMPATIBLE APPLICATION
Read old and new representations
Write both where appropriate
|
v
Phase C: BACKFILL + VERIFY
Migrate existing records
Reconcile discrepancies
|
v
Phase D: CUT OVER
New reads use new fields
Old version still supported
|
v
Phase E: CONTRACT
Remove full_name only after
rollback window has closed
The key invariant is that the old application must remain operational throughout the period when rollback is an approved response.

Dual writes are not free. They create consistency risk when one write succeeds and the other fails. The application needs an explicit failure policy, reconciliation process, and observability for divergence. For some migrations, database-generated compatibility views or triggers may be appropriate, but they introduce their own coupling and operational hazards.

The point is not that expand-and-contract removes all risk. It moves destructive operations outside the immediate deployment failure window.

  1. Schema compatibility is not data compatibility Even when the old schema still exists, the old code may not understand the new data.

Imagine version 2 introduces a new order status:
PENDING
AUTHORIZED
PARTIALLY_FULFILLED
COMPLETE
Version 1 recognizes only PENDING, AUTHORIZED, and COMPLETE. Rolling back the image may restore a valid executable that cannot process orders created by version 2.

This is a semantic compatibility failure, not a database DDL failure.

For each state transition, ask:

  1. Can the old version read records written by the new version?
  2. Can it safely update them without destroying new information?
  3. Can it preserve unknown enum values and optional fields?
  4. Will its background jobs reinterpret or replay newer state incorrectly? A compatibility test should populate a test environment with data produced by the candidate release, then execute the previous release against that data. Testing only old data against new code misses the rollback direction entirely.
  5. Messaging extends the rollback boundary Asynchronous systems are especially vulnerable because messages survive application deployments. producer v2 | v Kafka / SQS / RabbitMQ | +--> queued event v2 | v consumer v1 after rollback A producer rollback does not remove already published messages. A consumer rollback may encounter payloads that did not exist when it was built.

Design event evolution for mixed-version operation. Prefer additive optional fields, explicit schema versions where useful, and consumers that tolerate unknown fields. Do not assume that renaming a JSON field is harmless because the producer and consumer live in the same repository.

Also account for delayed delivery, retries, dead-letter queues, and replay. The compatibility window may need to outlive the application rollout by hours or days, depending on retention and processing guarantees.

A useful release contract specifies:
compatibility:
producer_version: v2
consumers_supported:
- v1
- v2
replay_supported: true
destructive_schema_change: false
This is an illustrative contract for a release-policy system, not a native Kubernetes object.

  1. External side effects cannot be undone by Kubernetes Suppose version 2 sends a payment authorization, creates a shipment, or emails a customer. The Pod later fails and is rolled back.

Those external actions remain.

Attempting to replay the request through version 1 can duplicate the side effect unless the workflow uses stable idempotency keys and appropriate deduplication.
Request
|
v
Payment authorized
|
v
Application fails
|
v
Rollback
|
v
Request retried
|
v
Duplicate authorization?
For side-effecting operations, recovery may require a compensating action rather than a technical rollback. A refund is not the same as erasing a payment. Canceling a shipment may fail after dispatch. Sending a correction email does not retract the first email.

Model these workflows explicitly:

  • Stable idempotency identifiers across retries and deployments.
  • Durable records of completed side effects.
  • Compensation steps with known failure modes.
  • Manual escalation when compensation is unsafe or impossible. This is why "automatic rollback" should not be universally enabled for stateful workflows without a recovery classification.
  • Separate traffic rollback from state rollback A Kubernetes Deployment rollback changes the desired Pod template. It does not restore database contents or coordinate external systems. Kubernetes also cannot guarantee that the old revision is operationally compatible with current state.

Use distinct recovery actions:
Bad release detected
|
v
Stop additional exposure
|
v
Classify changed state
|
+--> Backward compatible
| |
| v
| Revert workload
|
+--> Incompatible state
| |
| v
| Roll forward or repair
|
+--> External side effects
|
v
Compensate / reconcile
For canary delivery, first halt promotion and route traffic away from the failing version where possible. That limits additional mutations. Only then decide whether the previous version is safe to resume.

A failed canary is not automatically reversible if the canary wrote shared database state or published events. Small traffic percentages reduce exposure, but they do not eliminate irreversible effects.

  1. Backups are not a general rollback button Database backups are essential for disaster recovery. They are not a substitute for compatibility engineering.

Restoring a database snapshot to its pre-deployment state can discard legitimate transactions that occurred after the snapshot. In a distributed architecture, other services may already have acted on those transactions.

For example:
12:00 database snapshot
12:05 new release
12:07 legitimate orders accepted
12:09 release failure
12:10 restore 12:00 snapshot
The restore might remove the orders while external payment or fulfillment systems still retain their effects.

Point-in-time recovery can reduce the recovery-point gap, but it does not automatically reconcile cross-system side effects. Recovery objectives need to specify both data loss tolerance and cross-system consistency requirements.

Use backups for data-loss and corruption scenarios. Use backward-compatible migrations for routine deployment recovery. Do not confuse the two.

  1. Put rollback compatibility into the release gate A production change gate should evaluate more than test results and container vulnerabilities.

A proposed release can carry machine-readable metadata:
release:
service: orders-api
version: v2
state_changes:
database: additive
event_schema: backward_compatible
external_side_effects: true
rollback:
classification: conditional
previous_version_tested: true
compensation_runbook: orders-compensation-v3
Again, this is an illustrative policy document, not a Kubernetes built-in.

The gate should challenge unsupported claims. previous_version_tested: true is useful only when the test actually ran the old binary against new state and captured evidence.

For high-risk releases, require explicit approval and a narrower rollout. For destructive migrations, prevent contraction until the supported rollback window has elapsed and old consumers are retired.

A policy engine can automate the checks, but it cannot infer semantic compatibility from YAML alone. Application owners remain accountable for the contract.

  1. Build a recovery test, not just a deployment test Most pipelines test whether version 2 can start and serve requests. Few test whether version 1 can return safely after version 2 has modified state.

A stronger test sequence is:
Deploy v1
|
v
Create representative v1 data
|
v
Apply additive migration
|
v
Deploy v2
|
v
Execute state-changing workflows
|
v
Deploy v1 again
|
v
Validate reads, writes, events,
background jobs and invariants
This should include realistic edge cases: unknown enum values, partially processed messages, retries, old consumer versions, and in-flight requests.

Not every external system can be faithfully simulated. That limitation should be documented, not hidden behind a green pipeline.

  1. Observe invariants during rollout A Deployment may report all replicas Ready while data integrity is deteriorating.

Monitor application-level invariants alongside infrastructure metrics:

  • Orders accepted versus orders persisted.
  • Events published versus events consumed.
  • Duplicate side-effect attempts.
  • Dead-letter and retry growth.
  • Records missing newly required fields.
  • Divergence between old and new representations.
  • Failed compensations. An operational dashboard should connect rollout version, schema version, event version, and business outcomes. Release version | +--> Error rate / latency | +--> Database invariants | +--> Event compatibility | +--> Side-effect reconciliation | +--> User-facing SLOs A rollback decision made only from CPU, restart counts, and HTTP 5xx misses some of the most consequential stateful failures.
  • Define stop conditions before deployment The runbook should state when to halt promotion, when workload rollback is allowed, and when roll-forward or manual recovery is mandatory.

Examples of stop conditions include rising incompatible-message counts, schema reconciliation failures, unexpected duplicate operations, or breaches of critical transaction SLOs.

If the release is classified as roll-forward only, the incident commander should not discover that fact after attempting kubectl rollout undo.

The decision tree and owner assignments belong in the change record before production execution.

  1. The architecture of a recoverable release A safer delivery design separates application rollout, state migration, and compatibility verification: Release Pipeline | v Compatibility Tests | v Expand State Schema | v Canary Release | v Observe Business Invariants | +------+------+ | | Healthy Failure | | v v Promote Stop Exposure | | v v Close Rollback Recovery Classifier Window | | +----+-----+ v | | Contract State Revert Repair / Forward The design deliberately postpones destructive schema cleanup. It also recognizes that not every failure can be fixed by reverting the application.

The cost is additional engineering: compatibility code, temporary schema duplication, longer migration timelines, richer testing, and more complex release orchestration.

That cost is real. But it is usually lower than discovering during an outage that the previous version cannot safely operate on production data.
Conclusion
Kubernetes is good at reconciling desired workload state. It is not a distributed time machine.

A Deployment rollback can restore a previous container configuration. It cannot automatically reverse database writes, message publication, external transactions, or new business states.

Safe recovery starts before deployment, with explicit compatibility contracts, expand-and-contract migrations, event-version tolerance, idempotent side effects, and tests that exercise the rollback direction.

The question that matters is not "Can we run the old image?"

It is "Can the old system still operate safely after the new system has changed the world?"

If the answer is unknown, the release is not proven rollback-safe.
References

  • Kubernetes: Updating a Deployment
  • Kubernetes: Rolling Back a Deployment
  • Kubernetes: Deployment Strategy
  • AWS Prescriptive Guidance: Database migration