Rotating a long-lived credential sounds trivial: create a new one, hand it out, delete the old one. In practice "hand it out" is never instant. A pipeline that started five minutes ago still has the old token in its environment. A service read its config at startup and caches it. A deployment that pushes the new secret to three systems finishes the first one long before the third.

If the old credential is revoked the moment the new one exists, everything in that gap breaks. What you want is an overlap: for a while, both the old and the new credential are valid, and consumers are steered to the new one while the old one quietly ages out.

This post describes how I implement that overlap purely in OpenTofu, with no scheduler, no Lambda and no external state, and two bugs I hit along the way that are worth knowing about if you build something similar.

First things first: if your workload can use workload identity federation (OIDC), do that instead. A secret that doesn't exist needs no rotation. This pattern is for the long tail federation doesn't reach: tokens for self-hosted tools, SaaS API keys, client secrets for integrations, keys in config files.

The idea: two slots that take turns

Every rotated credential gets two slots, phase 1 and phase 2. Each slot holds its own, independent credential. They take turns being the one consumers get, and each switch has an overlap window during which the previous credential still works.

Two parameters per credential:

  • E (expires_in): how long a credential is handed out before it is replaced,
  • W (rotation_window): the overlap, e.g. 20% of E.

A full cycle lasts 2E. With t0 as the start of the cycle:

 0 E-W E 2E-W 2E phase 1 ├──── in use ─────┼── old ───┤ ├──── in use ───► phase 2 ├───────────── in use ────────┼── old ───┤ 

As rules:

  • phase 1 exists in [t0, t0+E) and [t0+2E-W, t0+2E),
  • phase 2 exists in [t0+E-W, t0+2E),
  • consumers get phase 2 in [t0+E-W, t0+2E-W), and phase 1 otherwise.

So a new credential is always created W before the switch, consumers move to it at the switch, and the previous one keeps existing for another W before it is destroyed. At 2E the cycle starts over.

Time without a scheduler

OpenTofu has no built-in notion of "run this at 03:00". It only knows what time it is when you plan: plantimestamp(). That turns out to be enough, as long as you run tofu apply on a schedule that is much shorter than W (a daily pipeline for credentials rotated monthly, say).

Three small resources hold the timeline of one credential:

locals { expiry_minutes = floor(provider::time::duration_parse(var.expires_in).minutes) rotation_window_minutes = ( var.rotation_window != null ? floor(provider::time::duration_parse(var.rotation_window).minutes) : floor(local.expiry_minutes * 0.2) ) } # Changing expires_in restarts the cycle. resource "terraform_data" "expiry" { input = { expiry = local.expiry_minutes } lifecycle { create_before_destroy = true } } # Fires once per full cycle (2E). resource "time_rotating" "cycle_timer" { rotation_minutes = floor(2 * local.expiry_minutes) lifecycle { create_before_destroy = true replace_triggered_by = [terraform_data.expiry] } } # t0: the plan time at creation, frozen by ignore_changes, reset whenever the cycle timer fires. resource "terraform_data" "deterministic_time" { input = { time = plantimestamp() } lifecycle { create_before_destroy = true ignore_changes = [input] replace_triggered_by = [time_rotating.cycle_timer] } } 

The trick is deterministic_time. plantimestamp() changes on every plan, but ignore_changes freezes the value stored at creation, so its input.time is a stable t0. When time_rotating reaches the end of the cycle it is replaced, and replace_triggered_by replaces deterministic_time with it, which captures a fresh t0.

The phase rules are then plain arithmetic, which I keep in a resource-free module so it can be unit tested with tofu test:

locals { e = var.expiry_minutes w = var.rotation_window_minutes after_e_minus_w = timecmp(var.now, timeadd(var.start_time, "${local.e - local.w}m")) >= 0 before_e = timecmp(var.now, timeadd(var.start_time, "${local.e}m")) < 0 after_2e_minus_w = timecmp(var.now, timeadd(var.start_time, "${2 * local.e - local.w}m")) >= 0 before_2e = timecmp(var.now, timeadd(var.start_time, "${2 * local.e}m")) < 0 } output "phase1_active" { value = local.before_e || local.after_2e_minus_w } output "phase2_active" { value = local.after_e_minus_w && local.before_2e } output "use_phase2" { value = local.after_e_minus_w && !local.after_2e_minus_w } 

with start_time = terraform_data.deterministic_time.input.time and now = plantimestamp().

Finally, the credential module (whatever mints the token) is instantiated once per phase, only while that phase exists:

module "phase1" { source = "./extensions/gitlab/rotation" count = module.schedule.phase1_active ? 1 : 0 configurations = { (local.secret_id) = local.config } } module "phase2" { source = "./extensions/gitlab/rotation" count = module.schedule.phase2_active ? 1 : 0 configurations = { (local.secret_id) = local.config } } locals { active_phase = module.schedule.use_phase2 ? module.phase2 : module.phase1 credential = local.active_phase[0].secret[local.secret_id] } 

Everything downstream (a GitHub Actions secret, a Key Vault secret, a variable group) consumes local.credential. When the active phase flips, OpenTofu updates those consumers in the same apply. When a phase stops existing, its count drops to 0 and the old credential is destroyed, which for most services means revoked.

Bug 1: the dependency cycle when the timer fires

The first version worked for exactly one cycle. When time_rotating expired, every plan failed with a dependency cycle through the rotation module, the merged credential map and every deployment.

The cause is the default replacement order, destroy then create:

  • destroying the old deterministic_time must wait until everything depending on it has been updated,
  • those dependents (the deployments) need the new credentials,
  • the new credentials need the new phases, which need the new t0,
  • and the new deterministic_time can't be created before the old one is destroyed.

That's a cycle. The fix is one line per resource: create_before_destroy = true on terraform_data.expiry, time_rotating.cycle_timer and terraform_data.deterministic_time (it's already in the snippet above). The new timer and t0 are created first, dependents update, and the old instances are destroyed last. For these three resources that's harmless, because they have no side effects outside the state. I wrote this down as an ADR, because "why is create_before_destroy on a timestamp?" is exactly the kind of line someone will helpfully delete a year later.

Bug 2: the credential must outlive its turn, not just its expiry

The second bug is subtler, and I only caught it while writing this post, by simulating the timeline instead of eyeballing the diagram.

Many services enforce an expiry themselves: an Entra ID client secret has an end_date, a GitLab token has an expires_at. The natural thing is to set that to creation + expires_in. But look at how long each slot actually exists:

  • phase 2 is created at t0+E-W and destroyed at t0+2E: it exists for E + W,
  • the phase 1 created at 2E-W lives on into the next cycle, until t0'+E, where t0' is the new cycle start: again E + W, plus however late the run that resets the cycle is.

With a service-side lifetime of only E, the first switch is fine (phase 1 is created at t0 and expires at t0+E, W after consumers have left it). From the second switch on, though, the old credential expires at the very moment consumers switch away from it. The overlap is gone, and because the switch happens at the first scheduled run after the boundary, consumers can even hold an already-expired credential until that run arrives.

The fix is to give credentials a service-side lifetime of expires_in + rotation_window, rounded up if the service only has day precision (GitLab does). Then every credential is valid for its whole existence, and the overlap holds on every switch.

The general lesson: in a scheme like this, "how long is a credential handed out" and "how long must it be valid" are different numbers. Write a 30-line simulation of the phase rules before trusting your diagram.

Keeping the plan readable

One more thing that matters in practice: you want tofu plan to show exactly which consumers a rotation will touch. That only works if the set of credentials (service and id) is known at plan time, even when the values aren't. Two habits help:

  • Build the credential map with for expressions over the configuration keys, not with merge() over whole module outputs. merge() turns the entire result unknown as soon as one argument is unknown.
  • Give unknown leaves a concrete type. In OpenTofu 1.13+ convert(value, object({ username = string, secret = string })) does that.

With that, a plan during a rotation shows "phase 2 credential will be created" and "these four secrets in these three systems will be updated", which is what a reviewer actually wants to see.

Operational notes

  • Run often. Rotation only happens when OpenTofu runs, and each switch happens at the first run after its boundary, so that delay comes out of the overlap. Schedule apply at intervals well under half of W; a missed run is then still caught up within the overlap.
  • Secrets live in state. Rotated values end up in the state and plan. Use OpenTofu's client-side state and plan encryption, and a backend only the rotation pipeline can read.
  • Two slots fit some APIs perfectly. AWS allows exactly two access keys per IAM user, which is exactly what the two phases need (as long as the user has no other keys).
  • Lifetimes are days to months. This is for long-lived credentials that consumers need to hold. For minutes-long, per-request credentials, use dynamic secrets (Vault/OpenBao).

Where this lives

All of this is part of Gaean Key, an open-source (MIT) OpenTofu project that rotates credentials from YAML files in Git and pushes them to their consumers. The phase arithmetic, the ADR on the dependency cycle and the extensions are in the repo if you want to read real code instead of snippets. There are also two implementations of the engine, a single root module and a Terramate stack per credential, with a write-up of their trade-offs.

If you've built rotation for credentials that can't be federated, I'd love to hear how you handled the overlap.