To plan or not to plan: the Terraform file that leaked every password we had

Committing an API key to a repo is the nightmare every developer has had at least once. Most of us have a scanner, a hook and a .gitignore that are supposed to make it impossible. I never thought I would be writing this post. Then a Terraform plan file showed up in our git history.

A Terraform plan file is a zip archive. It does not care what you name it. Our .gitignore cared. Our secret scanner cared. Both were looking for *.tfplan, tfplan and plan, and the file that was committed was called out.

I ran it. I know the convention. The command I meant to type was terraform plan -out=tfplan. The command I typed was terraform plan -out=out. Terraform does not care. The file went in with the next git add .. The one small mercy: the repository is private. "Private" still means every engineer, every contractor and every CI runner with a clone had it. The branch sat on the remote for nine days before I suddenly found it.

This post is about what was actually inside, how I built the list of what to rotate, and why rotating it across four environments and six providers was the hard part.

Why the extension mattered

Everything that was supposed to catch this before the push was keyed on a filename.

.gitignore had tfplan and out/. Neither matches a file named out. GitHub's own Terraform gitignore template is worse. It has fourteen patterns for state, tfvars and override files, and none for plan files. *tfplan* appears there only in a comment.
The secret scanner looked at text. file out says Zip archive data. A password inside a zip is not a string in a file. Gitleaks can look inside archives since 8.27.0, but only with --max-archive-depth set, and it defaults to off.
Reviewers saw a build artifact. A binary called out in a pull request shows as "Binary file not shown" and reads as something a build left behind.

What is inside a plan file

You do not need Terraform to read one. You need unzip.

$ file out out: Zip archive data, made by v2.0 $ unzip -l out Length Name --------- ---- 412880 tfplan 1983322 tfstate 1983322 tfstate-prev 14023 tfconfig/m-/main.tf 6711 tfconfig/m-/variables.tf 8190 tfconfig/m-database/main.tf ... 114 tfconfig/modules.json 1229 .terraform.lock.hcl

Five things live in there.

Entry	What it is
`tfplan`	The planned changes, plus every input variable value, sensitive or not. Binary protobuf, but strings are strings.
`tfstate`	The full prior state after refresh. Plain JSON. Every resource in the workspace, not only the ones changing.
`tfstate-prev`	The state before refresh. Also full, also plain JSON.
`tfconfig/`	A copy of every `.tf` file, root and modules.
`.terraform.lock.hcl`	Provider versions.

The line that matters: a leaked plan file is a leaked state file. The state is in there whole, and strings tfstate | grep -i password works without a single Terraform binary. The diff was about twenty resources. The state had a few hundred. All of them were in the file.

A -target list does not limit this either. I checked with a plan that targeted one random_pet resource: the saved file still held the full state, every random_password result in the workspace, and every input variable. Targeting changes what Terraform plans to do. It does not change what it saves.

terraform show out prints (sensitive value) for the marked attributes. That is a display choice, not an encryption. terraform show -json out prints them in the clear, and the docs say so: "any sensitive values in Terraform state will be displayed in plain text."

Building the inventory from the file, not from memory

The first instinct is to list the secrets you know are in Terraform. The right move is to read the file, because Terraform holds secrets you did not put there on purpose.

terraform show -json out > plan.json

Input variables are the easy part:

jq '.variables | keys' plan.json

Then every resource in state with at least one attribute Terraform marks sensitive. The sensitive_values object mirrors the resource with true where a value is sensitive:

jq '[.. | objects | select(has("address") and has("sensitive_values")) | {address, paths: [.sensitive_values | paths(. == true) | join(".")]} | select(.paths | length > 0)] | unique' plan.json

The unique is there because the JSON lists every resource twice, once under prior_state and once under planned_values.

And the same for planned changes, where the field is after_sensitive:

jq '[.resource_changes[] | select(.change.after_sensitive | type == "object") | {address, paths: [.change.after_sensitive | paths(. == true) | join(".")]} | select(.paths | length > 0)]' plan.json

The sensitive marker is a floor, not a ceiling. Providers do not mark everything. Connection strings, bearer tokens in HTTP headers of a monitor resource, and private keys inside a Kubernetes secret data map all showed up unmarked in ours. I followed the jq list with a plain grep over the unzipped tfstate for password, secret, token, key, and BEGIN.

What came out sorted itself into five groups.

Group	Example in our state	How it got there
Passwords set by config	`aws_db_instance.password`, `azurerm_mssql_server.administrator_login_password`	I wrote it
Generated secrets	`random_password.app.result`	Terraform generated it, stored it
Secrets I only read	`data.aws_secretsmanager_secret_version.stripe.secret_string`, `data.azurerm_key_vault_secret.smtp.value`	A data source copies the value into state
Input variables	`TF_VAR_datadog_api_key`, `TF_VAR_cloudflare_api_token`, a `-var-file` for prod	Passed in by CI, saved by `-out`
Credentials providers handed back	`aws_iam_access_key.ci.secret`, `azurerm_storage_account..primary_access_key`, `tls_private_key.ca.private_key_pem`, `google_service_account_key..private_key`	Provider returned it, Terraform kept it

The third group was the surprise. Reading a secret with a data source to pass it into a container definition felt read-only. It is not. The value lands in state and in every plan file.

The fourth group is why "it was only the prod plan" was not a comfort. The Datadog key and the Cloudflare token were the same variable in every environment's pipeline.

Why rotation was not one afternoon

Each row in that table rotates differently, and the provider decides how.

Secret	How it rotates	Who breaks when it does
RDS master password	Change `password`, in-place modify	Every app holding the old one, until redeployed
Azure SQL admin password	Same, in-place	Same
IAM access key	Cannot change value. Create a second key, switch consumers, delete the first	Nothing, if you do the overlap
Storage account keys	Regenerate outside Terraform, refresh picks up the new value	Anything using the key directly
`random_password`	Bump `keepers`, every dependent resource updates	Dependents, same apply
Data-source secrets	Rotate at the source (Secrets Manager, Key Vault)	Every consumer, and Terraform state still has the old copy until next refresh
Provider tokens passed as variables	Rotate in the SaaS, update the CI secret, in every pipeline that has it	Every pipeline, until updated
CA private key	Replace the resource, reissue everything it signed	Every cert in the chain

Four things made this slow.

Terraform knows the resource, not the readers. The state says the database has a password. It says nothing about the three services, one cron job and one analyst notebook that have a copy of it. For every secret, the question "who else holds this value?" had no answer in any file. I built that list by hand.

Shared values across environments. The plan was for prod. The SaaS tokens were shared by dev, staging and prod pipelines. The CA key signed internal certs in all four environments. "Rotate what leaked" became "rotate in every environment that has the same value", which for some secrets was all of them.

Some providers allow overlap, some do not. IAM keys and most SaaS tokens let two credentials exist at once, so you add, switch, remove, with no outage. A database master password does not. For the databases I created a second login, moved the apps to it, and rotated the master afterwards, which is the overlap pattern rebuilt by hand.

The order I used, if it helps:

Credentials that can mint other credentials first: IAM keys, provider tokens, the CA key.
Data-plane passwords second, with the add-switch-remove pattern where the provider does not give one.
Internal generated secrets last.
After every step, terraform refresh and confirm the old value is gone from the current state. Then remember the backend bucket is versioned and still holds every old state.

That last point reframes the whole thing. After rotation, the new secrets live in the same state file, in the same bucket, readable by the same roles. The plan file leak was an accident. The state file is a standing copy of the same data, by design.

What I changed

I stopped writing the file at all.* This is the cheapest fix. terraform apply with no saved plan builds the plan in memory, prints it, waits for a yes, and applies it. Nothing is written to disk. apply does not even have an -out flag. The plan file in this story existed because -out was in a pasted command, and the same command without it would have shown the same diff and written nothing. Plain apply is fine for a human in any environment, prod included, because the plan you read is the plan that runs. A pipeline that approves now and applies later still needs the saved plan, or the apply may do something the reviewer never saw. Keep the file there, and only there.

What else could be done

Three options I have not tried yet. They attack the state file, not just the plan file, so they are the bigger fix.

Write-only arguments. Terraform 1.11 added them. On aws_db_instance, password_wo is sent to the provider and never written to plan or state. password_wo_version is stored instead, and bumping it sends a new value. Look for the _wo suffix in other providers; adoption is uneven.
Ephemeral resources instead of data sources. Terraform 1.10 added ephemeral values, which are not saved to the plan file or the state. The AWS, Azure and Google providers each have an ephemeral version of their secret-reading data source. The data-source group in the table above should disappear with a one-word change from data to ephemeral.
Let the cloud hold the database password. manage_master_user_password on RDS means AWS generates the password, stores it in Secrets Manager, and Terraform never sees it. Azure and Google have identity-based database auth that removes the password entirely.