📝 Originally published (in Japanese) at forge.workstyle.tech.

TL;DR The DB admin password and Redis password were stored in plaintext in the infrastructure configuration repository history. DB and Redis are services that applications connect to. I kept the history unchanged and rotated the credentials in order: Secret, DB and Redis, then application restarts.

TL;DR

Because the database and Redis passwords were left in the Git history, I rotated the credentials instead of rewriting the history. By updating the Secret, switching the values on the database and Redis sides, and then gradually restarting the app, I was able to make the change without interrupting connections. The object storage root credentials required a restart, so I scaled the Pods down to zero before bringing them back up.

What to Decide Before Rewriting Git History

Rewriting Git history changes everyone’s clones (local copies of the repository), pull requests (proposed changes), and commit IDs. History can also remain in GitHub caches and forks (separately copied repositories), so rewriting history alone may not remove all traces of an exposure.

This time, we left the history unchanged and replaced the credentials. We recorded them as “replaced” in the secret scanner’s allowlist.

Before starting, we prepared rollback steps and a decision threshold. In this case, the condition was: “Roll back if login fails for more than five minutes.” Deciding what state triggers a rollback in advance helps avoid delays when problems arise.

How to Prepare New Passwords for the DB and Redis

I generated the new passwords with Python’s secrets module (a standard library for creating secret values) and wrote them to files. I made sure not to include a trailing newline, so I wouldn’t have to type the values directly into the CLI (the screen where you enter commands) or put plaintext in SQL.

For the DB change, I used a SCRAM-SHA-256 verifier (a value used for authentication instead of the password itself). I prepared the verifier and set it with ALTER ROLE … PASSWORD '<verifier>'. This avoids including the plaintext password in the SQL statement.

For Redis, I read the value from a file and ran redis-cli -x config set requirepass. The -x option passes the value through standard input (the input channel used to provide data to a command).

In What Order Should You Switch DB and Redis?

If you’re only changing passwords, neither the DB nor Redis needs to be restarted. Existing sessions can keep using their current connections. So we updated both credentials in a single maintenance window.

First, we updated the Secret (a Kubernetes resource for managing credentials and other sensitive data). At that point, the Pods (the units that run the application) hadn’t been restarted, so the apps were still using the old values. We then changed the credentials on the DB and Redis before restarting the apps. Reversing this order would cause apps that had loaded the new values to try connecting to the DB or Redis while they still had the old credentials.

We restarted the apps in batches of three or four. Apps that use GPUs were restarted one at a time. We also changed environment variables (env) that had been configured in plain text to use secretKeyRef (a setting that passes a Secret’s value to an environment variable).

The work took place on October 13, from 00:24 to 00:54 Japan time. The 15 apps that don’t use GPUs had all been restarted by 00:27. The six GPU-using apps were restarted between 00:30 and 00:53. There were zero authentication failures and zero restarts. The number of DB connections dropped from 55 to 20–24.

We monitored the systems from 00:54 to 01:04 and confirmed there were no authentication or connection errors. Before the work, we had estimated it would take 45–60 minutes.

Why We Scale Object Storage to Zero Pods

The object storage root was read only from the environment, so changing its value required a restart. There was no way to switch it while the app kept running, as we could with the database or Redis.

We didn’t use rollout restart. With a RollingUpdate (a deployment strategy that replaces old Pods with new ones), both the old and new Pods would open the same volume (storage used by a Pod) at the same time. Instead, we first ran scale --replicas=0 and confirmed that there were zero Pods, then scaled back up to one.

We recreated the Pod at 01:02. There were no authentication failures afterward. The planned downtime was 30–60 seconds, but the logs don’t confirm the actual duration. All signed URLs issued before the restart became unusable. We let users know in advance about the impact, including downtime while each app restarted.

What to Check After the Work

After the work, check for authentication failures and connection errors. This time, neither the database nor Redis restarted or reported any authentication failures. There were no authentication failures after the storage was recreated, either.

The automatic approval mechanism stopped the script that rewrote the keys. So I documented the procedure, and another team carried out the work after a person reviewed it. I kept preparing the procedure separate from the decision to make the change.

When a secret remains in history, consider not only whether to rewrite that history, but also how to revoke the value currently in use. This time, we switched the database and Redis credentials while keeping their connections open, and handled storage separately with a procedure that required downtime. We decided on rollback conditions in advance, then checked the logs and connection counts afterward to confirm the state.

Key takeaways

  • Check whether the values in history are still used in production, then decide whether to rewrite history or rotate credentials.
  • Set the rollback procedure and condition first; in this case, roll back if login fails for 5 minutes or more.
  • Prepare new values and update Secret, DB and Redis, then restart the application.
  • Restart applications gradually and check authentication errors, connection errors, and DB connection count.
  • For storage that requires a restart, confirm Pod count is 0 before bringing it back, and check downtime and user impact.