The tempting version of a storage change is a cutover.
Turn off the old path.
Turn on the new path.
Hope both still mean the same thing.
That is how you get a clean demo and a dirty night.
AL'S-OPS LLC is a Utah company building privacy-first infrastructure software. This week we did not cut over. We added a dual-write on the servers table and left the production flag off.
What shipped
Two write paths — upsertServer() and markServerOffline() — also write to an in-memory mirror. The mirror sits behind MIRROR_DB_ENABLED, which defaults to off.
Around that:
- A parity checker comparing Postgres to the mirror
- A queue-flush job
- A parity-check job
- A burn-in loop instead of assuming the first write was correct
The new store is not live in production. That is intentional.
Why the flag mattered
A real bug showed up in the write-path SQL.
Partial-column updates would have silently corrupted mirror data.
Not a crash. Not a useful stack trace. Wrong state later, when you no longer remember which write did it.
That is the class of failure dual-write is for. You want the new path to fail while the old path is still the source of truth.
The rest of the same stretch
- A stability and security sweep (9 fixes)
- Retirement of the previous store
- Early work on the next mirror layers — still dark
Order mattered more than any one item: retire what you can, put the new path behind a flag, write to both, compare, then argue about turning it on.
The rule
Storage migrations die in the gap between "it compiled" and "every write shape was tested."
Partial updates look harmless. They are how two systems drift until neither copy is trusted.
Find that while the mirror is dark. Keep the flag off until parity is boring.
Originally published on the Al's-Ops blog.
