My Incident Bot Offered to Break Production. One Click Away.

On Sunday morning my incident bot posted to Slack: 20 applications on my Kubernetes hub were on the
wrong git branch. It had a fix ready, one green Approve button away. The 20 applications were fine. The fix would have broken all of them. And over the next two hours,
getting the Deny button to work took three more fixes, each one a different way that a contract
between two systems had quietly gone stale.

This is that morning.

The setup

I run a small platform: a hub cluster with ArgoCD, Vault, observability and the rest, managed from
one git repo by me and a group of AI coding agents. ArgoCD keeps each app on the cluster in sync
with a branch of that repo. Hermes is the platform's incident bot. Every few
minutes it runs a set of health checks (CI, sync state, certificate expiry, which branch each app
deploys from), groups what's failing into one incident, and when it knows a pre-approved repair,
posts a Slack card with Approve and Deny buttons. A click goes to a small relay service (a Cloudflare Worker), which
checks that the request really came from Slack, that I'm on the approver list, and that I've
re-authenticated in the last 24 hours, then queues the repair.

Until last week, the rule was simple: the hub runs whatever is on the current release branch.
Push to it, and ArgoCD deploys it.

Last week I changed that rule. Release branches were reaching the running hub before anyone had
checked them, so the hub now deploys from a single production branch, deploy/hub. It
moves only when I promote a build: a CI-green commit, a typed confirmation, a tag. Every app on the
hub was repointed at that branch. The plan was reviewed, tested, and landed.

The plan never mentioned Hermes.

09:51: "20 apps not on the release branch"

(In the screenshot, values_branch is the branch check's internal name and r9-d9add2e2 is the
repair's ID.)

Hermes' branch check works out which branch every app should be on. It still worked that out the
old way: "whatever branch the repo is checked out on", which on the hub was the release branch.
ArgoCD reported the production branch, correctly. Twenty apps "on the wrong branch".

That alone is noise. The repair is the problem. The suggested fix re-deploys every app pointed at
the branch the check expected. Approving it would have moved every app on the hub and on the
second cluster back to the release branch: code nobody had promoted, live, with the promotion step
silently bypassed. The card even said so, in its own blast-radius line. And the repair is marked
not reversible.

The old rule still lived in three places nobody listed when it changed:

• the branch check's idea of "expected",
• the repair, which was built to move apps to that expected branch,
• the status report I run by hand, which guessed the expected branch the same way.

None of these were wrong when they were written. They encoded a contract ("apps should be on the
release branch"), and the contract changed under them.

I pressed Deny.

Failure 2: Deny did nothing

Slack's tooltip: "This app is not configured to handle interactive responses. Please configure
interactivity URL for this app under the app config page."

The relay that verifies clicks had existed for months. Its tests were green. No click had ever
reached it: the Slack app's Interactivity setting, the switch that sends button
clicks to an app, had never been turned on. The buttons had
been decoration since the day they shipped.

Worse, the card was posted by a different Slack app (an older notifications app) from the one
the relay belongs to. Slack sends a button click to the app that posted the message. So even with
Interactivity on, those cards could never work. The fix was to move Hermes' posting to the relay's app.

Failure 3: /hermes-auth "did not respond"

At 11:14 Hermes re-posted the card, now from the right app. I clicked Deny. The relay answered:
Re-authenticate first: run /hermes-auth. Progress: the click reached the relay, it really came from
Slack, and I was on the approver list.

/hermes-auth failed. In the Slack app's settings, the command was sent to the relay's
button-click address, not its slash-command address. The button-click handler didn't recognise
the request, rejected it, and Slack reported the app as unresponsive. The
setup guide just said "register /hermes-auth". It now names the URL.

Failure 4: the card was gone

Re-authenticated, I went back to the 11:14 card to press Deny. It wasn't there.

The relay's "re-authenticate first" reply had replaced it. When someone clicks a Slack button,
Slack gives the app a reply address. What the app posts there either replaces the original message
or adds a new one, depending on one flag, replace_original. The relay asked for a private reply
and left that flag out. In my workspace, that overwrote the card. The reply even kept the card's
11:14 timestamp. The buttons went with it.

The relay's tests checked that the reply said "hermes-auth". None checked the one flag that
decided whether the card survived.

What actually went wrong

Four failures, one pattern. Each was a contract between two components where one side changed, or
was never wired, and nothing on the other side noticed:

Every component passed its own tests. But tests share their author's assumptions, so they can't
catch a wrong one. What caught all four was a person pressing a button in production.

The rule I added

The same day I added one rule to the instructions my AI agents follow when they write a change plan:

Contract readers. A spec that changes a shared contract (a git ref, a Keychain item, a Slack
app or webhook, a port, an env var, a file path another component reads) lists every reader under
## Contract readers, found by grep, with file:line, and says for each whether it changes or why
it doesn't. A reader found after dispatch is a spec defect.

It would have caught the first failure: a search for code that works out "the expected branch"
finds the branch check in seconds. It's cheap, it's mechanical, and an AI agent writing a change plan will do it every
time if the rule says so. The precedent is written next to the rule, so the next agent knows why it
exists.

The other three are a different lesson: a feature that waits on a human action has no evidence
until a human takes that action. The approval buttons had unit tests, an approver list, signature
verification, re-authentication, expiry times. Nobody had ever clicked one. Next on my backlog:
live smoke tests as Hermes health checks (a test run of each slash command and a test button
click), so the next broken connection pages me before an incident needs it.

Where it stands

• The branch check, its repair and the status report are being fixed to expect the production branch, and the only repair offered will point apps at it. Until then, every such card gets Denied.
• Button clicks are switched on, Hermes posts from the right app, and the slash-command address is fixed.
• The relay's "not yet" replies will set replace_original: false, with a test that checks the flag.

What's the stalest contract in your system — and who reads it that you haven't listed?