A coordination protocol can keep its checked safety rules and still fall out of use. That is the main thing we learned from running LLM coding agents on seven machines for 103 days, and it is why we are releasing claw-consensus, the zero-LLM consensus engine we used, together with the aggregate telemetry that shows how its use collapsed.

The setup

Interactive coding agents (one model family) ran on seven separate machines, doing real knowledge and operations work for three people; scheduled headless runs worked alongside them (831 of the 2,069 work-claim records came from headless runs). The machines talked through a bus, a synced folder of markdown mailboxes, and made shared decisions through a consensus ledger, an append-only log where a change goes through propose, counter, accept, commit, verify. High-risk changes (Tier-2: money, irreversible, outward-facing or secret-handling actions) additionally needed a recorded human approval.

A snapshot of the bus taken on 3 October holds 22,242 messages from 20 June to 30 September. Between 29 June and 1 October the ledger recorded 1,131 events over 151 proposals.

Open artifacts

  • claw-consensus (MIT): the stdlib-only consensus engine, the mailbox bus and a failure-mode catalogue. The self-checking demo runs offline with no API key: 
git clone https://github.com/tonydzi/claw-consensus && cd claw-consensus python demo/demo.py
  • Data: the aggregation and statistics scripts and every aggregate count used here are in paper/telemetry/cain-2027/. Raw production logs are not published: they contain message bodies and third-party personal data. Only aggregate counts leave the fleet.
  • Paper: an earlier version is on Zenodo, doi:10.5281/zenodo.23189348; the numbers below come from the revised version, whose aggregates are in the repository above.

What held: two hard rules written as code

Two safety properties were enforced by a few hundred lines of deterministic Python, with no LLM in the loop, and checked over every recorded ledger event:

  1. No high-risk change without a human. None of the 27 committed Tier-2 proposals lacked a prior human approval event (rule-of-three approximate one-sided 95% upper bound: 11.1%).
  2. One proposal, one committer. No proposal received commit events from two different machines.

This is an observation from one fleet, not a safety guarantee, and it says nothing about actions that never went through the ledger. The rules that depended on discipline did worse: independent verification held for 66% of commits (76 of 115), text-rail acknowledgement for 46% (726 of 1,567), and most human approval requests were never decided.

What decayed: use of the protocol

Use of the consensus protocol fell from 121 proposals in July to 6 in September (Mann-Kendall on weekly counts, tau = -0.57, p = 0.005; a tie-corrected test gives p = 0.004). Over the same months, lighter mechanisms grew:

  • work claims ("I am editing this zone") reached 1,440 in September;
  • deployment packages ("this fix must reach every node and be proven there") reached 155 in September.

The human approval gate saturated too. Of 139 approval requests on one node, 18 (12.9%, CI [8.4, 19.5]) got an explicit decision; 117 went stale, three expired and one was closed by a later sweep.

Why? Three of the paper's four hypotheses

One observational case cannot prove causes, so the paper states four hypotheses with the evidence for each. Here are three; the fourth is the gate saturation described above.

H1, latency. For the 112 proposals that committed, the median time to commit was 162 minutes (95% bootstrap CI for the median [34, 417]) and the 90th percentile was 9.5 days. An exclusive work claim takes one command and is effective immediately. Most tiered proposals were reversible (106 of 148 were Tier-0 or Tier-1), and for reversible work the claim is a cheaper way to avoid the collision that actually happens: two sessions editing the same zone. The timing fits: claims were introduced on 6 July and proposals fell in August.

H2, alarm fatigue. Eight proposals produced 86% of all escalation events, each re-escalating on a timer while stuck: a channel whose alarms were mostly non-actionable. The operators' dated instructions (14 July, 6 August) record a deliberate turn away from approvals. Duplicate asks, obsolete work and notification failures are alternative explanations we cannot rule out.

H3, the wrong unit. The protocol coordinates decisions. What the fleet mostly needed was to coordinate changes: a fix made on one machine must reach every other machine and be proven there. The deployment register does that, and it grew while proposals fell. We cannot link individual pieces of work across channels, so this shows co-movement, not proven migration of the same work.

A caveat any reader should keep in mind: this is one fleet, built and operated by the author, and all approvals in the data were made by the author. Treat the operator evidence accordingly.

Six lessons for practitioners

These are lessons from one deployment, to be tested elsewhere.

  1. Measure usage, not only correctness. Our ledger would have reported "all green" in September with six proposals a month. Count proposals per active session per week and alarm when they fall well below the first month's level. We have not back-tested a normalized threshold on our own data, so treat any cut-off as a starting point.
  2. Put the hard rules in deterministic code. Agents can author proposals; they should not be able to approve them.
  3. Consider coordinating changes, not only decisions. In our workload a register with a per-node done marker handled "this fix must reach every node" with fewer steps than negotiation. Claims avoid concurrent edits; they do not settle real disagreements or authorize risky actions.
  4. Treat the human gate as a scarce channel. Budget asks per day, batch them, alarm on proposals that re-escalate on a timer, and give reversible asks a default on silence. For actions that require human approval, silence must keep the action blocked.
  5. Write timestamps in UTC and keep an append-only copy of everything you will want to count. 235 of 726 task/acknowledgement pairs (32%) had negative latency because machines in five time zones wrote local time; normalizing cut that to 129 (18%), so local time explains much of it but not all. Mailbox rotation left us able to recover only a part of September's messages. The append-only UTC ledgers were the sources we could still read in full three weeks later.
  6. Audit the cheap classifier before trusting the counts. A regular expression over the first line of each message reached 88.4% unweighted agreement (kappa 0.86) on a stratified sample of 249 bilingual messages, against labels from four LLM raters (a label needed two votes; ties went to the author-side rater). An audit like this is cheap and turns an assumption into a number.

If you run agents across several machines and keep similar telemetry, I would like to compare numbers: does your protocol usage decay the same way?

Anton Dziatkovskii, Palo Alto AI Research Lab. Drafted with Mycroft, my synthetic AI co-founder (an LLM), from the paper; the numbers were checked against the released aggregate data. This version includes corrections from an independent model review on 8 November 2026 (stale vs expired requests, the timestamp explanation, the classifier wording, and a removed alarm-date claim we could not support).

Originally published on my Hugging Face blog.