Last week I wrote about ten runs of Open Council, a tool where three AI seats argue over a plan before any code ships. The comments were better than the post. Two points stuck with me. A single agent with no council passed all ten of those tasks, so easy tasks can't show whether the debate helps. And one commenter argued that the real gain came from running code, not from the roles.
So this time I tried something different. I pointed the council's review mode at code I didn't write: an open pull request in a popular opensource Claude Code plugin. Then I kept only what I could prove with a test that fails on the maintainer's own commit and passes after a small fix.
I'm not naming the project or the maintainer. This isn't about their code. Both bugs are the kind anyone ships.
The setup
council review takes a diff and runs three seats over it:
- Lead PM writes a short brief of what the change claims to do. On this run it was Gemini 3.8 Flash. I'd planned on a larger Gemini model, but my quota on it had run out.
- Chief Engineer checks every claim against the code and objects with a file and line. This was Claude Opus.
- Chief Architect rules on whether it should ship. This was Grok.
The pull request was huge, about 2 MB of diff. I reviewed one slice of about 108 KB: the module that edits config files on the user's machine, including .env files and a lock file that guards the writes. The rest of the PR was applied underneath, so the surrounding code was complete. It ran on version 0.3.2 with the quick profile, which allows at most 2 rounds.
It ended STALLED after 2 rounds. The seats never all signed off, which in review mode means "dont ship as is".
What it raised
Round 1 had 8 objections, all from the Chief Engineer. Here's how they sorted out:
- 3 were about the code and held up when I read it.
- 1 was about a code comment describing how another tool parses .env files. It might be true, but it can't be checked from that repo.
- 4 were about the brief the Lead PM wrote, not the pull request. That's the council arguing with itself. A maintainer doesnt care.
So 3 of 8 were worth sending, and I could turn 2 of those into a failing test.
One detail matters for the "is it the roles or the execution" question. The Engineer said at the top of its reply that it ran nothing, because the checkout had no test runner installed. Every finding came from reading. But for each one it also spelled out the test that would prove it. Those are the tests I ran.
Bug 1: a backtick at the start of a .env value swallows the lines after it
The editor can set a value in a .env file. When it writes a value, it wraps it in quotes if it contains a space, a #, a quote, a $ or a tab. A backtick isn't on that list, so a value that starts with one is written bare.
The reader, though, treats a backtick at the start of a value as an opening quote. It keeps reading until the next backtick anywhere in the file, and everything in between becomes one value.
The test uses a small file: API_KEY, then DATABASE_URL, then a comment that happens to contain a word in backticks, then PORT. It sets API_KEY to a value that starts with a backtick, then reads DATABASE_URL back. On the maintainer's commit, DATABASE_URL comes back undefined, because it's now part of API_KEY. Adding the backtick to the list of characters that need quotes is a one-line fix. The value gets written in single quotes and the test passes.
The test stops at its first failed check, so I never saw the next step fail. By the same logic, though, deleting API_KEY afterwards would delete the DATABASE_URL line and the comment with it. That's what turns a parsing quirk into lost config.
It takes a value that starts with a backtick and another backtick later in the file, so it's rare. But this code edits people's real .env files, and the fix is one character.
Bug 2: a lock holder that dies mid-wait isn't noticed (minor)
Writes go through a lock file that records the holder's process id. A waiter checks whether the holder is still alive and breaks the lock if it isn't. To avoid calling ps on every poll, it caches its verdict for as long as the lock file stays the same.
The cache also hides a crash. If the holder is alive on the first check and dies a second later, the lock file doesn't change, so the waiter keeps its "alive" verdict. The test kills the holder 1 second into the wait. On the maintainer's commit, the waiter sat out the full 15 second limit and then failed with an error naming a process that no longer existed. The fix re-checks only the cheap part on each poll, whether that process id still exists, and keeps the cache for the rest. With it, the waiter got the lock after 1,018 ms.
This one is minor. Nothing is lost. You get a 15 second stall and a misleading error, and the next try works.
The third finding held up when I read the code but has no test: an error message promising that a holder who lost the lock "will refuse to commit", when most callers never check. It's not in the count.
What I take from it
- This doesn't show the council beats a single agent. I didn't run a single agent on this pull request. On my easy benchmark a single agent passed 10 of 10 and the council 7 of 10. What this run shows is narrower: on real code that was open for review, it found two bugs nobody had flagged yet, and it described them precisely enough to test.
- Both proven bugs came from the Claude seat. The Gemini seat ran a smaller model than planned, and the Grok seat ruled on the findings but didn't raise any of them. So this run doesn't show that mixing models helped. It shows one careful seat inside a process that made it cite lines and propose tests.
- A finding isn't a finding until a test fails. Eight objections came down to two bugs I'd stand behind. If I'd sent all eight, the maintainer would have been right to ignore me.
- Half the noise was the council arguing with its own brief. In review mode, objections to the Lead PM's draft shouldn't reach the output. That's mine to fix.
I sent the two tests and both fixes to the maintainer as a comment on the pull request. Nothing was pushed to their repo.
What's next
The harder benchmark is the real test: tasks where a single agent sometimes fails, with the task set and pass criteria published before any run, and about ten single-agent runs per task to get a noise floor. It adds a repo with a misleading note and an idempotency-key case, both from comments on the last post. The breakdown of whether late sign-off concerns would have flipped the gate, which someone asked for, comes with it. None of that has run yet.
The next release, 0.4.0, fixes first-run setup. It's merged but not on npm yet. The version on npm today is 0.3.2. In a git repo, this reviews your uncommitted change:
npx open-council-cli review The code and full replays are at https://github.com/goshtasb/OpenCouncil.
This article was written with the help of AI. An AI agent also helped write the two repro tests, which ran on my machine against a temporary copy of the maintainer's commit.
