An AI Reviewer Said PASS. Could the Implementer Have Written the Receipt?

An executable synthetic test for a subtle failure in AI-agent review evidence.

Two AI agents can approve the same pull request. They can even attach two different names to their reports. That still leaves a question: who actually had the authority to produce the review evidence?

In discussions following my posts on what an AI agent means by “done” and whether two AI reviewers are truly independent, TokenLat made a useful distinction: establish that the required evidence exists and is bound to the right candidate first; then test whether an unauthorized producer could make that evidence look independent.

I turned that question into a small, fully synthetic Node.js fixture. It uses no AI model, real repository, or production credentials.

The one-field problem

Imagine an evaluation event:

{ "id": "e-1", "type": "evaluation", "candidate": "sha256:synthetic-pr-head-a", "claimedProducer": "independent-reviewer" } 

A naive checker might look for an evaluation event, compare the candidate, see independent-reviewer, and return PASS.

But what if the implementer emitted that exact JSON?

Changing a producer label is not a change of authority. If the same component can write both the alleged independent review and the receipt naming its author, the evidence is self-attestation.

Four cases, not just one happy path

I used one fixed candidate identity and three event types: declaration, evaluation, and acceptance. The fixture tests:

The fourth test only makes sense after the first three: otherwise a rejection could simply be caused by a missing or mismatched record, not a failure to verify producer independence.

A claimed name is not an authenticated identity

The essential check is not:

receipt.claimedProducer === 'independent-reviewer' 

It is closer to:

hostRecord.authenticatedRole === 'independent-reviewer' && hostRecord.candidate === receipt.candidate && hostRecord.eventDigest === digest(receipt) 

The crucial distinction is where hostRecord comes from. It must come through a separately controlled, authenticated evidence channel. A producer-supplied field named authenticatedRole would accomplish nothing.

In my toy fixture, a mock host journal supplies that trusted provenance. The check is deliberately narrow: it demonstrates how a string-only verifier can wrongly accept a forged reviewer label, while a verifier using independent provenance rejects it. The mock does not provide real authentication, signature verification, filesystem isolation, or runtime enforcement.

I ran the four cases and one additional regression test against the naive checker. All five Node.js tests passed. In the spoof case, the string-only verifier returned PASS and the provenance-aware toy verifier returned DENY.

What this does—and does not—say about ARC

I maintain Agent Role Contracts (ARC), which checks the consistency of declared agent roles, task scopes, and review relationships. That is a declaration-level check.

An ARC declaration by itself does not authenticate the reviewer who produced an event, prevent an executor from writing to a review channel, or prove that the reviewed revision was the one merged. Those responsibilities belong to the host, authorization system, CI, and final acceptance process.

The same discipline applies to exemptions. “Independent review not required for this task” should be an explicit, authorized scope decision. Silence or a missing event is not an exemption.

The question I would ask your agent system

If the implementer changed just the producer field in its own evaluation receipt to the expected reviewer name, what independent control would prevent that receipt from authorizing the next step?

I am particularly interested in minimal examples that prove the boundary without adding an entire second orchestration platform.

Disclosure: This is a synthetic fixture and a proposed evidence boundary, not a claim that ARC currently enforces runtime producer authentication. The runnable .mjs file accompanies the draft and should be published at a stable source URL before referring to it as publicly reproducible.