Last week I published a benchmark where our reviewer caught 10 out of 10 re-introduced vulnerabilities, and then spent four paragraphs explaining why that number is close to worthless. The main reason was that it measured recall and never precision. A tool that finds every planted bug and also comments on clean code is not half-good, it is unusable.
So here is the missing half. Twenty ordinary commits, nothing planted, and every comment read by hand.
The samples
Twenty commits from psf/requests, in reverse chronological order, filtered to those that touch src/requests/**.py and are not pure type-annotation or CI churn. All twenty were merged by maintainers. None of them re-introduces a vulnerability.
For each one: check out the commit, run the security agent against HEAD~1 with no PR description and no hints, and record what it says. The security agent alone, a small model, one pass each. Twenty out of twenty completed, no failures.
The result
| Count | |
|---|---|
| Commits reviewed | 20 |
| Commits that received a comment | 3 |
| Total comments | 4 |
| Comments I would call noise | 3 |
Seventeen of the twenty commits got complete silence. That is the part that matters most: on most real changes the reviewer has nothing to say, which is the correct answer.
What it said when it did speak
a044b02 — "Move DigestAuth hash algorithms to use usedforsecurity=False" (5 lines changed), 2 comments, both noise.
The commit adds usedforsecurity=False to the hashlib calls in HTTP Digest auth. That flag exists so the code keeps working in FIPS-restricted environments, where MD5 and SHA1 are otherwise blocked. It is a deliberate, well-understood annotation: this hash is part of a wire protocol, not a security primitive.
The reviewer reported a high severity finding titled "FIPS security bypass via usedforsecurity=False", and a medium one about weak hash for cnonce generation. Both restart from "MD5 is weak", which is true and has been true since 2004, and neither engages with what the diff does. The maintainers added the flag because the hash is not a security control. The reviewer read the flag as a security control being disabled.
fd62809 — "Preserve leading slashes in request path_url" (2 lines changed), 1 comment, overstated.
The reviewer reported high: removing //-to-/ normalization may enable request smuggling, because some proxies resolve //evil.com/path as a network-path reference. There is a real debate in that area, and the finding is not stupid. But the change was made on purpose, after discussion, and "may enable request smuggling" is a much stronger claim than the evidence in a two-line diff supports.
a4f9a59 — "Port bpo-39057 to Requests" (proxy bypass fix), 1 comment, real but marginal.
The finding: if no_proxy contains an entry of only dots, host.lstrip(".") produces an empty string and the suffix match degenerates. That is a genuine edge case. It is also the kind of thing that will never happen in a real deployment, which is presumably why it was rated low.
The failure mode is one thing, repeated
All three false positives have the same shape: the diff contains something that looks like a security control being removed or disabled, and the reviewer reacts to the shape without recovering the intent.
usedforsecurity=False reads as "we are telling the crypto library to skip security". // normalization being removed reads as "we stopped sanitising the path". Both are locally true statements about the text and both are wrong about the change, because the reason lives outside the diff: in the FIPS requirement, in the linked issue, in the discussion that decided the trade-off.
Which is the mirror image of the problem I wrote about last time. A reverted security fix is easy to catch precisely because the reason is obvious from the diff (a guard clause disappeared and nothing replaced it). Real commits that touch security surfaces are hard because the reason is usually absent from the diff and the shape looks identical to a regression.
An LLM reviewer with no PR description is being asked to tell those two apart from the patch alone. Most of the time it can't, so the honest design goal is not "understand intent", it is "say less when intent is missing".
What this does and does not support
What it supports: on ordinary changes this reviewer is quiet. Fifteen percent of commits got a comment, and the absolute cost of that is low. Four comments across twenty commits means roughly one extra thing to read per five PRs, not a wall of noise.
What it does not support: a precision rate. Four comments is not a sample. I also judged "noise" myself, and I know what the diffs were supposed to do, which a reviewer in CI would not.
Three caveats worth stating plainly:
- One project, one language.
requestsis Python, small, and unusually well reviewed. A codebase with more security-sensitive surface will produce more comments, not fewer. - No PR description, by design. That is the harshest setting. In CI the reviewer usually gets a title and a description, which should reduce these false positives. It can also increase them, if the description hands the model a thesis to confirm.
- Security agent only. The logic and performance agents were not running. A full review is noisier, and I have no honest number for it yet.
What changed in the product
The two findings above are the reason the reviewer now has a confidence floor and only reports findings anchored to a specific line with a concrete failure path. "MD5 is weak" has no failure path, so it does not get to speak. "This path is used for auth, therefore disabling FIPS checks on it is a bypass" has no failure path either, because the premise is wrong.
Recall went down when we added that. On the twenty commits it would have cost us nothing, since none of the four comments were actionable. On the ten vulnerability cases it cost us nothing either. That is the trade I want: a reviewer that is wrong less often, at the price of being quiet more often.
The next version of this needs a hundred commits across three or four projects, with PR descriptions included, and two people labelling the comments independently. Four comments is an anecdote with a table, and I would rather say so than round it into a percentage.

