Can a dependency scanner's finding go straight into a code review report, with no model checking it? A vulnerable version in the lockfile is a fact, so I thought yes. I cut the model out of my code reviewer and had blind judges check those findings against the code. Of 10 distinct issues, none was fully real. After a fix that halved the noise, 0 of 5.

Gru's plan: skip the LLM for CVE findings, save about 6M tokens on one project, let blind judges check the findings, 0 of 10 real

The reviewer I tested this on is a Claude Code plugin I built that reviews a whole backend, not a PR diff. I wrote about Opus vs Sonnet as its reviewers and about testing CodeGraph in it. Here it's the instrument. The question is what evidence makes a scanner hit worth a reviewer's time, and what it costs to find out. Three setups of the reviewer appear below:

  • A, the early one: a model reads every advisory, a verifier checks what it keeps.
  • B: A without that model. Rules turn advisories into findings.
  • C: B with the fixes this post arrives at.

Three setups, how a scanner advisory reaches the report. A: advisories go to the security pass, a model, then the verifier, then the report with 2 dependency findings, 0 false. B: advisories go through rules (imported anywhere, or a direct dependency) straight to the report, 10 dependency findings, 4 false, 0 real. C: stricter rules (imported in the reviewed code) send reached packages to the report, 5 findings, 3 false, 0 real, and the rest to an undecided list that a model reads only on request

Why I took the model out

Before any agent starts, the reviewer runs per-language static tools in a Docker image. Three of them report vulnerable dependencies: govulncheck for Go, cve-lite for JavaScript lockfiles, and osv-scanner for everything else (PyPI, crates, the rest). Their output reaches the agents as leads, not as findings.

In setup A, every advisory went through two models. A project-wide security pass read all of them and kept the ones it thought the code could reach. Then the verifier tried to refute those findings, like any other finding.

Setup A: static tools pass every advisory to the project-wide security pass, which is a model agent; its findings go to the verifier and then to the report

That was expensive. In the model comparison from my earlier post, the security pass and its verifier were 21% of the cost of all eight runs. On one Python service with 231 advisories, that pass alone used about 6M tokens and turned them into one or two findings. It also wasn't reliable. In one run the model left out an advisory the scanner had reported, right in front of it.

So in setup B I replaced the model with rules. A known vulnerability is a fact the scanner establishes: the lockfile resolves version X, and version X has CVE Y. No model should be able to drop it. The rules were:

What a tool shows critical high
The vulnerable function is called (govulncheck's call graph) WILL_BREAK WILL_COST
The package is imported (cve-lite's import check) WILL_COST WILL_COST
No tool can tell, but it's a direct dependency WILL_COST WILL_COST
Not called, not imported, development only listed listed

WILL_BREAK, WILL_COST and WILL_ROT are the reviewer's three tiers: it will break in production, it will cost money or incidents, or it makes the code harder to change. These findings skipped the verifier because there was nothing to refute: the scanner had already established them.

How I tested it

I keep a fixed testbed for comparing setups of the reviewer: four private projects, each pinned to one commit and reviewed with the same scope every time.

Project Lines in scope Dependency files
Go service 827 go.mod
Rust proxy 1,217 Cargo.lock
NestJS app in an Nx monorepo 6,721 one root yarn.lock for all apps
Python pipeline 6,317 six uv.lock files (root + five services)

Each setup reviewed each project once, with Sonnet, the same aspects, and the tools in the reviewer's Docker image. Then one Opus judge per project got the kept findings of two setups under neutral letters, plus read-only access to the code. The judge never learned which setup wrote which list. It clustered the findings into distinct issues and marked each one real, partly (the mechanism exists, the impact is overstated) or false, with the tier it thought was right and a file:line reason.

Scanner findings were labelled as such in the judge's input, with one extra instruction: check that the lockfile really resolves that version, that the reviewed code uses the package, and that the advisory matters for how it's used.

The first round compared A with B. Two more rounds with fresh judges came later: B against C, and A against C.

The result

Blind Opus judges, 4 private projects, setups B and C. Issues a model reviewed and verified: 0 false in 154 verdicts. Scanner findings with no model: 0 real in 15 verdicts. Real / partly / false: model B 64/12/0, model C 67/11/0, scanner B 0/6/4, scanner C 0/2/3

The model-reviewed findings held up. Across all three setups and three judging rounds, no finding that a reviewer wrote and the verifier kept was false.

The scanner findings did not. Setup B put 15 scanner entries into the reports, which the judge clustered into 10 issues: 0 real, 6 partly, 4 false. Every one was filed as WILL_COST. The judge put every one at WILL_ROT: upgrade when convenient.

The scanner was never wrong about the version. Every vulnerable version matched the lockfile. What was wrong was the step from "this version has a CVE" to "this costs you money".

Three ways a fact turned into noise

The monorepo. The NestJS app lives in an Nx monorepo with one root yarn.lock for every app. I reviewed only one of the apps. cve-lite's import check searches the whole directory of the lockfile, so an import in any other app counted as "imported". Four of the nine packages from that project were imported only by other apps.

One package, six lockfiles. The Python project is mine: a local video-dubbing pipeline with a root uv.lock and one per service. All six resolve the same certifi, which has an advisory about compromised intermediate certificates of one CA. The reviewer grouped advisories per lockfile, so the report had the same finding six times.

Spider-Man pointing at Spider-Man: one advisory, six findings, from root uv.lock, service A uv.lock and service B uv.lock

"Direct dependency" as evidence. osv-scanner can't tell whether Python code reaches a vulnerable function. My rule treated a direct dependency with a high advisory as reachable anyway. That's how certifi became WILL_COST with no tool showing anything about its use. The judge did find that the code uses certifi.where() as the trust store for model downloads, so the advisory is relevant. But exploiting it needs a man-in-the-middle holding one of those compromised keys, against a tool that runs on one laptop. Partly real, WILL_ROT.

While fixing these I hit a fourth trap, one any pipeline can fall into: the scanner read my own tooling's leftovers. osv-scanner scans the repository recursively, and earlier review runs leave their tool inputs in a run directory inside the repo. Among them were the dependency lists written for GuardDog, the malicious-package checker, and osv-scanner took them for real manifests. 205 of the 736 advisory hits on the Python project came from those files. They became rows in the report's table, for package versions the project doesn't use.

Setup C, and a second round of judges

C is B with three rules changed and one scanner setting:

  • An import counts only if it's in the files under review. Imported only elsewhere (another app, or only tests) is now "undecided".
  • One package with the same advisories in several lockfiles is one row that names every lockfile.
  • A direct dependency alone is no longer evidence. Without a call graph or an import in the reviewed files, the package is undecided.
  • osv-scanner skips the reviewer's run directory.

Undecided packages are listed in the report. A model looks at them only when you ask for it with a flag, which brings back the cost of setup A for exactly those packages.

Setups B and C: static tools pass advisories to advisory rules with no model; reached packages become findings without the verifier, undecided ones go to the project-wide security pass only when opted in

Here's where those rules sit in the review method I use, shown as one review wave from the reviewer's docs. Per module, separate reviewers look at the code, security, performance and tests, and a verifier tries to refute every finding they make. The architect and a project-wide security pass run alongside. The advisory rules sit in their own lane: a reached package gets a finding code directly, an undecided one goes to the security pass only on request.

A review wave: per module the module, security, performance and test reviewers feed a verifier, then duplicates are merged and findings get codes; the architect and the security pass run alongside with their own verifier; advisory rules in their own lane send reached packages straight to codes and undecided ones to the security pass when opted in

I reran all four projects with setup C and judged again, this time B against C, with fresh judges and a new letter assignment. The new judges gave B's scanner findings the same 0 / 6 / 4 as the first round, which made me trust the comparison more.

From scanner hits to real issues, four projects: setup B 1046 hits, 203 rows, 15 findings without a model, 0 real; setup C 842 hits, 125 rows, 5 findings, 0 real. B's hits include 205 phantom matches from the reviewer's own leftover run files

The noise went down a lot: 15 report entries became 5, and the rows in the dependency table went from 203 to 125. The five issues that left the findings were 4 partly and 1 false, so nothing real was lost.

Where the scanner findings went: B findings 6 partly and 4 false; C still findings 2 partly and 3 false; C moved to undecided 4 partly and 1 false. Real: 0 in every row

But the five that stayed are still not real: 2 partly, 3 false. All five are imported by the reviewed app. For none of them does the code meet the advisory's preconditions:

  • a microservices framework whose advisory is about its TCP transport, while the app talks gRPC and Kafka;
  • an ID generator whose advisory needs a zero, negative or fractional length, always called with a fixed length of 12;
  • a utility library whose vulnerable functions are never used, only two unrelated helpers;
  • an ORM whose advisory needs a query feature the app never calls;
  • an HTTP client whose advisories need attacker-controlled URLs or config, while the app calls one fixed base URL.

Is this a pigeon: my rule engine, looking at "the package is imported", asks "is this a vulnerability?"

An import is better evidence than "somewhere in the lockfile". It's still not evidence that the vulnerable code runs.

A against C: what the model did with the same advisories

To close the triangle, a third round of fresh judges compared A with C directly. C's five scanner findings got the same verdict as in the second round: 2 partly, 3 false.

What A's model did with the advisories was more interesting. In the NestJS app it read the same scanner output and reported none of the packages that B and C turned into findings. The judges agree none of them is real. Elsewhere it reported two dependency issues: an advisory in the Rust proxy's TLS library, where no tool could check reachability, and a Go standard-library advisory that only A's run of govulncheck reported. One judge rated the TLS finding real, another partly. The Go one was partly both times. No false ones.

Dependency findings, all three setups: A, a model reads them, 2 partly; B, rules with no model, 6 partly and 4 false; C, stricter rules, 2 partly and 3 false. A's two are a TLS-library and a Go stdlib advisory; one judge rated the TLS one real, another partly

So the model did the filtering the rules couldn't. It dropped the noise and kept the one advisory worth a look, which B and C only list as undecided.

All three side by side

Each setup ran once per project and went to two judges in different rounds, so every number below is the range between those two judges.

Three setups side by side, two blind judges each. Real issues found: A 65 to 66, B 64 to 67, C 67 to 70. False model-reviewed findings: 0, 0, 0. Dependency findings: A 2 (0 to 1 real, 0 false), B 10 (0 real, 4 false), C 5 (0 real, 3 false). Tier matches the judge: A 70 to 75%, B 57 to 70%, C 73 to 81%. Cost for 4 projects: $8.99, $8.68, $8.06

None of the setups changed how the reviewers read code, and it shows: the real issues land within a few of each other, and no setup has a false finding that a model reviewed. B's tiers matched the judges least often, partly because its ten scanner findings were all filed a tier too high. A cost the most, but by under a dollar here. Its expensive case was elsewhere: the Python service with 231 advisories.

Real issues per project, all three setups, two judges each. Go service: A 4 to 5, B 6 to 7, C 4 to 6. Rust proxy: A 7, B 5, C 8 to 9. NestJS app: A 27, B 25 to 26, C 28 to 30. Python pipeline: A 26 to 28, B 27 to 30, C 24 to 28. Go and Rust never had a scanner finding: the spread there is run-to-run and judge-to-judge noise

Per project the picture is the same. Go and Rust never had a scanner finding, so their spread (Rust: 5 real issues with B, 8 to 9 with C) is just different runs of the same model and different judges.

If you put scanner output into code review

What I take from these numbers, for any pipeline that turns scanner hits into review findings:

  1. Only a call graph earns an automatic finding. A scanner hit can skip the model only when a tool shows the vulnerable function is called. Today that means Go with govulncheck, and maybe Rust if osv-scanner's call analysis is worth the build time. An import, a direct dependency or a lockfile entry is weaker evidence, and in my runs it produced nothing real.
  2. Check imports in the code under review, not around it. In a monorepo with one root lockfile, an import in a neighbouring app says nothing about the app you're reviewing.
  3. One row per package. The same advisory in six lockfiles is one problem. Group by package and list the lockfiles.
  4. Keep the scanner out of your tooling's own files. Exclude run directories and generated manifests, or you'll review versions your project doesn't use.
  5. Give the short list to a model, with context. After the rules, a few imported packages per project are left. Hand each one to a model with the advisory text and the import sites, and ask the question my judges answered by hand: does this code use the vulnerable feature? That's setup A on five packages instead of 231 advisories, and the third round suggests a model does that job well.

Limits

  • Small sample. 15 scanner entries with B, 5 with C, all from two of the four projects. The Go project had one advisory row, marked not called. So I have no call-graph finding to judge, and can't claim call graphs work, only that the weaker evidence didn't.
  • One run per setup. The Rust swing shows how much a single run varies. The scanner part is deterministic; the model part is not.
  • The judges are models. Opus with the code and a written protocol, not people. B's and C's scanner findings each went to two different judges in different rounds, and the verdicts matched exactly. That helps, but it's still a model checking a model's tool.
  • I built the reviewer and wrote the judge protocol. The letters were blind, the setup wasn't.

Scanner output is a reliable fact about your lockfile and a weak claim about your code. If your tool turns one into the other without looking at how the package is used, check what it's reporting. Mine was reporting 15 findings with nothing real behind them.

What do you do with scanner hits in your pipeline: gate on them, triage them, or let a model read them?