If the reviewer posts 200 comments and nothing improves, you measured the wrong thing.
A recent dev.to post nailed the moment we’re in: AI promoted every developer to reviewer, and nobody measured whether we got worse. The tooling arrived faster than the accountability. Teams wired an agent into CI, watched it post confident comments on every PR, and called it done.
Posting comments is not the job. The job is fewer bad merges. And almost nobody is checking whether the second follows from the first.
This is the measurement sequel to the AI-reviewer setup: the plan for proving your reviewer earns its place in the pipeline — or finding out it doesn’t, while you can still fix it.
The vanity trap
Activity metrics feel good. Outcome metrics tell the truth.
The first dashboards teams build for an AI reviewer measure the reviewer’s activity, not its impact. Comments posted. PRs reviewed. Review latency. Uptime of the reviewer job. These numbers always look good, because they’re measuring whether the tool ran — and the tool always runs.
None of them answer the only question that matters: is the codebase getting safer? An agent that posts 200 comments a week, of which engineers dismiss 190, is not reviewing. It’s generating notification noise with extra steps. Activity metrics will never tell you that. They’re designed to feel productive.
The five metrics that matter
1. Finding dismissal rate — and dismissal reasons. Of every finding the reviewer posts, what fraction gets dismissed without action? Track the rate per severity and per rule. But the rate alone isn’t enough: log the reason for each dismissal. “False positive,” “won’t fix,” “already handled,” “not applicable to this service” — each reason points at a different fix. A rule dismissed as “not applicable” thirty times shouldn’t exist in your policy. Dismissal reasons are the raw material for tuning; without them you’re guessing.
2. Escaped defects. What merged despite the reviewer? This is the metric teams avoid because it’s uncomfortable. Sample merged PRs each week — especially ones the reviewer flagged and a human dismissed — and check whether the flagged issue was real. Every confirmed escape is a double signal: the reviewer was right, and the process let it through anyway. If your escape count is zero, you’re probably not looking hard enough.
3. Signal ratio. Of the findings that aren’t dismissed, how many were HIGH severity versus noise? A healthy reviewer is quiet most of the time and loud about the things that matter. Watch the ratio of HIGH findings to total findings over time. When it decays — more comments, same number of real issues — your policy has drifted into nagging. That’s the leading indicator that engineers are about to start ignoring the reviewer entirely.
4. Time-to-merge delta. Compare merge times for PRs before and after the reviewer, and for PRs with findings versus without. A reviewer that adds meaningful review should add some time on PRs with real findings — that’s the cost of catching things — and near-zero time on clean PRs. If every PR gets slower, the reviewer is friction. If no PR gets slower, the reviewer might not be gating anything.
5. Agent–human agreement rate. On a sample of PRs, have a human reviewer independently note what they’d flag, then compare against the agent’s findings. You’re measuring two things at once: findings the agent caught that the human missed (the value add), and findings the human caught that the agent missed (the coverage gap). Run this monthly on a small sample — ten PRs is enough to see the shape. The gap list is your policy roadmap.
None of these need absolute targets on day one. What matters is the trend: dismissal rate falling, escapes flat or falling, signal ratio stable, merge delta explainable, agreement converging. Direction beats thresholds when you’re starting out.
What to measure vs. what most teams measure
Comments posted: The tool ran — Dismissal rate + reasons
PRs reviewed: Coverage of the pipeline — Escaped defects per week
Review latency: CI performance — Time-to-merge delta
Findings per PR: Verbosity — Signal ratio (HIGH vs noise)
Reviewer uptime: Infra health — Agent–human agreement rate
The left column is easy to build and useless for decisions. The right column is slightly harder and the only thing that tells you whether to keep, tune, or kill the reviewer.
The findings log: what to record
Everything downstream reads from one file. Keep it boring — one JSON line per finding, appended by the reviewer job:
{"ts": "2026-09-20T14:02:11Z", "rule_id": "tf-no-public-s3", "severity": "HIGH",
"pr": 4821, "file": "modules/storage/main.tf",
"outcome": "dismissed", "dismissal_reason": "not-applicable"}
Five fields do the heavy lifting: rule_id (which rule fired), severity (what the reviewer claimed), outcome (fixed, dismissed, or merged-as-flagged), dismissal_reason (one of a fixed vocabulary — false-positive, wont-fix, already-handled, not-applicable), and pr (so you can join back to the diff). The fixed vocabulary matters more than it looks: free-text reasons don’t aggregate, and aggregation is the entire job. If your reviewer posts findings as PR comments today, a small job that scrapes them into this shape nightly is a fine week-one implementation — the schema is the commitment, not the pipeline.[
One log in, three reviews out, one policy decision: the measurement pipeline.
The 30-day plan
Days 1–7: instrument. Log every finding with its severity, rule ID, PR, and outcome — dismissed, fixed, or merged-as-flagged. This is the foundation; everything else reads from this log. Add dismissal reasons as a required field wherever engineers dismiss a finding. No log, no measurement.
Days 8–14: first dismissal review. Pull the week’s dismissals. Group by rule and reason. You’ll find the obvious cuts immediately: the rule everyone dismisses, the severity that’s always wrong, the check that fires on generated files. Cut or fix them now — don’t wait for a “tuning phase.”
A first dismissal review usually looks something like this (illustrative numbers — one week, roughly 200 findings):
tf-require-tags: 41 — 38 (93%) — not-applicable (test fixtures) — Scope the rule to non-test paths
go-ineffectual-assign: 22 — 3 (14%) — — — Keep; engineers fix it
tf-no-public-s3: 9 — 1 (11%) — — — Keep; the one dismissal was a real debate, worth having
md-line-length: 34 — 34 (100%) — wont-fix — Delete the rule outright
k8s-latest-tag: 18 — 12 (67%) — already-handled (pinned in base image) — Reword to check the rendered manifest
A reviewer earns trust the same way a human does: right about what matters, quiet about the rest.
Two rules account for most of the noise. That’s the normal shape of it: tuning is rarely fifty small decisions — it’s two or three deletions and a rewording. The table also tells you what not to touch: go-ineffectual-assign at 14% dismissal is a rule engineers respect. Leave it alone.
Days 15–21: tune the policy. This is the loop: fewer rules, sharper wording, severity gates that match how engineers actually respond. Fifteen sharp rules beats fifty noisy ones — and the dismissal data tells you exactly which thirty-five to cut. Re-run the agreement sample after tuning.
Days 22–30: read the deltas. Compare the second half of the month against the first: dismissal rate, signal ratio, time-to-merge. If the trends moved in the right direction, the reviewer is earning its place. If they didn’t, you now have evidence about why — which is more than most teams ever get.
Then keep the loop running monthly. Reviewers decay: codebases change, models change, policies drift. The teams that keep measuring are the teams whose reviewers still work a year later.[
What “working” looks like over ninety days: dismissal rate falls, signal ratio holds, escapes stay flat.
Fifteen sharp rules beats fifty noisy ones. An agent reviewer earns trust the same way a human one does: by being right about the things that matter, and quiet about the rest.
The first-30-days checklist
[ ] Every finding logged with severity, rule ID, PR, and outcome
[ ] Dismissal reason captured on every dismissal
[ ] Weekly dismissal review on the calendar — same time, same people
[ ] Escaped-defect sample: 5–10 merged PRs reviewed weekly
[ ] Signal ratio tracked: HIGH findings vs total, trended weekly
[ ] Time-to-merge baseline captured before tuning begins
[ ] Monthly agent–human agreement sample: 10 PRs, independent human pass
[ ] Policy cut at least once based on dismissal data, not gut feel
[ ] Noisy rules removed or reworded; severity gates match reality
[ ] Day-30 read-out: trends vs day 1, keep/tune/kill decision written down
The uncomfortable version of this post is one sentence: if you can’t show the reviewer catching real issues that humans missed, you don’t have a reviewer — you have a very expensive linter with good PR. Measurement is what separates the two.
Honey Kiran · DevOps & Platform Engineering
I write about SRE, monitoring, Terraform, and the unsexy work that keeps systems alive at 3am.
Start logging your reviewer’s findings and outcomes this week — and send this to the teammate who deployed the AI reviewer and never looked at the dismissal rate.
Disclaimer
This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.
© 2026 Honey Kiran. All rights reserved.
