Most of my PRs now get written by an agent and reviewed by another agent before I even open them. The AI review catches the small stuff. So the slow part is me, reading a diff I didn't write and deciding if it's ok to merge.

And honestly most of those PRs are boring. A config tweak, a test, a small fix. A few of them are the ones that break things. I wanted a way to tell which is which before reading the diff.

That's what opendrs does. It gives every pull request a risk score between 0 and 1, computed from the repo's own git history. Then you set a rule like "under 0.3 and the AI review passed, approve it and send it to the merge queue". Everything else waits for a person.

It runs as a GitHub Action, a CLI, an MCP server and a Claude Code plugin. MIT licensed, nothing to host.

Repo: https://github.com/error-path/opendrs

Meta already does this (internally)

The idea isn't mine. Meta wrote about a "Diff Risk Score" in a 2024 paper (Abreu et al., Moving Faster and Reducing Risk: Using LLMs in Release Deployment) and a 2025 post on their engineering blog. They use it to gate code: low-risk diffs move faster, risky ones get more eyes, and on weekends and around peak events they hold back more.

Their baseline model is surprisingly plain. A logistic regression over about a dozen features, picked from about 100: size of the change, how many files it touches, how many people have worked on those files, past incidents in the file, the author's experience, complexity, whether there's a test plan.

They didn't release it, so I tried building it from git alone. opendrs isn't affiliated with Meta, it just follows what they published.

What it measures

Every signal comes from the diff, the git log and an import graph:

  • size of the change, and size relative to the file
  • how spread out it is (files, top-level dirs)
  • past fixes and reverts in the touched files
  • how many different people have touched those files
  • the author's own experience with them
  • complexity (a rough count of branch keywords, not a real parser)
  • blast radius: how many files depend on the changed ones, up to 3 levels deep
  • critical paths you mark yourself
  • missing or deleted tests

A logistic regression turns that into a probability. The nice thing about a linear model is that you can show how much each signal pushed the score, so the PR comment tells you why, not just a number.

Here's the CLI on a clone of pallets/itsdangerous, after a 2-line edit to serializer.py:

opendrs risk 0.32 YELLOW -> human review 1 files, +2 -2 (672971d..0bf8f91), default model, opendrs 0.5.2 Top contributions (push on the logit): +0.70 tests.missing 1 source file changed with no test lines added +0.40 size.churn 4 lines changed across 1 file (0 new) +0.39 blast.dependents 6 file(s) transitively depend on the changed files (depth ≤ 3) +0.24 diffusion.authors 4 distinct authors have touched these files +0.16 complexity.touched 48 branch keywords in 1 changed code file Policy: - tier yellow: score 0.32 ≥ green threshold 0.30

(It also suggests three reviewers from the file history, I cut that part out here.)

Where the labels come from

Meta has a list of real incidents. I don't. So opendrs labels past commits two ways: reverts ("This reverts commit ..."), and an SZZ-style trick where you find commits that look like fixes, blame the lines they changed, and mark whatever commit last touched those lines as the one that introduced the bug. If you keep your own incident log you can add it, and that's the best label you'll have.

It's noisy. SZZ blames reformatting commits, and "fix typo" looks like a fix. Keep that in mind for every number below.

Numbers on a public benchmark

ApacheJIT is a public dataset of 106,674 commits from 15 Apache Java projects, each labelled buggy or not. I fit on the training years (2003-2016) and scored the test years (2017-2019): 24,686 commits, 5,818 of them buggy. (apache/hadoop is left out, it has no buggy labels at all.)

Model AUC Buggy commits in riskiest 10% in riskiest 30%
opendrs, default weights 0.771 24.9% 59.6%
opendrs, calibrated 0.823 29.7% 66.8%
Classic JIT regression (Kamei) 0.821 29.6% 66.2%
DRS-OSS, fine-tuned Llama (different split) 0.895 50.3% 86.4%

So calibrating on your own history helps a lot. And calibrated opendrs ties the classic regression that uses the dataset's own metrics. That's a tie, not a win. Per project it's ahead in 4 of 14 and behind in 10.

DRS-OSS is clearly better. It's a fine-tuned Llama 3.1 8B that actually reads the diff, and you have to host it yourself. That gap is basically what reading the code buys you.

Does adding an LLM help?

opendrs has an optional LLM judge. It reads the diff, gives a probability of an incident, and puts each hunk in a category. Safe ones like refactor, docs, tests, logging. Risky ones like logic_bug, security, performance.

One rule I set early: the LLM can only raise the score, never lower it. If someone writes a PR that talks the model into "this is totally safe", they gain nothing.

I ran Claude Sonnet 5 over 800 sampled test commits (about $55 in API calls). Two results:

  • As a ranking signal it barely helped. About +0.02 AUC, which is within noise.
  • As a gate it was really useful. Commits it called all-safe with confidence 8 or more were buggy 5.3% of the time, against a 25.5% base rate. About 28.5% of commits passed.

So the LLM works better as a filter in front of auto-approval than as part of the score. That's also roughly how Meta's RADAR paper uses it.

Things that didn't work

I keep an experiment log in the repo, failures included. Some of them:

  • Reviewer-checklist signals (concurrency, removed safety code, weakened tests) added nothing to the ranking. AUC 0.782 with them and without.
  • Adding the rest of the classic JIT features (entropy, file age and so on) made it slightly worse, 0.819 to 0.817.
  • On big old repos the default weights push almost every PR into red. In a field test with an earlier version, 7 of 8 non-docs PRs on psf/black scored red, whether it was a 22-line whitespace fix or a 759-line release pipeline rewrite. I tried re-centring the defaults and it got worse (red share went from 25.6% to 40.3% on ApacheJIT). The real fix is calibrating on your repo.

I write down the question and the pass bar for each experiment before running it. A few of my own ideas died that way. Annoying, but useful.

Trying it on real repos

I ran calibrate and backtest on 27 open-source repos with no config. In the second run of 18 repos, the untuned score beat random in 17 of them (median AUC 0.71). Calibration helped in all 9 repos of the first run, but in the second run it helped in 11, was flat in 1 and hurt in 6. Small repos give jumpy numbers.

I also checked against what happened after merge. On 1,010 merged PRs from six active repos, the ones opendrs marked "careful" were later fixed or reverted 14.8% of the time, the "quick" ones 1.7%, against 9.8% overall.

From a score to an approval

A score alone shouldn't approve anything. The Action runs ordered rules and the first match wins:

  • Some paths always go to a human: CI workflows, migrations, Terraform and other infra files, agent config like .claude/ and .mcp.json, and the opendrs config itself.
  • Binary files, submodule bumps, deleted tests: human.
  • Over the size cap: human.
  • Not green: human.
  • A signal couldn't be computed and could have changed the tier: human. Unknown is never treated as safe.
  • Drafts, or a reviewer asked for changes: human.
  • Your AI review check hasn't passed yet: wait.
  • 10% of eligible PRs get sampled for a human anyway, so you keep measuring what the gate lets through.

PRs from forks are never approved. Config is read from the base branch, so a PR can't loosen its own gate. And it won't approve anything unless your branch rules dismiss stale approvals on push.

What's still open: an experienced author lowers the score, so a compromised maintainer account is a real risk. Required checks are matched by name. Test signals count lines, not assertions, so an assertion edited into a no-op isn't seen.

Approval is off by default. My advice: run it comment-only for two to four weeks and see if the scores match your gut.

Try it

On a branch with some commits:

npx @error-path/opendrs score

On main there's nothing to compare against, so add --base HEAD~5. No config, no API key, nothing leaves your machine.

To see how it would have done on your own history:

npx @error-path/opendrs backtest npx @error-path/opendrs calibrate npx @error-path/opendrs backtest

As a GitHub Action (comments and labels only, until you turn approval on):

- uses: actions/checkout@v5 with: fetch-depth: 0 # it needs the history - uses: error-path/opendrs@v0

In Claude Code: /plugin marketplace add error-path/opendrs, then /plugin install opendrs@opendrs. Or as an MCP server for any agent: npx -y @error-path/opendrs mcp.

Where it's at

It's early. I'm the only maintainer and as far as I know nobody runs the Action in public yet. A lot of the code and docs were written with Claude and reviewed by me, and every number in this post comes from a script in the repo.

What would help most: run backtest on your repo and tell me where the ranking feels wrong. Issues are open.