AI & ML
Decided or Open? Comparing Program Repair Techniques When Patch Correctness Is Unknown
arXiv cs.SE0 views
arXiv:2611.23287v1 Announce Type: new Abstract: Comparisons of automated program repair (APR) techniques depend on knowing which patches are correct, but many patches are never checked and some archived test results are wrong. We propose reporting each comparison as decided or open. For two techniques that each pick one patch per bug, we compute the full range of the difference in their success rates over every possible value of the unknown labels, and we call the comparison decided when that range excludes zero. Because a test can only show that a patch is wrong, more testing can move this range in one direction only. This lets us tell, before any test runs, which open comparisons testing could never settle. We study eight settings, including 55,466 candidate patches for Defects4J bugs. The labels already in the archive decide 12 of 16 comparisons between a learned patch selector and its baselines, and adding further evidence decides all 16. In a campaign whose predictions we published in advance, all eight predictions held. We also show that copies of the developer's fix, which must pass, are recorded as failing for 33.5% of GPT-4's copies but for only 2.5% of the RepairLLaMA model's, so the reliability of archived test results differs between models. With these checks, researchers can see which comparisons need more evidence and which kind of evidence can help.
Source:https://arxiv.org/abs/2611.23287PreviousThe Illusion of Relief: How GenAI Redistributes Cognitive Workload in Software OrganizationsNextAn Empirical Study of Visual Artifacts in Pull Request Descriptions
Related posts
Ruby2ßhell: Pre-authentication RCE on stock Rails
0Hacker News
Generalizing self-normalized importance sampling with couplings
0arXiv stat.CO
Bayesian Computation of Copula-based Multivariate Models with an Application to Doping Detection
0arXiv stat.CO
WAIVES: Weak-form Asymptotically-unbiased Instrumental-Variable Estimation for SDEs
0arXiv stat.CO
Treat Your Honeypot Analysis Like Code: Git, Pull Requests and a Human Gate for AI [Guest Diary], (Wed, Nov 18th)
0SANS ISC
ISC Stormcast For Thursday, November 19th, 2026 https://isc.sans.edu/podcastdetail/10186, (Thu, Nov 19th)
0SANS ISC