AI & ML
Decided or Open? Comparing Program Repair Techniques When Patch Correctness Is Unknown
arXiv cs.SE0 views
arXiv:2611.23287v1 Announce Type: new Abstract: Comparisons of automated program repair (APR) techniques depend on knowing which patches are correct, but many patches are never checked and some archived test results are wrong. We propose reporting each comparison as decided or open. For two techniques that each pick one patch per bug, we compute the full range of the difference in their success rates over every possible value of the unknown labels, and we call the comparison decided when that range excludes zero. Because a test can only show that a patch is wrong, more testing can move this range in one direction only. This lets us tell, before any test runs, which open comparisons testing could never settle. We study eight settings, including 55,466 candidate patches for Defects4J bugs. The labels already in the archive decide 12 of 16 comparisons between a learned patch selector and its baselines, and adding further evidence decides all 16. In a campaign whose predictions we published in advance, all eight predictions held. We also show that copies of the developer's fix, which must pass, are recorded as failing for 33.5% of GPT-4's copies but for only 2.5% of the RepairLLaMA model's, so the reliability of archived test results differs between models. With these checks, researchers can see which comparisons need more evidence and which kind of evidence can help.
Source:https://arxiv.org/abs/2611.23287PreviousThe Illusion of Relief: How GenAI Redistributes Cognitive Workload in Software OrganizationsNextAn Empirical Study of Visual Artifacts in Pull Request Descriptions
Related posts
![Quadruped Robot Software — Ch.12: Quadruped Case Studies [KR]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fawosw4u7jitr9h27f7c5.png)
Quadruped Robot Software — Ch.12: Quadruped Case Studies [KR]
0dev.to
How Python Is Used in Artificial Intelligence in 2026: Applications, Benefits and Career Opportunities
0dev.to

What Happens in Those Few Seconds Between Pressing the Power Button and Seeing the Windows Login Screen?
0dev.to
Getting cited by ChatGPT and Claude: what the research supports, and a free audit tool
0dev.to
Why Photoshot keeps the original color when you edit in black and white
0dev.to
ISTQB CTFL in 2027: is it still worth it? What changes, what does not
0dev.to