AI & ML
Revisiting the Survival of Agent-Attributed Code: How Measurement Choices Decide the Verdict
arXiv cs.SE0 views
arXiv:2611.21784v1 Announce Type: new Abstract: Studies of how long code from AI coding agents survives after it is merged reach conflicting verdicts: agent code is reported to be retained longer than human code, about as long, or modified sooner. Each study makes defensible measurement choices, but how much these choices decide the verdict is unknown. We revisit the survival of agent-attributed code in our earlier study: 854,139 lines that 2,802 merged pull requests (PRs) added to 215 repositories of the AIDev dataset. We vary nine measurement choices in five research questions and build a specification curve of 88 survival analyses, tested under three variance estimators, several observation windows, and permuted labels. For code from human-supervised agents, mostly GitHub Copilot, the choices change the size of the effect but not its direction. For code from the fully-autonomous agent Devin, they decide the verdict. Across repositories, 15 of 22 specifications report it modified sooner when lines are treated as independent, but only 3 when standard errors are clustered by repository, no more than permuted labels produce. Within repositories, no specification detects a difference. LLM intent classifiers agree with human labels about as well as two humans agree with each other, whereas keyword rules do not (kappa = 0.69 to 0.72 against 0.38), and the two reach different verdicts on why agent code is modified. Humans make most modifications to agent code under every unit and window. Re-run on our data, five published designs each change their result with a single choice. Whether agent code survives longer than human code is therefore decided as much by the measurement as by the code. We derive an operationalization checklist for post-merge studies.
Source:https://arxiv.org/abs/2611.21784PreviousA Survey of Modeling Languages for Automotive Digital TwinsNextCan Coding Agents Write and Transpile Unit Tests
Related posts

‘It’s a risk’: How Rockstar built the most detailed (and romantic) GTA yet
0The Verge
China dominates battery graphite, but this startup could break the monopoly
0TechCrunch
AI visibility: mentions, recommendations and citations need separate checks
0dev.to
Background AI Makes Inference a Scheduling Problem
0dev.to

DeepSeek vs Claude vs Kimi vs Codex vs Gemini: A 70-Call AI Ethics Test
0dev.to
Fluent English is not evidence of ASL understanding: lessons from building a video translator
0dev.to