AI & ML
What Moves the Ceiling on Automated Audit: from General Code to Scientific Code, Results and Data
arXiv cs.SE0 views
arXiv:2611.23250v1 Announce Type: new Abstract: Models increasingly check what other models write. We study one shipped arrangement, in which an auditor from a different vendor reads the work and deterministic checks verify what code can verify, and measure its limits against ground truth no model supplies. On general-purpose code with hidden tests, eight readings of the auditor reach 30.0% recall [20.0, 40.7] at 16.0% [10.1, 22.3] flags on correct code, with diminishing returns and no plateau. The reading model (confounded with its sampling), one rulebook sentence and the decision rule each move that figure, and much of what is missed is, on a post hoc rating, behaviour the specification never determined. In three preregistered science studies, checking that reported numbers trace to their sources flagged all 49 inconsistent reports and none of 25 consistent fabrications, of which the model auditor flagged 23 (21 with a sound finding, post hoc); on data, a documentation-derived validator and the auditor flagged overlapping but different faulty items, together 112 of 140 at a 2.9% [0.7, 5.7] flag rate on clean data; and on scientific code, reframing the task around one step cut flags on correct code from 68.7% to 36.0%. Where both tiers ran, the deterministic one flagged no clean item and could not see consistent error; the model saw some and erred itself. Studies making model calls were preregistered, the two unregistered analyses are named, and every quoted report passed independent cross-vendor review.
Source:https://arxiv.org/abs/2611.23250PreviousThe Obsolescence of High-Level Code: Agentic Generation from Semantic ArtifactsNextWho Changed the AI Agent's Rules? Measuring How AI Coding Agents Escalate Authority in CI/CD Pipelines
Related posts
![Quadruped Robot Software — Ch.12: Quadruped Case Studies [KR]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fawosw4u7jitr9h27f7c5.png)
Quadruped Robot Software — Ch.12: Quadruped Case Studies [KR]
0dev.to
How Python Is Used in Artificial Intelligence in 2026: Applications, Benefits and Career Opportunities
0dev.to

What Happens in Those Few Seconds Between Pressing the Power Button and Seeing the Windows Login Screen?
0dev.to
Getting cited by ChatGPT and Claude: what the research supports, and a free audit tool
0dev.to
Why Photoshot keeps the original color when you edit in black and white
0dev.to
ISTQB CTFL in 2027: is it still worth it? What changes, what does not
0dev.to