AI & ML
What Does a Harness Buy? Tokens, Mostly
arXiv cs.SE0 views
arXiv:2610.04433v2 Announce Type: replace-cross Abstract: A coding agent is a language model inside a harness: the system prompt, the tools, and the context management that let a chat model work in a repository. Most of the engineering on these agents goes into the harness, and vendors and leaderboards credit it with score gains. Earlier studies have fixed the model and swapped the harness, but have not measured whether the swap changes the pass rate more than running the same harness again. We run seven models through three production harnesses, Claude Code, mini-SWE-agent, and OpenCode, on SWE-bench Verified and two benchmarks of other kinds, and rerun them unchanged. No harness leads consistently across models. Where the data can bound the difference between two harnesses, the bound is 2 to 5 points, and a harness swap changes about as many task outcomes as a rerun. The exceptions are harness effects in particular pairings of model and harness, whose direction changes with the model: OpenCode trails by 8 to 9 points on one model and leads by 14 on another. Cost differences between harnesses also depend on the model, from under 10% to 3 to 4 times. Which harness is better therefore has no answer apart from the model, and a harness difference should be reported next to a rerun.
Source:https://arxiv.org/abs/2610.04433PreviousPrior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code OptimizationNextPreventing Duplicate Orders in PHP and MySQL: Transactions, Idempotency and Safe Retry Handling
Related posts
How do you perform AI code review?
0dev.to Git
gpt-5 and o3 shut down December 11: a find-and-replace would rewrite 274 table rows
0dev.to TypeScript
Exporting App Store reviews from every country in Node.js: 6 things the RSS feed doesn't tell you
0dev.to JavaScript
Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization
0arXiv cs.SE
Live Noise Fingerprints for Quantum Program Analysis
0arXiv cs.SE
Biomedical Knowledge Composition: A Software Engineering Perspective
0arXiv cs.SE