AI & ML
SeVerus: Context-Aware, Gamified Benchmarking for Rust Verification
arXiv cs.SE0 views
arXiv:2611.07194v1 Announce Type: new Abstract: Coding agents achieve high success rates on existing benchmarks for Verus, a verifier for Rust programs. These benchmarks typically provide the auxiliary lemmas needed to complete a proof. In practice, however, such lemmas must be discovered and proved. To capture this challenge, we present SEVERUS, a context-aware technique to generate Verus benchmarks that organizes verification tasks into quests, each divided into levelsin a gamified manner. Using the original proof dependencies in verified projects, SEVERUS constructs one quest per verification target and progressively removes proof context to create individual levels. These levels range from the easiest, requiring completing only a single missing lemma, to the hardest, requiring proving the target without any supporting lemmas. An independent checker rejects changes to executable code, weakened specifications, and unsound proof constructs. A noisy binary search estimates the highest level that an agent can solve reliably. Applied to six open-source Verus projects, SEVERUS generates 819 quests comprising 10,095 levels. Experiments with Codex using GPT-6 Sol and GPT-6 Luna, and with Claude Code using Sonnet 4.6, show that success decreases and cost increases as proof context is withdrawn. The strongest agent solves the final level of 52 of 60 sampled quests, yet can only reliably solve 35% of their levels, with failures concentrated in longer quests. Successful GPT-6 Sol proofs rarely reproduce the withheld lemmas, instead, using alternative proof decompositions. By systematically varying proof context, SEVERUS provides a foundation for constructing verification benchmarks that can continue to challenge increasingly capable coding agents.
Source:https://arxiv.org/abs/2611.07194PreviousMulti-step SWE: Can Coding Agents Keep Building on Their Own Code Over Long Horizons?NextHow Should Agents See Code? Rethinking Code Representations for Agentic Software Engineering
Related posts

AI teddy bears spark fresh safety fears after report discovers malfunctioning toys
0TechRadar

ChatGPT can now create interactive tools instead of just answering questions
0TechRadar

70% of Brits say please or thank you when speaking to AI — and some even apologize
0TechRadar
From Reference Image to 3D Asset: A Practical Workflow for Developers
0dev.to
Cut one conflicting rule before you trim anything else
0dev.to
Eine App darf deinen Kalender lesen und deine Mails zusammenfassen – und trotzde
0dev.to