AI & ML
SPARK: Semantic-preserving Prompt Mutation for Code Generation using Consensus-based Verification Framework
arXiv cs.SE0 views
arXiv:2611.13401v1 Announce Type: new Abstract: Automated code generation with large language models (LLMs) has improved in recent years. With the lack of test cases in real-world usages, generated code is not automatically verifiable and often requires manual validation. Previous studies have leveraged consensus verification, where code solutions and corresponding test cases are generated and evaluated based on functional consistency. Building on this, SPARK incorporates semantic-preserving mutations to problem descriptions such as paraphrasing, grammatical adjustments, function renaming, and other alterations to enhance the robustness and correctness of LLM-generated code. For each mutated description, SPARK generates a set of code solutions and test cases. These candidate solutions are evaluated using consensus-based verification, where the solutions' quality is measured by how consistently they pass generated tests. We assess the effectiveness of SPARK on three benchmarks (LiveCodeBench, BigCodeBench, and DLBench) across three LLMs (o4-mini, DeepSeek-R1-Basic, and Gemini-2.5-Pro). Our evaluation demonstrates that SPARK outperforms state-of-the-art consensus-based verification approaches by up to 4.1%. Our generalization experiment highlights the practical potential of SPARK as an effective strategy for enhancing the reliability and correctness of LLM-based code generation systems in scenarios where traditional validation resources are scarce, while our qualitative analysis shows insight into how SPARK's mutation helps LLMs generate better code solutions.
Source:https://arxiv.org/abs/2611.13401PreviousThe AI Era's Lines of Code? A Case Study on What an Engineer's Token Spend MeasuresNextAn Empirical Study on Improving LLM-Based Issue Classification through Domain-Informed Analysis
Related posts
I wanted one screen that said whether a kids' app was worth installing
0dev.to

Before I Could Build GPT, I Had to Teach a Machine to Learn
0dev.to
Cogent and OpenAI build cyber defense factories for enterprises
0OpenAI Blog
ChatGPT Plus for veterans and servicemembers
0OpenAI Blog
Hitting 200 Strokes: What the Signature Maker Does at Its Stroke Limit
0dev.to
A Lightweight, Open-Source Network Digital Twin Blueprint for IoT-Edge Environments
0arXiv cs.NI