AI & ML
Can LLMs Write Backdoors? An Empirical Security Study of Weight-Space Programming
arXiv cs.CR0 views
arXiv:2611.17395v1 Announce Type: new Abstract: Recent advances in weight-space programming (WSP) have demonstrated that large language models (LLMs) can directly inspect and modify neural network weights without conventional gradient-based training. While this capability opens new possibilities for model improvement, it also raises an important security question: Can LLMs exploit weight-space programming to inject backdoors into neural networks? To answer this question, we present an empirical security study of LLM-driven weight-space programming for backdoor injection. Specifically, we investigate whether general-purpose LLMs can directly modify the weights of target models according to natural-language instructions, without accessing task-specific data, performing forward passes, or computing gradients. We evaluate five LLMs as weight editors across three target architectures: an MLP on MNIST, ResNet-18 on CIFAR-10, and Qwen2.5-0.5B-Instruct on AG News. Our experiments reveal three key findings. First, LLM-driven weight editing can successfully inject backdoors across different architectures while largely preserving clean performance. Second, successful attacks can require remarkably few parameter modifications, including only six parameters for the MLP and 0.02% of the weights for a fine-tuned Qwen model. Third, editing capabilities vary substantially across LLMs and target architectures: although all five LLMs successfully backdoor the MLP, only GPT-6 Astra succeeds across all three targets. These findings highlight an emerging security risk of weight-space programming and underscore the importance of safeguarding LLM-driven weight editing.
Source:https://arxiv.org/abs/2611.17395PreviousEvolutionary-based Transfer Learning for Physical Attack on End-to-End Autoencoder CommunicationsNextI2C-GRPO: Insight Importance Checklist for Reinforcement Learning in a Multi-Agent DDoS Alert Triage Workflow
Related posts
In which John becomes a Machine Learning researcher
0Hacker News New
Can We Fix AI?
0Hacker News New
I'm Glad That Literary Awards Are Going to AI
0Hacker News New
Show HN: TalentRadar – 3D geospatial tech compensation and role radar
0Hacker News Show
Show HN: Brief2mp4 – ClaudeCode/Codex skill that checks videos against the brief
0Hacker News Show
Show HN: Figma Like AI Landing Page Builder
0Hacker News Show