AI & ML
DiagDBench: Evaluating Agentic LLMs on Interactive Database Diagnosis and Anomaly Remediation
arXiv cs.DB0 views
arXiv:2611.07339v1 Announce Type: new Abstract: Database diagnosis and anomaly remediation are essential to reliable database management system (DBMS) operation. Despite advances in large language model (LLM)-based database agents, constructing live DBMS environments that reproduce diverse anomalies at scale remains challenging, limiting the availability of executable samples for database agent training and evaluation. To this end, we introduce DiagDBench, a large-scale benchmark for training and evaluating database agents on interactive diagnosis, anomaly remediation, and optional workload performance optimization across diverse, realistic anomaly scenarios. Each sample pairs a user request, comprising an issue description and an anomaly-verified workload under which the issue was observed, with a self-contained, executable DBMS environment that supports real-time monitoring, interactive diagnosis, remediation, and workload execution. To construct these samples at scale, we propose an LLM-based framework with two complementary pipelines: Manual-Guided Anomaly Synthesis, which sequentially synthesizes user requests, database modifications, and anomaly-injection scripts grounded in DBA manuals and official DBMS documentation; and Incident-Guided Anomaly Reproduction, which reconstructs anomaly samples end to end from real-world enterprise online diagnosis incidents. Built using this framework with sample filtering, expert review, and refinement, DiagDBench comprises 6,940 synthesized training samples and a held-out test set of 454 expert-annotated samples, spanning three hardware configurations, two DBMSs, and three database schemas. Extensive evaluations of database agents powered by multiple LLMs demonstrate that even the best-performing agent, ReAct with GPT-6 Astra, scores only 56.24, whereas most agents score below 45, underscoring the difficulty of DiagDBench.
Source:https://arxiv.org/abs/2611.07339PreviousRecursively Testing and Vibe Debugging a Google AI Studio Generated Vibe-Coded Database Software ApplicationNextSelective Retrieval Routing for Multi-Violation Knowledge Graph Repair
Related posts
cBridge: Do You Need Gas on Both Chains?
0dev.to
Honor Puts a 4-DOF Titanium Gimbal Inside a Phone at 7,999 Yuan
0dev.to

Protesters break into Microsoft data center site in Amsterdam
0The Verge

After printing 40,000 whistles, this is the 3D printer I’d buy
0The Verge

A New Community For In-House UX People
0Smashing Magazine
A flaw in ChatGPT, a lesson for our front-ends
0dev.to