AI & ML
Cost-Efficient LLM Evaluation: A Framework for Narrow Confidence Intervals Under Fixed Budgets
arXiv math.IT0 views
arXiv:2609.29140v2 Announce Type: replace-cross Abstract: Large language models (LLMs) play an increasingly important role in both academic research and industry. Reliable evaluation helps researchers understand model capabilities and assess progress during model development. However, evaluation can be costly, and researchers often need to assess performance within a fixed budget. Because model outputs can vary across runs, this budget must be allocated between testing different tasks and repeating selected tasks. A confidence interval quantifies the uncertainty in the resulting performance estimate. Obtaining narrower intervals at the same confidence level and evaluation cost provides a more precise assessment of model performance, helping researchers draw clearer conclusions from limited observations. We introduce a framework that allocates a fixed evaluation budget across tasks and repeated runs, with or without full task coverage. The framework then uses disagreements between runs on the same task to bound estimation error, allowing narrower confidence intervals at the same confidence level when the sampled repeats show little variability. When repeated runs on each task yield the same outcome, we prove that our framework achieves the fastest theoretically possible rates for narrowing confidence intervals in the worst case as the number of tasks and the budget for repeats grow. We establish matching lower bounds both when every task must be covered and when tasks may be skipped, even for evaluators who use earlier results to decide which run to carry out next. On LiveCodeBench, our framework yielded narrower confidence intervals on average than uniform random sampling in 15 of 16 model configurations, with a median width reduction of 30.6% at the same confidence level and the same budget of 1,100 outputs per configuration.
Source:https://arxiv.org/abs/2609.29140PreviousMinimum Neural Weight Norm = Kolmogorov ComplexityNextThe Illusion of Relief: How GenAI Redistributes Cognitive Workload in Software Organizations
Related posts
![Quadruped Robot Software — Ch.12: Quadruped Case Studies [KR]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fawosw4u7jitr9h27f7c5.png)
Quadruped Robot Software — Ch.12: Quadruped Case Studies [KR]
0dev.to
How Python Is Used in Artificial Intelligence in 2026: Applications, Benefits and Career Opportunities
0dev.to

What Happens in Those Few Seconds Between Pressing the Power Button and Seeing the Windows Login Screen?
0dev.to
Getting cited by ChatGPT and Claude: what the research supports, and a free audit tool
0dev.to
Why Photoshot keeps the original color when you edit in black and white
0dev.to
ISTQB CTFL in 2027: is it still worth it? What changes, what does not
0dev.to