AI & ML
Reliability is a user-specific property of large language models
arXiv cs.SE0 views
arXiv:2611.23732v1 Announce Type: new Abstract: Public benchmarks support comparisons of large language model (LLM) capabilities, yet users can experience the same highly ranked model differently. Drawing on traditional software reliability, we argue that the reliability of an LLM service should be assessed for specified users and conditions of use. This requires more than a capability score: evaluation evidence must support a claim about failure during future operation. We distinguish testing to discover faults from testing to assess reliability and identify ten modelling considerations behind such claims. We propose a six-workstream research roadmap linking claim specification, dynamic operational profiles, graded outcomes and noisy judgements, local probabilistic robustness, interaction dependence, and statistical inference with reassessment. Local probabilistic robustness addresses the behavioural variation hidden when different concrete inputs are treated as the same task. Published findings and an illustrative customer-support example motivate the roadmap; its integrated validation remains an open research task. Reliability claims complement, but do not replace, separate safety and security arguments. Capability scores should therefore be complemented by versioned operational claims that state whose requirements are covered, under which conditions, and with what uncertainty.
Source:https://arxiv.org/abs/2611.23732PreviousCratesBump: A Version-Bump Dataset for the Rust EcosystemNextRe-Testing LLM-Based C-to-Rust Translations Against Their Original C
Related posts
Ruby2ßhell: Pre-authentication RCE on stock Rails
0Hacker News
Generalizing self-normalized importance sampling with couplings
0arXiv stat.CO
Bayesian Computation of Copula-based Multivariate Models with an Application to Doping Detection
0arXiv stat.CO
WAIVES: Weak-form Asymptotically-unbiased Instrumental-Variable Estimation for SDEs
0arXiv stat.CO
Treat Your Honeypot Analysis Like Code: Git, Pull Requests and a Human Gate for AI [Guest Diary], (Wed, Nov 18th)
0SANS ISC
ISC Stormcast For Thursday, November 19th, 2026 https://isc.sans.edu/podcastdetail/10186, (Thu, Nov 19th)
0SANS ISC