AI & ML
Cascadeability and the Realization Gap: An Empirical Study of LLM-Powered Semantic Filtering
arXiv cs.DB0 views
arXiv:2611.23898v1 Announce Type: new Abstract: Semantic filters enable natural language predicates over unstructured documents, but evaluating every document with a large language model (LLM) oracle is expensive. Proxy cascades reduce oracle invocations using cheaper surrogates, but oracle reduction alone cannot distinguish proxy limitations from calibration bottlenecks and establish practical efficiency. We present a unified benchmark of four representative systems, which formulate query processing as a three-stage workflow comprising proxy construction, threshold calibration, and cascade inference, using a two-threshold acceptance rule. We introduce cascadeability: the maximum empirical oracle reduction attainable from fixed proxy scores at a specified accuracy target, computed retrospectively using complete oracle labels. We investigate 8 research questions, spanning proxy quality, end-to-end performance, workload sensitivities, and component ablations, and reveal our key findings accordingly. Comparing cascadeability with achieved oracle reduction yields a realization gap that, interpreted alongside accuracy, exposes distinct proxy and calibration bottlenecks. The benchmark and diagnostics support evaluation beyond oracle reduction and motivate joint proxy-calibration design, class-sensitive quality assessment, and end-to-end cost optimization. We release a dataset-query benchmark, containing 10 real-world workloads across various domains and difficulties, with annotations of two LLM families.
Source:https://arxiv.org/abs/2611.23898PreviousLogical Design Tuning via Workload-aware E/R-to-Relational MappingNextTowards Online Adaptive LLM-based Semantic Filtering
Related posts
Senate passes BAAJA permitting reform bill backed by AI, energy sectors
0Hacker News New
Animated Steller's Jay, a beautiful AI benchmark
0Hacker News New
The Discipline of Deliberate Building
0Hacker News New
Autonomous Trail Racing
0Hacker News New
Foundering of the sailing yacht Bayesian with the loss of seven lives
0Hacker News New
Prompt, Workflow, Agent or Agentic System: Which One Does Your Process Need?
0dev.to