AI & ML
Probabilistic Relevance Decisions for Semantic Retrieval
arXiv cs.IR0 views
arXiv:2611.08014v1 Announce Type: new Abstract: Dense similarity supplies reusable corpus representations, while probabilistic relevance decisions offer query-dependent document scoring. We compare dense-only BGE-M3 with TypeSafe Jev Noul and OpenAI Decisions Predicate scores on the official BEIR SciFact test benchmark. The study evaluates 300 official test queries using identical positive-enriched 100-document pools and full-corpus dense retrieval followed by reranking at candidate depths of 50 and 100. Unlike the controlled pools, neither dense prefix receives positive injection. Jev has the highest observed Recall@5 and nDCG@10 under the evaluated conditions. At depth 50, full-corpus nDCG@10 is 0.6413 for BGE-M3, 0.7545 with OpenAI and 0.7805 with Jev. Extending depth from 50 to 100 increases known-positive candidate recall but produces small, uncertain final-effectiveness changes. All four paired-bootstrap depth intervals include zero. Unique-condition API scoring cost approximately doubles, while compatible cached decisions reduce incremental expenditure. OpenAI returns many exact-zero scores, and secondary exact-tie randomization reveals ranking sensitivity without replacing deterministic primary results. Top-100 latency mixes cached and new execution. Historical hybrid latency is a saved-component sum under different execution environments and timer boundaries. These exploratory, unadjusted analyses support relevance decisions as a complement to dense retrieval within this protocol, without establishing general superiority, probability calibration or equivalence between depths.
Source:https://arxiv.org/abs/2611.08014PreviousMEMOIR: A Training-Free Memory Lifecycle for Long-Horizon LLM AgentsNextDomain-specific information retrieval
Related posts

Make your first map video in Studio
0dev.to
How octopuses squeeze through too small gaps
0Hacker News
From the front row to the hot seat — what the midterms mean for Big Tech
0TechCrunch
Run Qwen3.8-Flash-Next Locally with Strata: 64 GB Quickstart
0dev.to
I ran 33 LLM models behind one endpoint for a month. Here is what broke.
0dev.to
Retries taught me that fixed-interval backoff is just synchronized failure
0dev.to