AI & ML
Can Programs Replace LLM Judges? Executable Program Judges for Scalable and Efficient Relevance Labeling
arXiv cs.IR0 views
arXiv:2611.24849v1 Announce Type: new Abstract: Large language model (LLM) judges have become a common way to create large-scale relevance judgments for retrieval evaluation. However, each judgment requires a potentially expensive call to a frontier model, and the judgments are hard to interpret and may not be reproducible. In this paper, we ask whether an LLM can write a readable program for each topic that replaces its per-document judgments. To address these issues, we propose ExecJudges, a method for generating fast, inspectable and scalable judges for automatic relevance labeling. Specifically, ExecJudges uses a frontier LLM to turn assessment guidelines into executable programs, which rely only on lexical matching, standard NLP libraries like parsers, and pool statistics. An ExecJudges-L variant also uses examples of human labels if available. We evaluate ExecJudges on seven TREC collections by comparing the produced labels with trained NIST assessors, frontier LLM judges, and recently proposed per-topic classifiers. Our results show that ExecJudges agrees with NIST assessors at the level of the Claude Sonnet 5.5 LLM judges on TREC DL 2019-2021. With a small number of human labels per topic, the ExecJudges-L variant agrees with NIST more than per-topic classifiers with the same labels on all seven collections, and either exceeds the agreement of LLM judges provided the same labels or is within about 0.05 of it in Krippendorff's alpha. The labels rank the official TREC DL runs with Kendall's tau of 0.87 to 0.94, and writing the programs incurs a one-time cost of $0.17 to $0.49 per topic, with no additional costs for grading. Our results suggest that ExecJudges offers a reproducible, inspectable, efficient and cost-effective alternative to LLM judges for labeling deep result pools for system evaluation or ranking improvements.
Source:https://arxiv.org/abs/2611.24849PreviousX-split Argus: Learning Reusable History Tokens for Efficient CTR PredictionNextPrioritize, Don't Decide: What We Learned Auditing a Deployed Human-in-the-Loop CV Sorter
Related posts
Fuzzy name matching for sanctions screening: normalisation, transliteration and Jaro-Winkler
0dev.to
The Claude Architect Professional exam changes on 7 December - Here is what's different
0dev.to
SEO Isn't Enough Anymore. Meet GEO.
0dev.to

Prompt Engineering vs. Context Engineering vs. Harness Engineering: What’s the Difference?
0dev.to
P_SAPEA Exam Practice Questions: The Comparisons That Decide SAP Enterprise Architect Scenarios
0dev.to
Olala 7A1B says 8 GB, but the BF16 file is 13.59 GB: what a 1B-active hybrid MoE needs to run
0dev.to