AI & ML
Veritas: A Verifier Generation Harness for Agent Tasks
arXiv cs.CR0 views
arXiv:2611.13336v1 Announce Type: cross Abstract: The effectiveness of a self-improving system relies heavily on the quality of its verifier, which judges the outcome of each improvement round. However, the verifiers of most tasks are written by hand, which is expensive and does not scale to tasks that come without one. We propose Veritas, a harness that generates verifiers automatically. Unlike general-purpose agent harnesses, Veritas adds three mechanisms for verifier generation: an evidence header stating what the verifier will observe, a library of domain rules that tasks leave unstated, and authoring principles, partly refined by a self-improvement loop. We evaluate on six benchmarks: four scored by execution, whose official verifiers we first certify, and two that compare pairs of responses. With the model fixed, the verifiers Veritas generates agree better with the official verdicts than those of seven other harnesses on all four execution-scored benchmarks. On the two pairwise benchmarks they are at or near the best. They also agree better than those of the vendor's own harness for two of three other model families. Its mechanisms also improve 9 of 12 harness-benchmark pairs scored by execution across three harnesses.
Source:https://arxiv.org/abs/2611.13336PreviousHow wide must an eavesdropper be? Leakage of a toric-code logical bit to a strip and the Nishimori pointNextNo finite rulebook can certify every safe program: an impossibility theorem for gatekeeping AI agents
Related posts
FDR-Controlled Variable Selection for Generalized Linear Models and Cox Regression with Virtual Dummies
0arXiv eess.SP
Terrain Perception for Agricultural UAVs in Complex Farmland via Rotating mmWave Radar
0arXiv eess.SP
FUTON: Fourier Tensor Networks for Implicit Neural Representations
0arXiv eess.SP
Writing Bug Reports for Software Repair Agents: What Information Matters Most?
0arXiv cs.SE
SACS: A Code Smell Dataset using Semi-automatic Generation Approach
0arXiv cs.SE
Resolution of Vinzant's Conjecture on Phase Retrieval Injectivity
0arXiv math.IT