Thirty‑two‑million‑parameter per‑option encoders now outperform larger joint models on closed‑set decision benchmarks [1]. The surprise is that a tiny per‑option reader beats a monolithic scorer even when the latter has an order of magnitude more parameters.
Until recently, practitioners have leaned on large joint encoders or full‑scale language models to rank options in tasks such as tool selection, assuming that scaling up the encoder would automatically improve accuracy and stability. Joint scoring treats all candidates as a single input sequence, while per‑option scoring evaluates each candidate independently with a shared query encoder.
Per‑option scoring beats joint scoring by 5.8 points of chance‑corrected skill on a held‑out sample of a 37‑benchmark decision suite when matched at optimizer steps [1]. The same study reports a 2.8‑point edge at roughly equal training compute, and two external benchmarks echo the trend though without statistical significance on unseen tasks.
The smallest encoder answers a median request in 14 ms on 16 processor threads, versus 728 ms for the 435‑M DeBERTa‑v3 zero‑shot classifier [1]. This more than fifty‑fold reduction in latency makes per‑option models viable for real‑time pipelines where response time is critical.
Even the mid‑size 150‑M per‑option model exceeds the large classifier by 5.9 accuracy points on 209 overlap‑audited System1‑Arena tasks and stays on par across 145 unseen tasks [1]. The result shows that scaling beyond a few tens of millions of parameters yields diminishing returns for closed‑set decisions.
Reordering options flips the answers of joint models in up to 23 % of cases, whereas per‑option scoring is completely invariant to option permutation [1]. This robustness gap highlights an intrinsic brittleness of joint encoders that can jeopardize downstream reliability.
The advantage hinges on a reranker‑style initialization whose query‑candidate pair input matches the per‑option reading scheme, and it comes at the cost of 5–13× more encoder passes [1]. Consequently, mean latency can be more than four times higher even though median latency stays under twice that of the joint baseline.
Benchmarks that currently rank decision encoders should be rerun with per‑option baselines, and production teams can replace multi‑hundred‑million joint models with sub‑20 M parameter readers without sacrificing effectiveness. Will the next wave of decision‑making services abandon monolithic encoders altogether?
