AI & ML
How Reward Gain Imbalance Shapes Ranking: Rethinking Rubric-Based Reward Aggregation
arXiv stat.ML0 views
arXiv:2611.24005v1 Announce Type: new Abstract: Rubric-based reinforcement learning constructs rewards for aligning large language models with human preferences, typically by equally aggregating continuous scores that a judge assigns to individual criteria. However, scoring heterogeneity across criteria can distort candidate rankings. Existing work has focused on improving judges, learning criterion weights, or refining aggregation, whereas the interaction between scoring heterogeneity and ranking remains insufficiently understood. We formalize this heterogeneity as \emph{reward gain imbalance}: each criterion enters the aggregated reward with its own reward gain, the gap between its expected scores on satisfied and unsatisfied responses. Counterintuitively, our experiments show that removing this imbalance, even with ground-truth labels, lowers ranking accuracy. To explain this, we derive an exact decomposition of the reward difference between any two candidates into a base satisfaction signal, a reward gain imbalance term, and a residual term, showing that reward gain imbalance can attenuate or amplify the base signal, and finding that misrankings usually concentrate on pairs where it attenuates. Guided by this analysis, we propose the adaptive \textbf{D}ual-\textbf{C}enter \textbf{C}lipping aggregation (DCC), which bounds each criterion's contribution to the reward difference instead of simply removing the imbalance. Using only the scores of the current candidate group, it determines a clipping interval that imposes a common upper bound on per-criterion score differences between candidates, limiting the excessive influence of extreme scores while preserving continuous judgment information....
Source:https://arxiv.org/abs/2611.24005PreviousReliFlow: Reliability-Calibrated Residual Transport for Combinatorial Single-Cell Perturbation PredictionNextNOTREKS: Marginal Independence in Score-based Structure Learning
Related posts
Prompt, Workflow, Agent or Agentic System: Which One Does Your Process Need?
0dev.to
Running Claude Code and Codex from your phone: the official options, and the self-hosted app I built
0dev.to
Unlocking Enhanced Visuals: Dlss5 Without Extra Load --An unofficial side-story version
0dev.to
Fiona the Hippo Died After Eating a Hat. But How Does a Hat Kill a Hippo?
0Wired

Why ChatGPT Cites Review Sites Instead of Your Company Website
0dev.to
Basalt walkthrough is up: why 776 tok/s needs the whole card to itself
0dev.to