AI & ML
Spectrum-Adaptive Generalization Bounds for Trained Deep Transformers
arXiv stat.ML0 views
arXiv:2605.07297v2 Announce Type: replace Abstract: Understanding why trained Transformers generalize well is a fundamental problem in modern machine learning theory, and complexity-based generalization bounds provide a principled way to study this question. While existing norm-based bounds for Transformers remove the explicit polynomial dependence on the hidden dimension, they typically impose fixed norm constraints specified a priori and can exhibit unfavorable exponential dependence on depth. In this paper, we derive spectrum-adaptive post hoc generalization bounds for multi-layer Transformers. Under layerwise spectral norm control, the bounds are expressed in terms of layerwise Schatten quantities of the query-key, value, and feedforward weight matrices. Since the Schatten indices need not be fixed a priori and can instead be selected after training, separately for each matrix type and layer, the bounds adaptively trade off spectral complexity against the dimension- and depth-dependent factors according to the learned singular-value profiles. Empirical comparisons of BERT-adapted proxies for the leading complexity factors suggest that the proxies induced by our bounds grow more slowly with depth and hidden dimension than the corresponding norm-based proxies. Overall, our results provide a complexity-based perspective on how the spectral structure of trained Transformers is reflected in generalization analyses.
Source:https://arxiv.org/abs/2605.07297PreviousSparse Network Inference under Imperfect Detection and its Application to Ecological NetworksNextMeasuring and Decomposing Mode Separation via the Canonical Diffusion
Related posts
A Straight Line Is Not a Camera Pose: Auditing Short-Clip Pose Benchmarks
0arXiv cs.RO
Mechanisms of Misgeneralization in Physical Sequence Modeling
0arXiv cs.RO
Behavior Graphs: Unsupervised Graph Structures for Interpreting Robot Deployment Data
0arXiv cs.RO
RoboFQE: Rollout-Free Evaluation of Visual Robot Policies from Offline Trajectories
0arXiv cs.RO
Improving Imitation Learning Efficiency for Manipulation through Geometric Prior Pretraining
0arXiv cs.RO
MATE: Asynchronous Multi-Rate Diffusion Policy Composition
0arXiv cs.RO