Backend
Training Neural Networks to be Locally Adaptive
arXiv stat.ML0 views
arXiv:2611.22198v1 Announce Type: cross Abstract: Deep neural networks fitted by global empirical risk minimization commit to a single model for the entire covariate space. In a regression problem, if the regression function $f_0$ is inhomogeneous --- varying rapidly in one region, while being nearly flat in another --- no single model can be appropriate everywhere, and the fit degrades where the function varies most rapidly. Here we study an alternative in which, instead of fitting the network globally, we fit it locally. For each query point $x$, a \emph{local neural network} is trained on the $k$ nearest-neighbours of $x$ alone, making the training sample and the fitted function query-dependent. We analyze the performance of the resulting estimator through a localization approach, establishing that the pointwise risk of the estimator with an appropriate choice of $k$ attains the rate $n^{-2\alpha/(2\alpha+d^{*})}$, up to logarithmic factors, when $f_0$ is locally $\alpha$-H\"older with $\alpha\in(0,1]$ and the covariates are supported on a $d^{*}$-dimensional manifold embedded in $\mathbb{R}^{d}$. A matching Le Cam lower bound shows that this rate is pointwise minimax optimal. However, the local rate-optimality requires $k$ to be calibrated to the smoothness of $f_0$, which is unknown in practice. We therefore provide a locally adaptive choice of $k$ that makes the neural network adapt to the local H\"older exponent of $f_0$ and to the intrinsic dimension simultaneously, both of which are potentially varying over the covariate domain. An extensive simulation study demonstrates the dominance of the locally adaptive neural network over a globally trained network and several state-of-the-art methods.
Source:https://arxiv.org/abs/2611.22198PreviousCentroid Spectral Coupling: Implicit Alignment and Representation Diversity in Self-Supervised LearningNextSOTA or Luck? The Winner's Curse in LLM Leaderboards
Related posts
Autoscaling Adds Waiters, Not Ovens. Here's What Actually Breaks at 10x Load
0dev.to
[백엔드 모의면접] 중복 결제는 어떻게 막을까?
0dev.to
Server-Side Pagination Looked Easy. It Took Me Longer Than Any React Concept Yet.
0dev.to
Before You Create an AWS Account, Spend 45 Minutes Learning These Security Tools
0dev.to
Claude “Invalid Signature” in a Thinking Block: How to Debug It Safely
0dev.to
How to catch an expiring or broken SSL certificate before your visitors do
0dev.to