Backend
Training Neural Networks to be Locally Adaptive
arXiv stat.ML0 views
arXiv:2611.22198v1 Announce Type: cross Abstract: Deep neural networks fitted by global empirical risk minimization commit to a single model for the entire covariate space. In a regression problem, if the regression function $f_0$ is inhomogeneous --- varying rapidly in one region, while being nearly flat in another --- no single model can be appropriate everywhere, and the fit degrades where the function varies most rapidly. Here we study an alternative in which, instead of fitting the network globally, we fit it locally. For each query point $x$, a \emph{local neural network} is trained on the $k$ nearest-neighbours of $x$ alone, making the training sample and the fitted function query-dependent. We analyze the performance of the resulting estimator through a localization approach, establishing that the pointwise risk of the estimator with an appropriate choice of $k$ attains the rate $n^{-2\alpha/(2\alpha+d^{*})}$, up to logarithmic factors, when $f_0$ is locally $\alpha$-H\"older with $\alpha\in(0,1]$ and the covariates are supported on a $d^{*}$-dimensional manifold embedded in $\mathbb{R}^{d}$. A matching Le Cam lower bound shows that this rate is pointwise minimax optimal. However, the local rate-optimality requires $k$ to be calibrated to the smoothness of $f_0$, which is unknown in practice. We therefore provide a locally adaptive choice of $k$ that makes the neural network adapt to the local H\"older exponent of $f_0$ and to the intrinsic dimension simultaneously, both of which are potentially varying over the covariate domain. An extensive simulation study demonstrates the dominance of the locally adaptive neural network over a globally trained network and several state-of-the-art methods.
Source:https://arxiv.org/abs/2611.22198PreviousCentroid Spectral Coupling: Implicit Alignment and Representation Diversity in Self-Supervised LearningNextSOTA or Luck? The Winner's Curse in LLM Leaderboards
Related posts
Building a Raspberry Pi AI Marine Computer: Lessons in Security, Open Source, and Trust
0dev.to

Google Rebuilt a Monolithic AI Agent Into 5 Sub-Agents. Here Is the Same Refactor in 57 Lines of Python.
0dev.to
App Backup Restore Costs for Authenticated Media Reports in Private Object Storage
0dev.to

You Got Rejected. Don’t Rewrite Your Entire Interview Prep.
0dev.to
Snov.io vs Hunter.io Role-Based Email Detection: False-Negative Rates, Sender Score Impact, and the Regex Pre-Filter Both Tools Need
0dev.to
Why My AI Writer Waits 10 Posts Before Learning
0dev.to