AI & ML
Improved Leverage Score Sampling for Constrained Active Linear Regression
arXiv stat.ML0 views
arXiv:2611.23801v1 Announce Type: new Abstract: In this paper we study active learning for constrained linear regression. Given a design matrix $A\in\mathbb{R}^{n\times d}$, a constraint set $C\subseteq\mathbb{R}^d$, and query access to $b\in\mathbb{R}^n$, we seek an approximate solution to $\min_{x\in C}\|Ax-b\|_2^2$ using as few queries as possible. This problem arises in many scientific and engineering applications, and randomized leverage score sampling provides a powerful method to solve it. However, existing leverage score methods ignore the constraint or encode it through ridge regularization, suffering unfavorable query complexity for anisotropic constraint sets. We propose ellipsoid-regularized leverage scores, where we approximate the constraint set by an outer ellipsoid and use its shape matrix to bias sampling toward directions that matter under the constraint geometry. We prove that the query complexity is governed by an anisotropic effective dimension $d_{\text{eff}}(C)\le d$. With appropriate parameters, $O(d_{\text{eff}}(C)\log d/\varepsilon^2)$ queries suffice to return a point in $C$ with objective value at most $(1+\varepsilon)$ times the optimum. This bound can substantially improve on the standard leverage-score bound, since anisotropic constraints can make $d_{\text{eff}}(C)$ much smaller than $d$. For convex constraints, we improve the dependence on $\varepsilon$ from $1/\varepsilon^2$ to $1/\varepsilon$. We also prove two lower bounds: first, the $d/\varepsilon^2$ dependence is unavoidable without convexity assumptions on $C$, and second, in some regimes, the effective dimension dependence $d_{\mathrm{eff}}(C)/\varepsilon$ is necessary for ellipsoidal constraints. Experiments on synthetic regression problems and a parametric heat conduction problem show lower regression error than standard baselines for similar query budget.
Source:https://arxiv.org/abs/2611.23801PreviousA Free Lunch: Gradient Averaging Achieves Optimal Statistical Efficiency in Newton SketchNextOptimal Deep Nonparametric Regression with Uniformly Bounded Lipschitz Constants
Related posts
VOS: Variate Ordering Strategies for Skyline Query Optimization
0arXiv cs.DB
MetaSieve: Faster Relational Deep Learning through SQL-Based Metapath Selection
0arXiv cs.DB
Exploring Agentic Visual Analytics: A Co-Evolutionary Framework of Roles and Workflows
0arXiv cs.DB
ProvCards: Artifact-Oriented Provenance for Heterogeneous Scientific Workflows
0arXiv cs.DB
Towards Online Adaptive LLM-based Semantic Filtering
0arXiv cs.DB
Cascadeability and the Realization Gap: An Empirical Study of LLM-Powered Semantic Filtering
0arXiv cs.DB