AI & ML
Bias-Reduced and Parallel Heavy-Tailed Stochastic Convex Optimization in $\ell_p$ Geometries
arXiv stat.ML0 views
arXiv:2611.21844v1 Announce Type: cross Abstract: We consider the fundamental problem of parallel stochastic convex optimization (SCO), whose aim is to approximately minimize a convex objective when given the ability to query a moment-bounded stochastic gradient oracle multiple times in parallel per round. Whereas most previous results have focused on the variance-bounded Euclidean ($\ell_2$-norm) setting, meaning the objective is Lipschitz with respect to the $\ell_2$-norm over an $\ell_2$-bounded feasible set in $\mathbb{R}^d$, in this work we study parallel SCO in $\ell_p$ settings for $2 \le p < \infty$ under a heavy-tailed noise model. We begin by introducing a new class of $\ell_p$-optimal-distance-convergence (ODC$_p$) algorithms, drawing on prior stochastic bias-reduced techniques [ACJ+21], which combine with multilevel Monte Carlo methods to efficiently estimate the optimum of uniformly convex objectives. As a consequence of our ODC$_p$ algorithms, we first establish rates for projection-efficient $\ell_p$-Lipschitz optimization, composite convex minimization, and efficient $\ell_p$-proximal point estimation. Combining our ODC$_p$ algorithms with a new $\ell_p$-generalized Gaussian-based stochastic gradient estimator ($p$-GREG), we provide two efficient $\ell_p$ parallel algorithms, EpochDA and $\ell_p$-AC-SA, which achieve optimization error $\epsilon > 0$ with at most $\tilde{O}(d^{\frac{p}{2p + 2}} \epsilon^{-\frac{p}{p + 1}})$ query depth, improving upon the sequential heavy-tailed stochastic query depth of $O(\epsilon^{-p})$ for $\epsilon \leq \tilde{O}(d^{-1/(2p)})$. We also establish dimension-dependent lower bounds of $\Omega\bigl(\min\{\epsilon^{-p},d^{p/(3p-2)}\}\bigr)$ adaptive rounds for nonsmooth convex optimization over a subset of the unit $\ell_p$ ball. Our results improve upon the $\Omega(\min\{\epsilon^{-p},\epsilon^{2/3}d^{1/3}\})$ bounds of [DG19], when adapted to the same feasible set.
Source:https://arxiv.org/abs/2611.21844PreviousBandit Online Submodular Minimisation Using Zeroth-Order OraclesNextAdaptive Memory for Online Reinforcement Learning under Sparse Model Changes
Related posts

I lied to ChatGPT and Gemini and pretended to be an expert
0TechRadar

The overlooked technology that could determine the ROI of enterprise AI
0TechRadar

Microsoft is giving agents their own Cloud PC capabilities to help you with crucial tasks
0TechRadar

The Shure MV360 is an affordable mic for spatial audio capture
0TechRadar

Chief AI Officers are now a common sight in businesses — but new study finds they often don't really have any power
0TechRadar

Nvidia board faction angling for Jensen Huang return as CEO, report claims
0Tom's Hardware