AI & ML
A Free Lunch: Gradient Averaging Achieves Optimal Statistical Efficiency in Newton Sketch
arXiv stat.ML0 views
arXiv:2611.23671v1 Announce Type: new Abstract: We study statistical inference for online sketched Newton methods, in which each Newton system is approximately solved by an inner randomized sketching solver. Existing work reveals a computational--statistical trade-off: the sketching solver (and its accelerated variants) reduces the per-step cost of Newton methods from $O(d^3)$ to $O(d^2)$, but the randomness introduced by sketching also inflates the limiting covariance of both the last and the averaged Newton iterates. In this work, we address this trade-off via gradient averaging. We show that simply replacing the stochastic gradient with a moving average at each step can mitigate, and in some cases eliminate, the trade-off in Newton sketch. Importantly, gradient averaging requires no additional oracle evaluations and leaves the contraction bound of the inner sketching solver unchanged. In particular, we prove that the last sketched Newton iterate with gradient averaging attains a limiting covariance no larger than that without gradient averaging, while the averaged iterate even attains the minimax optimal limiting covariance, matching that of the exact Newton method. To our knowledge, this is the first analysis showing that randomness introduced to improve computational efficiency need not compromise statistical efficiency as long as proper gradient estimates are used. The key analytical challenge is that gradient averaging destroys the crucial martingale difference structure underlying central limit theorem arguments. We overcome this challenge by analyzing the coupled two-timescale recursions governing the iterate and the gradient estimator. Numerical experiments on linear and logistic regression support our theoretical covariance comparisons.
Source:https://arxiv.org/abs/2611.23671PreviousBounded-Order Non-Identifiability in Latent Structure Block Models: The Fiber in Closed FormNextImproved Leverage Score Sampling for Constrained Active Linear Regression
Related posts
Prompt, Workflow, Agent or Agentic System: Which One Does Your Process Need?
0dev.to
Running Claude Code and Codex from your phone: the official options, and the self-hosted app I built
0dev.to
Unlocking Enhanced Visuals: Dlss5 Without Extra Load --An unofficial side-story version
0dev.to
Fiona the Hippo Died After Eating a Hat. But How Does a Hat Kill a Hippo?
0Wired

Why ChatGPT Cites Review Sites Instead of Your Company Website
0dev.to
Basalt walkthrough is up: why 776 tok/s needs the whole card to itself
0dev.to