Backend
Gradient Regularized Newton Boosting Trees with Global Convergence
arXiv stat.ML0 views
arXiv:2605.00581v2 Announce Type: replace Abstract: Gradient Boosting Decision Trees (GBDTs) dominate tabular machine learning, with modern implementations like XGBoost, LightGBM, and CatBoost being based on Newton boosting: a second-order descent step in the space of decision trees. Despite its empirical success, the global convergence of Newton boosting is poorly understood compared to first-order boosting. In this paper, we introduce Restricted Newton Descent, a framework for convex optimization with Newton's method on Hilbert spaces with inexact iterates, based on the concepts of cosine angle and weak gradient edge. Within this framework, we recover Newton boosting with GBDTs and classical finite-dimensional theory as special cases. We first prove that vanilla Newton boosting achieves a linear rate of convergence for smooth, strongly convex losses that satisfy a Hessian-dominance condition. To handle general convex losses with Lipschitz Hessians, we extend a recent gradient regularized Newton scheme to the restricted weak learner setting, establishing the first global $\mathcal{O}(\frac{1}{k^2})$ convergence rate for second-order GBDTs. This scheme minimally modifies the classical algorithm by introducing an adaptive $\ell_2$-regularization term proportional to the square root of the gradient norm at each iteration. In numerical experiments, we show that this scheme converges while vanilla Newton boosting may diverge.
Source:https://arxiv.org/abs/2605.00581PreviousLearning Curves and Prediction Persistence to Under-Regularization of Spectral Algorithms in Large DimensionsNextSurvey-aware Machine Learning: A Guideline for Valid Population Health Inference based on Scoping Review
Related posts
Replaying Yesterday: Designing a Date Picker for a Daily Puzzle
0dev.to
A refusal that changes no bytes
0dev.to
NestJS for Spring Boot developers: a TypeScript backend that will feel familiar
0dev.to
NestJS for ASP.NET Core developers: a TypeScript backend that will feel familiar
0dev.to

The Hopper Trap: When the Optimal Tile Size Wastes Half Your Register File
0dev.to
Offline reverse geocoding on iPhone with one memory-mapped file
0dev.to