AI & ML
Factual recall in linear associative memories: sharp asymptotics and mechanistic insights
arXiv stat.ML0 views
arXiv:2605.10795v2 Announce Type: replace Abstract: Large language models excel at factual recall, yet the fundamental limits of storing input--output associations with neural networks remain unclear. We study these limits in a minimal setting: a linear associative memory that maps $p$ input embeddings in $\mathbb{R}^d$ to their $d$-dimensional targets via a linear layer, requiring each mapped input to score its own target above all other outputs. Since all inputs compete for the same outputs, these constraints are strongly correlated, which makes the storage capacity $p_c$ --- the largest number of associations that can be stored --- hard to characterise. We address this in three steps. First, we introduce a decoupled model, in which each input competes with its own independent set of outputs, and provide numerical and analytical evidence that it is equivalent to the original model, both in capacity and in the structure of the learnt solution. Second, using tools from statistical physics, we show that the decoupled model has capacity $p_c \log p_c / d^2 = 1/2$, extend this computation to linear two-layer networks, and go beyond the capacity by predicting the spectrum of the learnt weights and the storage mechanism. The latter explains how the optimal solution outperforms a na\"ive Hebbian rule: rather than boosting input--output alignments with broad fluctuations, it raises the correct scores just above the extreme-value threshold set by the competing outputs. Third, we prove the decoupled model's capacity thresholds rigorously, including in the non-convex two-layer case; combined with a recent proof for the original model, this shows that both models have the same capacity for full-rank weights. These findings give a sharp characterisation of factual storage in linear networks, and suggest decoupling as a general tool for memorisation and learning problems in which each input must be matched to one of an extensive number of outputs.
Source:https://arxiv.org/abs/2605.10795PreviousEstimating Implicit Regularization in Deep LearningNextData-Driven Stochastic LQR Control with Probabilistic Guarantees and Uncertainty Refinement
Related posts
Polynomial-Time Differentially Private Synthetic Graph Release with Nearly-Linear Additive Error
0arXiv stat.TH
A geometric proof of multivariate self-normalized moderate deviations via ridge regularization
0arXiv stat.TH
Geometric ergodicity of the block Gibbs sampler for Bayesian linear mixed models with scale mixtures of normal random effects
0arXiv stat.TH
On Natural Exponential Families with Log-Affine Generalized Variance
0arXiv stat.TH
Inference for Realized Quadratic Covariation of $R$ and $R^2$ as a Measure of Return Asymmetry
0arXiv stat.TH
Variational Splines and Generalized Gaussian Processes
0arXiv stat.TH