AI & ML
IQS-BO: In-Context Query Selection for Bayesian Optimisation
arXiv stat.ML0 views
arXiv:2610.01269v2 Announce Type: replace-cross Abstract: Bayesian Optimisation (BO) is a powerful framework for optimising expensive black-box functions, but typically requires refitting a surrogate and maximising an acquisition function at every evaluation. In-context approaches based on Prior-data Fitted Networks (PFNs) amortise part of this cost by pre-training transformers on functions drawn from synthetic priors. PFNS4BO amortises the surrogate but still relies on a numerically maximised acquisition function, while FIBO performs BO fully in-context by sampling optimiser locations from a learned density, which fixes the decision rule and admits no surrogate. Learned acquisition functions score a finite candidate set with a network, but, lacking a label for the query, learn the score by reinforcement learning. We propose IQS-BO, a PFN that learns the query decision by supervised learning on synthetic priors. In a single forward pass, IQS-BO predicts the probability that each candidate is the best in the set, and we show that training minimises its expected KL divergence to the posterior probability of this event. The model can be pre-trained without a surrogate for fully in-context BO, or take the predictions of a fixed probabilistic surrogate as additional input, amortising only the decision step. Our method proposes queries at a fraction of the cost of acquisition-based methods, while matching or outperforming BO with Gaussian processes (GPs) and available in-context methods in sequential and batch optimisation, in low and high dimensions. Finally, we propose a mixture prior for pre-training PFNs which combines samples from GPs with functions exhibiting warped inputs, narrow optima, or plateaus that are poorly modelled by stationary GP kernels. We show that pre-training on this prior can improve performance and help the network generalise to a broader range of optimisation problems.
Source:https://arxiv.org/abs/2610.01269PreviousSwiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode TimeNextPrescriptive Learning of Linear Policies with Feasibility Guarantees for Binary Decision-Making
Related posts
The Heater Room So Far, in Seven Posts
0dev.to TypeScript
How I Built a Zero-Signup, Ad-Free Image Host on Cloudflare R2 (and Stopped Fighting Cluttered Image Sites)
0dev.to JavaScript
Why the link box is a plain input, not a drag-and-drop zone
0dev.to JavaScript
Re-Testing LLM-Based C-to-Rust Translations Against Their Original C
0arXiv cs.SE
Reliability is a user-specific property of large language models
0arXiv cs.SE
Safe Personal RSI: Evaluating and Repairing Personal-Agent Updates
0arXiv cs.SE