Cornell Center for Applied Mathematics Bill Sears Blitz

Algorithmic Design Principles for

Bayesian Optimization

Alexander Terenin

https://avt.im/ · @avt_im

Bayesian Optimization

Automatic explore-exploit tradeoff

Gaussian Processes

Probabilistic formulation provides uncertainty

Thompson Sampling

Same model, different decisions, similar performance

Challenges

Modeling:

  • Scalability and reliability
  • Uncertainty and generalization
  • Smoothness and non-uniformity

Decision-making:

  • Multi-stage feedback
  • Scheduling and asynchronicity
  • What kind of uncertainty is needed?
  • Multi-objective optimization
  • Adversarial objectives

Theoretical guarantees and empirical performance

Thank you!

https://avt.im/· @avt_im

Q. Xie, R. Astudillo, P. Frazier, Z. Scully, A. Terenin. Cost-aware Bayesian Optimization via the Pandora's Box Gittins Index. arXiv:2406.20062, 2024.

J. A. Lin,* S. Padhy,* J. Antorán,* A. Tripp, A. Terenin, Cs. Szepesvári, J. M. Hernández-Lobato, D. Janz. Stochastic Gradient Descent for Gaussian Processes Done Right. ICLR, 2024.

J. A. Lin,* J. Antorán,* S. Padhy,* D. Janz, J. M. Hernández-Lobato, A. Terenin. Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent. NeurIPS, 2023. Selected for Oral Presentation.

A. Terenin,* D. R. Burt,* A. Artemev, S. Flaxman, M. van der Wilk, C. E. Rasmussen, H. Ge. Numerically Stable Sparse Gaussian Processes via Minimum Separation using Cover Trees. JMLR, 2024.

J. T. Wilson,* V. Borovitskiy,* P. Mostowsky,* A. Terenin,* M. P. Deisenroth. Efficiently Sampling Functions from Gaussian Process Posteriors. ICML, 2020. Honorable Mention for Outstanding Paper Award.

J. T. Wilson,* V. Borovitskiy,* P. Mostowsky,* A. Terenin,* M. P. Deisenroth. Pathwise Conditioning of Gaussian Process. JMLR, 2021.