Skip to content
Last Dictionary
Plane
Machine LearningMathematics & StatisticsAI Engineering & MLOps

Bayesian optimization

nounAlso: BO, sequential model-based optimization, SMBO, acquisition function, expected improvement, tree-structured Parzen estimator, TPE

In plain wordsA smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next.

  1. A method for finding the best input to a costly, possibly noisy function that can be evaluated but not analyzed, such as a model's validation score as a function of its hyperparameters (hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking., its best-known use). After each trial it refits a probabilistic stand-in for the objective (the surrogate) to all results so far and consults it to pick the next point. The classic surrogate is a Gaussian processGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them., which gives both a predicted value and an uncertainty everywhere; alternatives include random forests (as in the SMAC tool) and the tree-structured Parzen estimator, which models the densities of good and bad configurations. An acquisition function, such as expected improvement, probability of improvement or an upper confidence boundupper confidence bound, n.: A rule for choosing among uncertain options that gives each one a bonus for how little is known about it, so promising but rarely tried options get a fair chance., scores candidate points by balancing exploitation (sampling where the surrogate predicts good values) against exploration (sampling where it is uncertain); maximizing it is cheap compared with evaluating the real objective. It suits problems with up to roughly 20 continuous dimensions; beyond tuning, it is used in experimental design, materials and drug discovery, and robotics.

    Bayesian optimization found a better learning-rate schedule in 30 trials than random search did in 150.

αEI(x)=E[max⁡(0,  f(x)−f(x+))]\alpha_{\mathrm{EI}}(\mathbf{x}) = \mathbb{E}\big[\max\big(0,\; f(\mathbf{x}) - f(\mathbf{x}^{+})\big)\big]

Origin

Harold Kushner's 1964 work on locating the maximum of a noisy function, and Jonas Mockus's research in the 1970s, which introduced the expected-improvement criterion and to which the name is usually credited, founded the approach. The EGO algorithm of Donald Jones, Matthias Schonlau and William Welch (1998) popularized it for engineering design, and Jasper Snoek, Hugo Larochelle and Ryan Adams's 2012 paper made it a standard tool for tuning machine-learning models.

Usage note

Not a method for training Bayesian models: 'Bayesian' refers to the probabilistic surrogate, which is updated by Bayes' rule as observations arrive. In the formula, x⁺ is the best point found so far.

Related

See also
  • hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking.
  • Gaussian processGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them.
  • random searchrandom search, n.: Tuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values.
  • grid searchgrid search, n.: Tuning a model by trying every combination from a short list of candidate settings, like testing every pairing on a menu, and keeping whichever combination scores best.
  • AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed.
  • upper confidence boundupper confidence bound, n.: A rule for choosing among uncertain options that gives each one a bonus for how little is known about it, so promising but rarely tried options get a fair chance.
  • HyperbandHyperband, n.: A tuning method that starts many candidate settings on small budgets, keeps only the most promising ones, and gives the survivors more training time, so effort isn't wasted on losers.
Contrast with
  • Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability.

Sources

  1. Mockus, J. (1975). On Bayesian Methods for Seeking the Extremum. In Optimization Techniques IFIP Technical Conference, LNCS 27. Springer.
  2. Jones, D. R., Schonlau, M. & Welch, W. J. (1998). Efficient Global Optimization of Expensive Black-Box Functions. Journal of Global Optimization 13(4).
  3. Snoek, J., Larochelle, H. & Adams, R. P. (2012). Practical Bayesian Optimization of Machine Learning Algorithms. Advances in Neural Information Processing Systems 25.

Mentioned in

optimizationoptimization, n.: Finding the best possible choice according to some score, such as the lowest error or the highest reward. Training an AI is optimization: adjusting its settings until its mistakes are as small as possible., AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed., Bayes' theoremBayes' theorem, n.: A formula for updating how much you believe something after seeing new evidence: weigh your earlier belief by how well that explanation predicts what you actually observed., Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability., Gaussian processGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them., grid searchgrid search, n.: Tuning a model by trying every combination from a short list of candidate settings, like testing every pairing on a menu, and keeping whichever combination scores best., hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking., random searchrandom search, n.: Tuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values., surrogate modelsurrogate model, n.: A simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply., decision theorydecision theory, n.: The study of how to make the best choice when outcomes are uncertain, by weighing how likely each result is against how much it matters., evolution strategiesevolution strategies, n.: A family of evolution-inspired methods for tuning a list of numbers: try many random variations of the current best settings, keep those that work better, and adjust how big the variations are as you go., HyperbandHyperband, n.: A tuning method that starts many candidate settings on small budgets, keeps only the most promising ones, and gives the survivors more training time, so effort isn't wasted on losers.