Skip to content
Last Dictionary
Plane
Machine LearningAI Engineering & MLOps

Hyperband

nounAlso: successive halving, asynchronous successive halving, ASHA, BOHB, multi-fidelity optimization

In plain wordsA tuning method that starts many candidate settings on small budgets, keeps only the most promising ones, and gives the survivors more training time, so effort isn't wasted on losers.

  1. A hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking. algorithm that makes random searchrandom search, n.: Tuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values. cheaper by giving most of the training budget to configurations that look promising early and abandoning the rest. Its building block, successive halving, starts n randomly sampled configurations on a small budget (for example a few training epochs or a subset of data), evaluates them, keeps the best fraction (typically a half or a third) and repeats with a proportionally larger budget until one configuration remains. Because the right trade-off between trying many configurations and training each one longer is unknown in advance, Hyperband runs several brackets of successive halving with different starting numbers and budgets, hedging between aggressive early stopping and full training. It assumes that early performance predicts final performance, which fails when good configurations start slowly, as with small learning rates. Asynchronous successive halving (ASHA) adapts the idea to large parallel clusters, and BOHB replaces random sampling with Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next.; such multi-fidelity methods underlie many modern tuning tools.

    Hyperband discarded 80% of the configurations after a single epoch and found a winner in a fifth of the GPU-hours.

Origin

Successive halving was analyzed for multi-armed bandits by Zohar Karnin, Tomer Koren and Oren Somekh (2013) and applied to hyperparameters by Kevin Jamieson and Ameet Talwalkar (2016). Lisha Li, Jamieson, Giulia DeSalvo, Afshin Rostamizadeh and Talwalkar introduced Hyperband in a 2016 preprint, later published at ICLR 2017 and in the Journal of Machine Learning Research (2018).

Related

See also
  • hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking.
  • random searchrandom search, n.: Tuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values.
  • Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next.
  • AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed.
  • early stoppingearly stopping, n.: Stopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples.
  • multi-armed banditmulti-armed bandit, n.: A puzzle named after a gambler choosing between slot machines: you must keep picking among options whose payoffs are unknown, learning which is best while wasting as few tries as possible on the others.

Sources

  1. Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A. & Talwalkar, A. (2018). Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. Journal of Machine Learning Research 18(185).
  2. Jamieson, K. & Talwalkar, A. (2016). Non-stochastic Best Arm Identification and Hyperparameter Optimization. Proc. AISTATS 2016.
  3. Falkner, S., Klein, A. & Hutter, F. (2018). BOHB: Robust and Efficient Hyperparameter Optimization at Scale. Proc. ICML 2018.

Mentioned in

AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed., Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next., hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking., random searchrandom search, n.: Tuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values.