Skip to content
Last Dictionary
Plane
Machine LearningAI Engineering & MLOps

AutoML

nounAlso: automated machine learning, Auto-WEKA, auto-sklearn, CASH problem

In plain wordsSoftware that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed.

  1. The automation of the design decisions involved in building a machine-learning model: data preprocessing and feature engineeringfeature engineering, n.: Turning raw data into the inputs a model learns from, using knowledge of the problem; for example, converting a timestamp into 'hour of day' and 'is it a weekend'., choice of algorithm, hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking., and often ensembling and deployment. A central formulation is the combined algorithm selection and hyperparameter optimization (CASH) problem: search jointly over model families and their settings to maximize validation performance. Systems combine search strategies such as Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next., evolutionary algorithms and multi-fidelity methods like HyperbandHyperband, n.: A tuning method that starts many candidate settings on small budgets, keeps only the most promising ones, and gives the survivors more training time, so effort isn't wasted on losers. with meta-learningmeta-learning, n.: Training a system across many different tasks so that it gets better at learning itself, letting it pick up a new task quickly from only a few examples. from results on earlier datasets, which warm-starts the search, and frequently finish by stackingstacking, n.: Combining several different models by training one more model to decide how much to trust each of them, based on how they performed on data they had not seen. or ensembling the best models. For deep learning, AutoML includes neural architecture searchneural architecture search, n.: Using an automated search, rather than human trial and error, to design the structure of a neural network: which layers to use, how many, and how to connect them.. Open-source tools (Auto-WEKA, auto-sklearn, TPOT, AutoGluon, H2O AutoML) and cloud services provide it for tabular datatabular data, n.: Data arranged as a table, like a spreadsheet: each row is a case — a customer, a loan, a patient — and each column a measured property. Much business and scientific data looks like this., images and text. Since the 2020s, pretrained foundation modelsfoundation model, n.: A very large AI model, built once at great cost from vast amounts of text, images or code gathered mostly from the internet, that many products then reuse as their starting point, from chatbots to coding tools. and coding agentscoding agent, n.: An AI agent that writes and fixes software by itself: it reads a codebase, edits files, runs commands and tests, and keeps working until a feature is built or a bug is fixed. that write and run modeling code have offered further routes to automated model building.

    AutoML produced a stacked ensemble in an hour that our hand-tuned model only matched after a week.

Origin

Automated algorithm choice traces to John Rice's 1976 formulation of the 'algorithm selection problem'. Chris Thornton, Frank Hutter, Holger Hoos and Kevin Leyton-Brown's Auto-WEKA (KDD 2013) framed the CASH problem. The name 'AutoML' spread through ICML workshops from 2014, the ChaLearn AutoML challenges and auto-sklearn (2015), and Google's neural-architecture-search work and Cloud AutoML service (2017–2018).

Usage note

AutoML automates search over modeling choices, not problem framing: defining the target, collecting representative data and avoiding data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints. remain human responsibilities.

Related

See also
  • hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking.
  • neural architecture searchneural architecture search, n.: Using an automated search, rather than human trial and error, to design the structure of a neural network: which layers to use, how many, and how to connect them.
  • Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next.
  • meta-learningmeta-learning, n.: Training a system across many different tasks so that it gets better at learning itself, letting it pick up a new task quickly from only a few examples.
  • model selectionmodel selection, n.: Choosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best.
  • HyperbandHyperband, n.: A tuning method that starts many candidate settings on small budgets, keeps only the most promising ones, and gives the survivors more training time, so effort isn't wasted on losers.
  • stackingstacking, n.: Combining several different models by training one more model to decide how much to trust each of them, based on how they performed on data they had not seen.

Sources

  1. Thornton, C., Hutter, F., Hoos, H. H. & Leyton-Brown, K. (2013). Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms. Proc. KDD 2013.
  2. Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M. & Hutter, F. (2015). Efficient and Robust Automated Machine Learning. Advances in Neural Information Processing Systems 28.
  3. Hutter, F., Kotthoff, L. & Vanschoren, J. (eds.) (2019). Automated Machine Learning: Methods, Systems, Challenges. Springer.

Mentioned in

Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next., grid searchgrid search, n.: Tuning a model by trying every combination from a short list of candidate settings, like testing every pairing on a menu, and keeping whichever combination scores best., hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data., hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking., meta-learningmeta-learning, n.: Training a system across many different tasks so that it gets better at learning itself, letting it pick up a new task quickly from only a few examples., random searchrandom search, n.: Tuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values., HyperbandHyperband, n.: A tuning method that starts many candidate settings on small budgets, keeps only the most promising ones, and gives the survivors more training time, so effort isn't wasted on losers., LionLion, prop. n.: A training method discovered by a computer search over possible algorithms. It moves every setting by the same small amount at each step, choosing only the direction, and needs less memory than the popular Adam method., neural architecture searchneural architecture search, n.: Using an automated search, rather than human trial and error, to design the structure of a neural network: which layers to use, how many, and how to connect them., stackingstacking, n.: Combining several different models by training one more model to decide how much to trust each of them, based on how they performed on data they had not seen., tabular foundation modeltabular foundation model, n.: A neural network pretrained on millions of made-up tables so that, given a new spreadsheet of labeled examples, it can predict the missing answers at once, without being retrained for that table.