grid search
In plain wordsTuning a model by trying every combination from a short list of candidate settings, like testing every pairing on a menu, and keeping whichever combination scores best.
A hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking. strategy that fixes a short list of candidate values for each hyperparameter, then trains and scores a model for every combination in their Cartesian product (three learning rates crossed with three tree depths gives nine runs), usually with cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn., and keeps the best. It is simple, reproducible and easily parallelized, and for one or two hyperparameters it yields an interpretable map of performance. Its cost grows exponentially with the number of hyperparameters (m values on each of d hyperparameters requires m^d training runs), and it wastes effort when only a few hyperparameters matter, because it tests the same handful of values of each important one again and again. For these reasons random searchrandom search, n.: Tuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values. or Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next. is usually preferred beyond two or three dimensions. Common refinements are coarse-to-fine grids, which zoom in around the best region, and logarithmically spaced values for scale parameters such as the learning rate and regularization strength.
A grid search over C and gamma found the SVM's sweet spot at C = 10 and gamma = 0.01.
Origin
Exhaustive evaluation over a grid is an old technique of numerical optimization and experimental design. It became the default tuning method of machine-learning practice in the 2000s, recommended for support vector machines in the LIBSVM authors' 2003 practical guide and built into libraries such as scikit-learn as GridSearchCV.
Related
- hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking.
- random searchrandom search, n.: Tuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values.
- Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next.
- cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn.
- hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data.
- AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed.
- random searchrandom search, n.: Tuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values.
Sources
- Hsu, C.-W., Chang, C.-C. & Lin, C.-J. (2003). A Practical Guide to Support Vector Classification. Technical report, Department of Computer Science, National Taiwan University.
- Bergstra, J. & Bengio, Y. (2012). Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research 13.
Mentioned in
Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next., hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking., random searchrandom search, n.: Tuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values.