boosting
In plain wordsBuilding a strong predictor from many weak ones trained one after another, where each new model concentrates on the mistakes the previous ones made.
A family of ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model. methods that combine many weak learners — models only slightly better than chance, typically shallow decision treesdecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer. — into a strong one by training them sequentially, each new learner focusing on the examples or residual errors the current ensemble handles worst, and summing their weighted predictions. AdaBoostAdaBoost, prop. n.: One of the first practical boosting methods: it trains simple models in turn, giving extra weight to examples the earlier models got wrong, then lets all the models vote, weighted by accuracy. does this by reweighting misclassified examples; gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it. fits each new learner to the negative gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill. of a loss with respect to the current predictions, which casts boosting as gradient descent in function space and as stagewise fitting of an additive model. Boosting chiefly lowers bias, though it can lower variance too; it can overfit if run too long and is controlled by the number of rounds, the learning rate (shrinkage) and the depth of each learner. Gradient-boosted trees such as XGBoostXGBoost, prop. n.: A fast, free software tool that makes predictions by adding up hundreds of small decision trees, each one fixing the mistakes of those before it. For years it was behind many winning entries in competitions on spreadsheet-style data., LightGBMLightGBM, prop. n.: A widely used tool for making predictions from very large tables of data, quickly and in little memory, by combining many small decision trees. Microsoft released it as free software in 2016. and CatBoostCatBoost, prop. n.: A free gradient-boosting library from the tech company Yandex, known for handling categories such as city names or product types directly, without first converting them into numbers by hand. are among the strongest methods for tabular datatabular data, n.: Data arranged as a table, like a spreadsheet: each row is a case — a customer, a loan, a patient — and each column a measured property. Much business and scientific data looks like this..
Boosting stumps for 400 rounds beat a single deep tree by six points of accuracy.
Origin
Grew from a theoretical question posed by Michael Kearns and Leslie Valiant (1988–1989), first in Kearns's unpublished note 'Thoughts on Hypothesis Boosting' (1988): can a learner that is only weakly better than guessing be 'boosted' into an arbitrarily accurate one? Robert Schapire proved it could in 'The Strength of Weak Learnability' (1990), and Yoav Freund's boost-by-majority followed. Freund and Schapire's AdaBoost (1995) made boosting practical; Jerome Friedman, Trevor Hastie and Robert Tibshirani recast it as additive modeling (2000).
Usage note
Contrast with baggingbagging, n.: Training many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error., which trains models independently and in parallel to reduce variance.
Related
- AdaBoostAdaBoost, prop. n.: One of the first practical boosting methods: it trains simple models in turn, giving extra weight to examples the earlier models got wrong, then lets all the models vote, weighted by accuracy.
- gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it.
- ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model.
- baggingbagging, n.: Training many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error.
- XGBoostXGBoost, prop. n.: A fast, free software tool that makes predictions by adding up hundreds of small decision trees, each one fixing the mistakes of those before it. For years it was behind many winning entries in competitions on spreadsheet-style data.
- PAC learningPAC learning, n.: A mathematical definition of what it means to learn from examples: given enough data, a learner should, with high probability, find a rule that is almost always right.
- decision treedecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer.
- baggingbagging, n.: Training many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error.
Sources
- Schapire, R. E. (1990). The Strength of Weak Learnability. Machine Learning, 5(2), 197–227.
- Friedman, J., Hastie, T. & Tibshirani, R. (2000). Additive Logistic Regression: A Statistical View of Boosting. The Annals of Statistics, 28(2), 337–407.
Mentioned in
AdaBoostAdaBoost, prop. n.: One of the first practical boosting methods: it trains simple models in turn, giving extra weight to examples the earlier models got wrong, then lets all the models vote, weighted by accuracy., baggingbagging, n.: Training many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error., bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between., ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model., gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it., marginmargin, n.: How far, and on which side, an example lies from a classifier's dividing line. A wide gap between the line and the nearest examples suggests the classifier will handle new data well., C4.5C4.5, prop. n.: A widely used 1990s program that learns decision trees and rule sets from data; unlike its predecessor, it trims away unhelpful branches, copes with gaps in the data and can split on measured numbers., PAC learningPAC learning, n.: A mathematical definition of what it means to learn from examples: given enough data, a learner should, with high probability, find a rule that is almost always right., stackingstacking, n.: Combining several different models by training one more model to decide how much to trust each of them, based on how they performed on data they had not seen.