random forest
In plain wordsA prediction method that builds hundreds of decision trees, each from a random sample of the data and features, and lets them vote. The crowd is far more reliable than any single tree.
An ensembleensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model. of decision treesdecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer. in which each tree is grown on a bootstrapbootstrap, n.: Estimating how much a result might vary by repeatedly re-drawing samples, with replacement, from the data you already have — like re-running a study many times without collecting new data. sample of the training data (baggingbagging, n.: Training many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error.) and each split considers only a random subset of the features — commonly about √p of p features for classification and p/3 for regression. Trees are usually grown deep without pruning, and predictions are combined by majority vote or by averaging probabilities or values. Random feature selection decorrelates the trees, so averaging cuts variance more than bagging alone with little increase in bias. Each tree's out-of-bag examples, those left out of its bootstrap sample, give a nearly free estimate of generalization error, and permuting a feature within them yields permutation feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect.. Random forests need little tuning, do not overfit as more trees are added and remain strong baselinesbaseline, n.: A simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value. for tabular datatabular data, n.: Data arranged as a table, like a spreadsheet: each row is a case — a customer, a loan, a patient — and each column a measured property. Much business and scientific data looks like this.. Extremely randomized trees (extra-trees) also draw split thresholds at random.
A 500-tree random forest with default settings came within a point of our tuned boosting model.
Origin
Tin Kam Ho introduced 'random decision forests' in 1995, training trees on random subsets of features (later called the random subspace method), and Yali Amit and Donald Geman (1997) randomized the choice of splits. Leo Breiman's 'Random Forests' (Machine Learning, 2001) combined bagging with random feature selection, used out-of-bag estimates, introduced permutation importance and fixed the method's standard form. 'Random Forests' was registered as a trademark in 2006, credited to Breiman and Adele Cutler and held by Minitab as of 2019.
Usage note
Gradient-boosted trees (XGBoostXGBoost, prop. n.: A fast, free software tool that makes predictions by adding up hundreds of small decision trees, each one fixing the mistakes of those before it. For years it was behind many winning entries in competitions on spreadsheet-style data., LightGBMLightGBM, prop. n.: A widely used tool for making predictions from very large tables of data, quickly and in little memory, by combining many small decision trees. Microsoft released it as free software in 2016., CatBoostCatBoost, prop. n.: A free gradient-boosting library from the tech company Yandex, known for handling categories such as city names or product types directly, without first converting them into numbers by hand.) usually edge out random forests in accuracy, but forests are harder to misconfigure. Impurity-based importances from forests are biased toward features with many distinct values.
Related
- decision treedecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer.
- baggingbagging, n.: Training many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error.
- ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model.
- gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it.
- feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect.
- isolation forestisolation forest, n.: A method for spotting unusual data points by repeatedly slicing the data at random: odd points get cut off from the rest after only a few slices, while ordinary points take many.
- XGBoostXGBoost, prop. n.: A fast, free software tool that makes predictions by adding up hundreds of small decision trees, each one fixing the mistakes of those before it. For years it was behind many winning entries in competitions on spreadsheet-style data.
- gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it.
Sources
- Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5–32. stat.berkeley.edu/~breiman/randomforest2001.pdf
- Ho, T. K. (1995). Random Decision Forests. Proceedings of the 3rd International Conference on Document Analysis and Recognition, Montreal, 278–282.
Mentioned in
baggingbagging, n.: Training many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error., bootstrapbootstrap, n.: Estimating how much a result might vary by repeatedly re-drawing samples, with replacement, from the data you already have — like re-running a study many times without collecting new data., decision treedecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer., ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model., feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect., Gini impurityGini impurity, n.: A score for how mixed a group is: zero when every item belongs to the same category, higher when categories are jumbled together. Decision trees choose splits that make groups purer., gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it., imputationimputation, n.: Filling in missing values in a dataset with sensible estimates, such as the typical value or a prediction from the other columns, so the data can be analyzed or used for training., isolation forestisolation forest, n.: A method for spotting unusual data points by repeatedly slicing the data at random: odd points get cut off from the rest after only a few slices, while ordinary points take many., Leo BreimanLeo Breiman, prop. n.: An American statistician who invented random forests and bagging, decision-tree methods still in wide use, and urged statisticians to judge models by how well they predict., scikit-learnscikit-learn, prop. n.: A free, widely taught Python library of classic machine-learning methods — such as decision trees, regression and clustering — that all work in the same simple way., tabular datatabular data, n.: Data arranged as a table, like a spreadsheet: each row is a case — a customer, a loan, a patient — and each column a measured property. Much business and scientific data looks like this., XGBoostXGBoost, prop. n.: A fast, free software tool that makes predictions by adding up hundreds of small decision trees, each one fixing the mistakes of those before it. For years it was behind many winning entries in competitions on spreadsheet-style data., CARTCART, abbr.: A classic recipe for building decision trees that splits data two ways at every step, works for both categories and numbers, and then trims the tree back so it does not memorize noise., explainable boosting machineexplainable boosting machine, n.: A prediction method for spreadsheet-style data that draws each input's influence as its own readable graph and adds those effects up to make every prediction, usually losing little accuracy compared with opaque methods., survival analysissurvival analysis, n.: Methods for predicting how long until something happens — a patient's relapse, a machine's failure, a customer's cancellation — that properly use the cases where it has not happened yet., uplift modelinguplift modeling, n.: Predicting not who will buy, recover or respond, but whose behavior will actually change because of an action such as a discount or a treatment, so that it is offered only where it makes a difference.