bagging

An ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model. method that fits one base learner B times, each time to a fresh bootstrapbootstrap, n.: Estimating how much a result might vary by repeatedly re-drawing samples, with replacement, from the data you already have — like re-running a study many times without collecting new data. resample of the training set (n examples drawn with replacement from the original n), and then pools the fitted models, taking their mean prediction in regressionregression, n.: Predicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known. and their majority vote in classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled.. Averaging leaves bias roughly unchanged but reduces variance, so bagging helps most with unstable, high-variance learners such as deep decision treesdecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer. and little with stable ones such as k-nearest neighborsk-nearest neighbors, n.: Predicting something about a new example by finding the few most similar examples already known and going with their majority answer, or their average. or linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest.. Because each bootstrap sample omits on average about 36.8% (1/e) of the examples, every example can be predicted by the models that never saw it; these out-of-bag predictions estimate test error without a separate validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one.. Members are trained independently, so bagging parallelizes trivially. Random forestsrandom forest, n.: A prediction method that builds hundreds of decision trees, each from a random sample of the data and features, and lets them vote. The crowd is far more reliable than any single tree. are bagged trees with random feature selection added.

Bagging fifty deep trees cut the test error by a third, and the out-of-bag estimate matched the holdout.