stacking

An ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model. method in which the predictions of several diverse base models (level 0) become the input features of a second-level model, the meta-learner or combiner, which is trained to produce the final prediction. To avoid leakage, the combiner is trained on out-of-fold predictions: each base model is fitted within k-fold cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn. and predicts only on the fold it did not see, so the combiner learns how the base models behave on unseen data. Simple regularized combiners — linear or logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve., often with non-negative weights — are typical, and stacks can have several levels. Blending is a simpler variant that trains the combiner on a single held-out set. Stacking is a staple of KaggleKaggle, prop. n.: A popular website, owned by Google, where people compete to build the most accurate prediction models on shared datasets, publish their code and data, and learn data science. winning solutions and of AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed. systems.

Our final submission stacked a CatBoost model, a LightGBM model and a neural net under a ridge-regression combiner.