CatBoost
In plain wordsA free gradient-boosting library from the tech company Yandex, known for handling categories such as city names or product types directly, without first converting them into numbers by hand.
A gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it. library for decision treesdecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer., released as open source by Yandex, whose distinguishing ideas deal with categorical features and with a subtle form of target leakage inside boosting itself. Categorical values are encoded with ordered target statistics: each example's category is replaced by an average of the target over examples that precede it in a random permutation, so an example's own label never leaks into its encoding; combinations of categorical features are added greedily. Ordered boosting applies the same idea to the boosting procedure, computing each example's residual with models that have not seen that example, which counters the 'prediction shift' its authors identified in standard implementations. By default it grows oblivious (symmetric) trees, which use the same split at every node of a given depth, giving fast prediction and some regularization. It supports GPUs, ranking and text features, and is known for strong results with default settings.
We passed the raw store and product IDs straight to CatBoost instead of building a target encoding ourselves.
Origin
Developed at Yandex, which had earlier used its proprietary MatrixNet boosting for search ranking, with Andrey Gulin as original author; open-sourced under the Apache 2.0 license in July 2017. Its methods were described by Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush and Gulin in 'CatBoost: Unbiased Boosting with Categorical Features' (arXiv 2017; NeurIPS 2018). The name joins 'category' and 'boosting'.
Related
- gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it.
- XGBoostXGBoost, prop. n.: A fast, free software tool that makes predictions by adding up hundreds of small decision trees, each one fixing the mistakes of those before it. For years it was behind many winning entries in competitions on spreadsheet-style data.
- LightGBMLightGBM, prop. n.: A widely used tool for making predictions from very large tables of data, quickly and in little memory, by combining many small decision trees. Microsoft released it as free software in 2016.
- tabular datatabular data, n.: Data arranged as a table, like a spreadsheet: each row is a case — a customer, a loan, a patient — and each column a measured property. Much business and scientific data looks like this.
- one-hot encodingone-hot encoding, n.: Turning a category such as 'red', 'green' or 'blue' into a row of zeros with a single one marking which category it is, so that a model can use it as numbers.
- data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints.
- decision treedecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer.
Sources
- Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V. & Gulin, A. (2018). CatBoost: Unbiased Boosting with Categorical Features. Advances in Neural Information Processing Systems 31. arXiv:1706.09516. arxiv.org/abs/1706.09516
Mentioned in
boostingboosting, n.: Building a strong predictor from many weak ones trained one after another, where each new model concentrates on the mistakes the previous ones made., gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it., KaggleKaggle, prop. n.: A popular website, owned by Google, where people compete to build the most accurate prediction models on shared datasets, publish their code and data, and learn data science., LightGBMLightGBM, prop. n.: A widely used tool for making predictions from very large tables of data, quickly and in little memory, by combining many small decision trees. Microsoft released it as free software in 2016., random forestrandom forest, n.: A prediction method that builds hundreds of decision trees, each from a random sample of the data and features, and lets them vote. The crowd is far more reliable than any single tree., tabular datatabular data, n.: Data arranged as a table, like a spreadsheet: each row is a case — a customer, a loan, a patient — and each column a measured property. Much business and scientific data looks like this., XGBoostXGBoost, prop. n.: A fast, free software tool that makes predictions by adding up hundreds of small decision trees, each one fixing the mistakes of those before it. For years it was behind many winning entries in competitions on spreadsheet-style data., Jerome FriedmanJerome Friedman, prop. n.: An American statistician who invented gradient boosting, the method behind XGBoost and other leading tools for spreadsheet-style data, and co-wrote a standard machine-learning textbook., learning to ranklearning to rank, n.: Training a model to put items in the best order — search results, products, recommendations — by learning from examples of which items people judged more relevant or chose more often.