LightGBM
In plain wordsA widely used tool for making predictions from very large tables of data, quickly and in little memory, by combining many small decision trees. Microsoft released it as free software in 2016.
A gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it. library for decision treesdecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer. ('Light Gradient Boosting Machine') built around shortcuts in how trees are grown, so that ensembles can be trained on millions of rows with modest time and memory. It buckets continuous features into histograms to find splits quickly; grows trees leaf-wise, always splitting the leaf with the largest loss reduction (with depth and leaf-count limits to curb overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.) rather than level by level; and splits categorical features natively by dividing categories into two groups ordered by gradient statistics. Two techniques from its 2017 paper cut work further: gradient-based one-side sampling (GOSS) keeps examples with large gradients and subsamples those with small ones, and exclusive feature bundling (EFB) merges sparse features that are rarely nonzero together. It supports learning to ranklearning to rank, n.: Training a model to put items in the best order — search results, products, recommendations — by learning from examples of which items people judged more relevant or chose more often. objectives and distributed and GPU training, and is a mainstay of KaggleKaggle, prop. n.: A popular website, owned by Google, where people compete to build the most accurate prediction models on shared datasets, publish their code and data, and learn data science. solutions alongside XGBoostXGBoost, prop. n.: A fast, free software tool that makes predictions by adding up hundreds of small decision trees, each one fixing the mistakes of those before it. For years it was behind many winning entries in competitions on spreadsheet-style data. and CatBoostCatBoost, prop. n.: A free gradient-boosting library from the tech company Yandex, known for handling categories such as city names or product types directly, without first converting them into numbers by hand..
LightGBM trained on the 40-million-row click log in minutes where our old pipeline took hours.
Origin
Released as open source by Microsoft in 2016 under the MIT License, with Guolin Ke of Microsoft Research as original author. It was described in 'LightGBM: A Highly Efficient Gradient Boosting Decision Tree' by Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye and Tie-Yan Liu at NeurIPS 2017.
Usage note
Leaf-wise growth reaches lower training loss for a given number of leaves but can overfit small datasets unless num_leaves, max_depth or min_data_in_leaf are constrained.
Related
- gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it.
- XGBoostXGBoost, prop. n.: A fast, free software tool that makes predictions by adding up hundreds of small decision trees, each one fixing the mistakes of those before it. For years it was behind many winning entries in competitions on spreadsheet-style data.
- CatBoostCatBoost, prop. n.: A free gradient-boosting library from the tech company Yandex, known for handling categories such as city names or product types directly, without first converting them into numbers by hand.
- decision treedecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer.
- tabular datatabular data, n.: Data arranged as a table, like a spreadsheet: each row is a case — a customer, a loan, a patient — and each column a measured property. Much business and scientific data looks like this.
- learning to ranklearning to rank, n.: Training a model to put items in the best order — search results, products, recommendations — by learning from examples of which items people judged more relevant or chose more often.
- KaggleKaggle, prop. n.: A popular website, owned by Google, where people compete to build the most accurate prediction models on shared datasets, publish their code and data, and learn data science.
Sources
- Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q. & Liu, T.-Y. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. Advances in Neural Information Processing Systems 30. papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision-tree.pdf
Mentioned in
boostingboosting, n.: Building a strong predictor from many weak ones trained one after another, where each new model concentrates on the mistakes the previous ones made., CatBoostCatBoost, prop. n.: A free gradient-boosting library from the tech company Yandex, known for handling categories such as city names or product types directly, without first converting them into numbers by hand., gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it., KaggleKaggle, prop. n.: A popular website, owned by Google, where people compete to build the most accurate prediction models on shared datasets, publish their code and data, and learn data science., random forestrandom forest, n.: A prediction method that builds hundreds of decision trees, each from a random sample of the data and features, and lets them vote. The crowd is far more reliable than any single tree., tabular datatabular data, n.: Data arranged as a table, like a spreadsheet: each row is a case — a customer, a loan, a patient — and each column a measured property. Much business and scientific data looks like this., XGBoostXGBoost, prop. n.: A fast, free software tool that makes predictions by adding up hundreds of small decision trees, each one fixing the mistakes of those before it. For years it was behind many winning entries in competitions on spreadsheet-style data., Jerome FriedmanJerome Friedman, prop. n.: An American statistician who invented gradient boosting, the method behind XGBoost and other leading tools for spreadsheet-style data, and co-wrote a standard machine-learning textbook., learning to ranklearning to rank, n.: Training a model to put items in the best order — search results, products, recommendations — by learning from examples of which items people judged more relevant or chose more often.