| LightGBM | |
|---|---|
 | |
| Original author | Guolin Ke[1] /Microsoft Research |
| Developers | Microsoft and LightGBM contributors[2] |
| Initial release | 2016; 10 years ago (2016) |
| Stable release | |
| Written in | C++,Python,R,C |
| Operating system | Windows,macOS,Linux |
| Type | Machine learning,gradient boosting framework |
| License | MIT License |
| Website | lightgbm |
| Repository | github |
LightGBM, short forLight Gradient-Boosting Machine, is afree and open-source distributedgradient-boosting framework formachine learning, originally developed byMicrosoft.[4][5] It is based ondecision tree algorithms and used forranking,classification and other machine learning tasks. The development focus is on performance and scalability.
The LightGBM framework supports different algorithms including GBT,GBDT,GBRT,GBM,MART[6][7] andRF.[8] LightGBM has many ofXGBoost's advantages, including sparse optimization, parallel training, multiple loss functions, regularization, bagging, and early stopping. A major difference between the two lies in the construction of trees. LightGBM does not grow a tree level-wise — row by row — as most other implementations do.[9] Instead it grows trees leaf-wise. It will choose the leaf with max delta loss to grow.[10] Besides, LightGBM does not use the widely used sorted-based decision tree learning algorithm, which searches the best split point on sorted feature values,[11] asXGBoost or other implementations do. Instead, LightGBM implements a highly optimized histogram-based decision tree learning algorithm, which yields great advantages on both efficiency and memory consumption.[12] The LightGBM algorithm utilizes two novel techniques called Gradient-Based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB) which allow the algorithm to run faster while maintaining a high level of accuracy.[13]
LightGBM works onLinux,Windows, andmacOS and supportsC++,Python,[14]R, andC#.[15] The source code is licensed underMIT License and available onGitHub.[16]
When usinggradient descent, one thinks about the space of possible configurations of the model as a valley, in which the lowest part of the valley is the model which most closely fits the data. In this metaphor, one walks in different directions to learn how much lower the valley becomes.
Typically, in gradient descent, one uses the whole set of data to calculate the valley's slopes. However, this commonly used method assumes that every data point is equally informative.
By contrast, Gradient-Based One-Side Sampling (GOSS), a method first developed forgradient-boosted decision trees, does not rely on the assumption that all data are equally informative. Instead, it treats data points with smaller gradients (shallower slopes) as less informative by randomly dropping them. This is intended to filter out data which may have been influenced by noise, allowing the model to more accurately model the underlying relationships in the data.[13]
Exclusive feature bundling (EFB) is a near-lossless method to reduce the number of effective features. In a sparse feature space many features are nearly exclusive, implying they rarely take nonzero values simultaneously. One-hot encoded features are a perfect example of exclusive features. EFB bundles these features, reducing dimensionality to improve efficiency while maintaining a high level of accuracy. The bundle of exclusive features into a single feature is called an exclusive feature bundle.[13]
| Main projects |
| ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MSR Labs applied research |
| ||||||||||||||
