Movatterモバイル変換

[0]ホーム
URL:

Jump to content
WikipediaThe Free Encyclopedia
Search

XGBoost

From Wikipedia, the free encyclopedia
(Redirected fromXgboost)
Gradient boosting machine learning library
XGBoost
DeveloperThe XGBoost Contributors
Initial releaseMarch 27, 2014; 11 years ago (2014-03-27)
Stable release
3.0.0[1] Edit this on Wikidata / 15 March 2025; 8 months ago (15 March 2025)
Repository
Written inC++
Operating systemLinux,macOS,Microsoft Windows
TypeMachine learning
LicenseApache License 2.0
Websitexgboost.ai

XGBoost[2] (eXtreme Gradient Boosting) is anopen-sourcesoftware library which provides aregularizinggradient boosting framework forC++,Java,Python,[3]R,[4]Julia,[5]Perl,[6] andScala. It works onLinux,Microsoft Windows,[7] andmacOS.[8] From the project description, it aims to provide a "Scalable, Portable and Distributed Gradient Boosting (GBM, GBRT, GBDT) Library". It runs on a single machine, as well as the distributed processing frameworksApache Hadoop,Apache Spark,Apache Flink, andDask.[9][10]

XGBoost gained much popularity and attention in the mid-2010s as the algorithm of choice for many winning teams ofmachine learningcompetitions.[11]

History

[edit]

XGBoost initially started as a research project by Tianqi Chen[12] as part of the Distributed (Deep) Machine Learning Community (DMLC) group at theUniversity of Washington. Initially, it began as a terminal application which could be configured using alibsvm configuration file. It became well known in the ML competition circles after its use in the winning solution of theHiggs Machine Learning Challenge. Soon after, the Python and R packages were built, and XGBoost now has package implementations for Java,Scala, Julia,Perl, and other languages. This brought the library to more developers and contributed to its popularity among theKaggle community, where it has been used for a large number of competitions.[11]

It was soon integrated with a number of other packages making it easier to use in their respective communities. It has now been integrated withscikit-learn forPython users and with the caret package forR users. It can also be integrated into Data Flow frameworks likeApache Spark,Apache Hadoop, andApache Flink using the abstracted Rabit[13] and XGBoost4J.[14] XGBoost is also available onOpenCL forFPGAs.[15] An efficient, scalable implementation of XGBoost has been published by Tianqi Chen andCarlos Guestrin.[16]

While the XGBoost model often achieves higher accuracy than a single decision tree, it sacrifices the intrinsic interpretability of decision trees.  For example, following the path that a decision tree takes to make its decision is trivial and self-explained, but following the paths of hundreds or thousands of trees is much harder.

Features

[edit]

Salient features of XGBoost which make it different from other gradient boosting algorithms include:[17][18][16]

The algorithm

[edit]

XGBoost works asNewton–Raphson in function space unlikegradient boosting that works as gradient descent in function space, a second orderTaylor approximation is used in the loss function to make the connection to Newton–Raphson method.

A generic unregularized XGBoost algorithm is:

Input: training set{(xi,yi)}i=1N{\displaystyle \{(x_{i},y_{i})\}_{i=1}^{N}}, a differentiable loss functionL(y,F(x)){\displaystyle L(y,F(x))}, a number of weak learnersM{\displaystyle M} and a learning rateα{\displaystyle \alpha }.

Algorithm:

Note that this is the initialization of the model and therefore we set a constant value for all inputs. So even if in later iterations we use optimization to find new functions, in step 0 we have to find the value, equals for all inputs, that minimizes the loss functions.
  1. Form = 1 toM:
    1. Compute the 'gradients' and 'hessians':[clarification needed]g^m(xi)=[∂L(yi,f(xi))∂f(xi)]f(x)=f^(m−1)(x).h^m(xi)=[∂2L(yi,f(xi))∂f(xi)2]f(x)=f^(m−1)(x).{\displaystyle {\begin{aligned}{\hat {g}}_{m}(x_{i})&=\left[{\frac {\partial L(y_{i},f(x_{i}))}{\partial f(x_{i})}}\right]_{f(x)={\hat {f}}_{(m-1)}(x)}.\\{\hat {h}}_{m}(x_{i})&=\left[{\frac {\partial ^{2}L(y_{i},f(x_{i}))}{\partial f(x_{i})^{2}}}\right]_{f(x)={\hat {f}}_{(m-1)}(x)}.\end{aligned}}}
    2. Fit a base learner (or weak learner, e.g. tree) using the training set
      {xi,g^m(xi)h^m(xi)}i=1N{\displaystyle \left\{x_{i},{\dfrac {{\hat {g}}_{m}(x_{i})}{{\hat {h}}_{m}(x_{i})}}\right\}_{i=1}^{N}}[clarification needed] by solving the optimization problem below:ϕ^m=arg⁡minϕ∈Φ∑i=1N12h^m(xi)[ϕ(xi)−g^m(xi)h^m(xi)]2.{\displaystyle {\hat {\phi }}_{m}={\underset {\phi \in \mathbf {\Phi } }{\arg \min }}\sum _{i=1}^{N}{\frac {1}{2}}{\hat {h}}_{m}(x_{i})\left[\phi (x_{i})-{\frac {{\hat {g}}_{m}(x_{i})}{{\hat {h}}_{m}(x_{i})}}\right]^{2}.}[clarification needed]f^m(x)=αϕ^m(x).{\displaystyle {\hat {f}}_{m}(x)=\alpha {\hat {\phi }}_{m}(x).}
    3. Update the model:f^(m)(x)=f^(m−1)(x)−f^m(x).{\displaystyle {\hat {f}}_{(m)}(x)={\hat {f}}_{(m-1)}(x)-{\hat {f}}_{m}(x).}
  2. Outputf^(x)=f^(M)(x)=∑m=0Mf^m(x).{\displaystyle {\hat {f}}(x)={\hat {f}}_{(M)}(x)=\sum _{m=0}^{M}{\hat {f}}_{m}(x).}

Awards

[edit]

See also

[edit]

References

[edit]
  1. ^"Release 3.0.0 stable". 15 March 2025. Retrieved4 May 2025.
  2. ^"GitHub project webpage".GitHub. June 2022.Archived from the original on 2021-04-01. Retrieved2016-04-05.
  3. ^"Python Package Index PYPI: xgboost".Archived from the original on 2017-08-23. Retrieved2016-08-01.
  4. ^"CRAN package xgboost".Archived from the original on 2018-10-26. Retrieved2016-08-01.
  5. ^"Julia package listing xgboost". Archived fromthe original on 2016-08-18. Retrieved2016-08-01.
  6. ^"CPAN module AI::XGBoost".Archived from the original on 2020-03-28. Retrieved2020-02-09.
  7. ^"Installing XGBoost for Anaconda in Windows".IBM.Archived from the original on 2018-05-08. Retrieved2016-08-01.
  8. ^"Installing XGBoost on Mac OSX".IBM.Archived from the original on 2018-05-08. Retrieved2016-08-01.
  9. ^"Dask Homepage".Archived from the original on 2022-09-14. Retrieved2021-07-15.
  10. ^"Distributed XGBoost with Dask — xgboost 1.5.0-dev documentation".xgboost.readthedocs.io.Archived from the original on 2022-06-04. Retrieved2021-07-15.
  11. ^ab"XGBoost - ML winning solutions (incomplete list)".GitHub.Archived from the original on 2017-08-24. Retrieved2016-08-01.
  12. ^"Story and Lessons behind the evolution of XGBoost". Archived fromthe original on 2016-08-07. Retrieved2016-08-01.
  13. ^"Rabit - Reliable Allreduce and Broadcast Interface".GitHub.Archived from the original on 2018-06-11. Retrieved2016-08-01.
  14. ^"XGBoost4J".Archived from the original on 2018-05-08. Retrieved2016-08-01.
  15. ^"XGBoost on FPGAs".GitHub.Archived from the original on 2020-09-13. Retrieved2019-08-01.
  16. ^abChen, Tianqi; Guestrin, Carlos (2016). "XGBoost: A Scalable Tree Boosting System". In Krishnapuram, Balaji; Shah, Mohak; Smola, Alexander J.; Aggarwal, Charu C.; Shen, Dou; Rastogi, Rajeev (eds.).Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016. ACM. pp. 785–794.arXiv:1603.02754.doi:10.1145/2939672.2939785.ISBN 9781450342322.S2CID 4650265.
  17. ^Gandhi, Rohith (2019-05-24)."Gradient Boosting and XGBoost".Medium.Archived from the original on 2020-03-28. Retrieved2020-01-04.
  18. ^"Tree Boosting With XGBoost – Why Does XGBoost Win "Every" Machine Learning Competition?".Synced. 2017-10-22.Archived from the original on 2020-03-28. Retrieved2020-01-04.
  19. ^"John Chambers Award Previous Winners".Archived from the original on 2017-07-31. Retrieved2016-08-01.
  20. ^"HEP meets ML Award".Archived from the original on 2018-05-08. Retrieved2016-08-01.
Retrieved from "https://en.wikipedia.org/w/index.php?title=XGBoost&oldid=1321651837"
Categories:
Hidden categories:
[8]ページ先頭
©2009-2025 Movatter.jp