Install XGBoost

To install XGBoost, follow instructions inInstallation Guide.

To verify your installation, run the following in Python:

importxgboostasxgb

Data Interface

The XGBoost Python module is able to load data from many different types of data format including both CPU and GPU data structures. For a complete list of supported data types, please reference theSupported data structures for various XGBoost functions. For a detailed description of text input formats, please visitText Input Format of DMatrix.

The input data is stored in aDMatrix object. For the sklearn estimator interface, aDMatrix or aQuantileDMatrix is created depending on the chosen algorithm and the input, see the sklearn API reference for details. We will illustrate some of the basic input types with theDMatrix here.

• To load a NumPy array intoDMatrix:

  data=np.random.rand(5,10)# 5 entities, each contains 10 featureslabel=np.random.randint(2,size=5)# binary targetdtrain=xgb.DMatrix(data,label=label)

• To load ascipy.sparse array intoDMatrix:

  csr=scipy.sparse.csr_matrix((dat,(row,col)))dtrain=xgb.DMatrix(csr)

• To load a Pandas data frame intoDMatrix:

  data=pandas.DataFrame(np.arange(12).reshape((4,3)),columns=['a','b','c'])label=pandas.DataFrame(np.random.randint(2,size=4))dtrain=xgb.DMatrix(data,label=label)

• SavingDMatrix into a XGBoost binary file will make loading faster:

  dtrain=xgb.DMatrix('train.svm.txt?format=libsvm')dtrain.save_binary('train.buffer')

• Missing values can be replaced by a default value in theDMatrix constructor:

  dtrain=xgb.DMatrix(data,label=label,missing=np.NaN)

• Weights can be set when needed:

  w=np.random.rand(5,1)dtrain=xgb.DMatrix(data,label=label,missing=np.NaN,weight=w)

When performing ranking tasks, the number of weights should be equalto number of groups.

• To load a LIBSVM text file or a XGBoost binary file intoDMatrix:

  dtrain=xgb.DMatrix('train.svm.txt?format=libsvm')dtest=xgb.DMatrix('test.svm.buffer')

  The parser in XGBoost has limited functionality. When using Python interface, it’srecommended to use sklearnload_svmlight_file or other similar utilites thanXGBoost’s builtin parser.

• To load a CSV file intoDMatrix:

  # label_column specifies the index of the column containing the true labeldtrain=xgb.DMatrix('train.csv?format=csv&label_column=0')dtest=xgb.DMatrix('test.csv?format=csv&label_column=0')

  The parser in XGBoost has limited functionality. When using Python interface, it’srecommended to use pandasread_csv or other similar utilites than XGBoost’s builtinparser.

Setting Parameters

XGBoost can use either a list of pairs or a dictionary to setparameters. For instance:

• Booster parameters

  param={'max_depth':2,'eta':1,'objective':'binary:logistic'}param['nthread']=4param['eval_metric']='auc'

• You can also specify multiple eval metrics:

  param['eval_metric']=['auc','ams@0']# alternatively:# plst = param.items()# plst += [('eval_metric', 'ams@0')]

• Specify validations set to watch performance

  evallist=[(dtrain,'train'),(dtest,'eval')]

Training

Training a model requires a parameter list and data set.

num_round=10bst=xgb.train(param,dtrain,num_round,evallist)

After training, the model can be saved.

bst.save_model('0001.model')

The model and its feature map can also be dumped to a text file.

# dump modelbst.dump_model('dump.raw.txt')# dump model with feature mapbst.dump_model('dump.raw.txt','featmap.txt')

A saved model can be loaded as follows:

bst=xgb.Booster({'nthread':4})# init modelbst.load_model('model.bin')# load model data

Methods includingupdate andboost fromxgboost.Booster are designed forinternal usage only. The wrapper functionxgboost.train does somepre-configuration including setting up caches and some other parameters.

Early Stopping

If you have a validation set, you can use early stopping to find the optimal number of boosting rounds.Early stopping requires at least one set inevals. If there’s more than one, it will use the last.

train(...,evals=evals,early_stopping_rounds=10)

The model will train until the validation score stops improving. Validation error needs to decrease at least everyearly_stopping_rounds to continue training.

If early stopping occurs, the model will have two additional fields:bst.best_score,bst.best_iteration. Note thatxgboost.train() will return a model from the last iteration, not the best one.

This works with both metrics to minimize (RMSE, log loss, etc.) and to maximize (MAP, NDCG, AUC). Note that if you specify more than one evaluation metric the last one inparam['eval_metric'] is used for early stopping.

Prediction

A model that has been trained or loaded can perform predictions on data sets.

# 7 entities, each contains 10 featuresdata=np.random.rand(7,10)dtest=xgb.DMatrix(data)ypred=bst.predict(dtest)

If early stopping is enabled during training, you can get predictions from the best iteration withbst.best_iteration:

ypred=bst.predict(dtest,iteration_range=(0,bst.best_iteration+1))

Plotting

You can use plotting module to plot importance and output tree.

To plot importance, usexgboost.plot_importance(). This function requiresmatplotlib to be installed.

xgb.plot_importance(bst)

To plot the output tree viamatplotlib, usexgboost.plot_tree(), specifying the ordinal number of the target tree. This function requiresgraphviz andmatplotlib.

xgb.plot_tree(bst,num_trees=2)

When you useIPython, you can use thexgboost.to_graphviz() function, which converts the target tree to agraphviz instance. Thegraphviz instance is automatically rendered inIPython.

xgb.to_graphviz(bst,num_trees=2)

Scikit-Learn interface

XGBoost provides an easy to use scikit-learn interface for some pre-defined modelsincluding regression, classification and ranking. SeeUsing the Scikit-Learn Estimator Interfacefor more info.

# Use "hist" for training the model.reg=xgb.XGBRegressor(tree_method="hist",device="cuda")# Fit the model using predictor X and response y.reg.fit(X,y)# Save model into JSON format.reg.save_model("regressor.json")

User can still access the underlying booster model when needed:

booster:xgb.Booster=reg.get_booster()