1.11.1.1.1.Usage#

Most of the parameters are unchanged fromGradientBoostingClassifier andGradientBoostingRegressor.One exception is themax_iter parameter that replacesn_estimators, andcontrols the number of iterations of the boosting process:

>>>fromsklearn.ensembleimportHistGradientBoostingClassifier>>>fromsklearn.datasetsimportmake_hastie_10_2>>>X,y=make_hastie_10_2(random_state=0)>>>X_train,X_test=X[:2000],X[2000:]>>>y_train,y_test=y[:2000],y[2000:]>>>clf=HistGradientBoostingClassifier(max_iter=100).fit(X_train,y_train)>>>clf.score(X_test,y_test)0.8965

Available losses forregression are:

• ‘squared_error’, which is the default loss;

• ‘absolute_error’, which is less sensitive to outliers than the squared error;

• ‘gamma’, which is well suited to model strictly positive outcomes;

• ‘poisson’, which is well suited to model counts and frequencies;

• ‘quantile’, which allows for estimating a conditional quantile that can laterbe used to obtain prediction intervals.

Forclassification, ‘log_loss’ is the only option. For binary classificationit uses the binary log loss, also known as binomial deviance or binarycross-entropy. Forn_classes>=3, it uses the multi-class log loss function,with multinomial deviance and categorical cross-entropy as alternative names.The appropriate loss version is selected based ony passed tofit.

The size of the trees can be controlled through themax_leaf_nodes,max_depth, andmin_samples_leaf parameters.

The number of bins used to bin the data is controlled with themax_binsparameter. Using less bins acts as a form of regularization. It is generallyrecommended to use as many bins as possible (255), which is the default.

Thel2_regularization parameter acts as a regularizer for the loss function,and corresponds to\(\lambda\) in the following expression (see equation (2)in[XGBoost]):

Note thatearly-stopping is enabled by default if the number of samples islarger than 10,000. The early-stopping behaviour is controlled via theearly_stopping,scoring,validation_fraction,n_iter_no_change, andtol parameters. It is possible to early-stopusing an arbitraryscorer, or just the training or validation loss.Note that for technical reasons, using a callable as a scorer is significantly slowerthan using the loss. By default, early-stopping is performed if there are at least10,000 samples in the training set, using the validation loss.

1.11.1.1.2.Missing values support#

HistGradientBoostingClassifier andHistGradientBoostingRegressor have built-in support for missingvalues (NaNs).

During training, the tree grower learns at each split point whether sampleswith missing values should go to the left or right child, based on thepotential gain. When predicting, samples with missing values are assigned tothe left or right child consequently:

>>>fromsklearn.ensembleimportHistGradientBoostingClassifier>>>importnumpyasnp>>>X=np.array([0,1,2,np.nan]).reshape(-1,1)>>>y=[0,0,1,1]>>>gbdt=HistGradientBoostingClassifier(min_samples_leaf=1).fit(X,y)>>>gbdt.predict(X)array([0, 0, 1, 1])

When the missingness pattern is predictive, the splits can be performed onwhether the feature value is missing or not:

>>>X=np.array([0,np.nan,1,2,np.nan]).reshape(-1,1)>>>y=[0,1,0,0,1]>>>gbdt=HistGradientBoostingClassifier(min_samples_leaf=1,...max_depth=2,...learning_rate=1,...max_iter=1).fit(X,y)>>>gbdt.predict(X)array([0, 1, 0, 0, 1])

If no missing values were encountered for a given feature during training,then samples with missing values are mapped to whichever child has the mostsamples.

Examples

• Features in Histogram Gradient Boosting Trees

1.11.1.1.3.Sample weight support#

HistGradientBoostingClassifier andHistGradientBoostingRegressor support sample weights duringfit.

The following toy example demonstrates that samples with a sample weight of zero are ignored:

>>>X=[[1,0],...[1,0],...[1,0],...[0,1]]>>>y=[0,0,1,0]>>># ignore the first 2 training samples by setting their weight to 0>>>sample_weight=[0,0,1,1]>>>gb=HistGradientBoostingClassifier(min_samples_leaf=1)>>>gb.fit(X,y,sample_weight=sample_weight)HistGradientBoostingClassifier(...)>>>gb.predict([[1,0]])array([1])>>>gb.predict_proba([[1,0]])[0,1]np.float64(0.999)

As you can see, the[1,0] is comfortably classified as1 since the firsttwo samples are ignored due to their sample weights.

Implementation detail: taking sample weights into account amounts tomultiplying the gradients (and the hessians) by the sample weights. Note thatthe binning stage (specifically the quantiles computation) does not take theweights into account.

1.11.1.1.4.Categorical Features Support#

HistGradientBoostingClassifier andHistGradientBoostingRegressor have native support for categoricalfeatures: they can consider splits on non-ordered, categorical data.

For datasets with categorical features, using the native categorical supportis often better than relying on one-hot encoding(OneHotEncoder), because one-hot encodingrequires more tree depth to achieve equivalent splits. It is also usuallybetter to rely on the native categorical support rather than to treatcategorical features as continuous (ordinal), which happens for ordinal-encodedcategorical data, since categories are nominal quantities where order does notmatter.

To enable categorical support, a boolean mask can be passed to thecategorical_features parameter, indicating which feature is categorical. Inthe following, the first feature will be treated as categorical and thesecond feature as numerical:

>>>gbdt=HistGradientBoostingClassifier(categorical_features=[True,False])

Equivalently, one can pass a list of integers indicating the indices of thecategorical features:

>>>gbdt=HistGradientBoostingClassifier(categorical_features=[0])

When the input is a DataFrame, it is also possible to pass a list of columnnames:

>>>gbdt=HistGradientBoostingClassifier(categorical_features=["site","manufacturer"])

Finally, when the input is a DataFrame we can usecategorical_features="from_dtype" in which case all columns with a categoricaldtype will be treated as categorical features.

The cardinality of each categorical feature must be less than themax_binsparameter. For an example using histogram-based gradient boosting on categoricalfeatures, seeCategorical Feature Support in Gradient Boosting.

If there are missing values during training, the missing values will betreated as a proper category. If there are no missing values during training,then at prediction time, missing values are mapped to the child node that hasthe most samples (just like for continuous features). When predicting,categories that were not seen during fit time will be treated as missingvalues.

Examples

• Categorical Feature Support in Gradient Boosting

1.11.1.1.5.Monotonic Constraints#

Depending on the problem at hand, you may have prior knowledge indicatingthat a given feature should in general have a positive (or negative) effecton the target value. For example, all else being equal, a higher creditscore should increase the probability of getting approved for a loan.Monotonic constraints allow you to incorporate such prior knowledge into themodel.

For a predictor\(F\) with two features:

• amonotonic increase constraint is a constraint of the form:

  \[x_1 \leq x_1' \implies F(x_1, x_2) \leq F(x_1', x_2)\]

• amonotonic decrease constraint is a constraint of the form:

  \[x_1 \leq x_1' \implies F(x_1, x_2) \geq F(x_1', x_2)\]

You can specify a monotonic constraint on each feature using themonotonic_cst parameter. For each feature, a value of 0 indicates noconstraint, while 1 and -1 indicate a monotonic increase andmonotonic decrease constraint, respectively:

>>>fromsklearn.ensembleimportHistGradientBoostingRegressor... # monotonic increase, monotonic decrease, and no constraint on the 3 features>>>gbdt=HistGradientBoostingRegressor(monotonic_cst=[1,-1,0])

In a binary classification context, imposing a monotonic increase (decrease) constraint means that higher values of the feature are supposedto have a positive (negative) effect on the probability of samplesto belong to the positive class.

Nevertheless, monotonic constraints only marginally constrain feature effects on the output.For instance, monotonic increase and decrease constraints cannot be used to enforce thefollowing modelling constraint:

Also, monotonic constraints are not supported for multiclass classification.

For a practical implementation of monotonic constraints with the histogram-basedgradient boosting, including how they can improve generalization when domain knowledgeis available, seeMonotonic Constraints.

Examples

• Features in Histogram Gradient Boosting Trees

1.11.1.1.6.Interaction constraints#

A priori, the histogram gradient boosted trees are allowed to use any featureto split a node into child nodes. This creates so called interactions betweenfeatures, i.e. usage of different features as split along a branch. Sometimes,one wants to restrict the possible interactions, see[Mayer2022]. This can bedone by the parameterinteraction_cst, where one can specify the indicesof features that are allowed to interact.For instance, with 3 features in total,interaction_cst=[{0},{1},{2}]forbids all interactions.The constraints[{0,1},{1,2}] specify two groups of possiblyinteracting features. Features 0 and 1 may interact with each other, as wellas features 1 and 2. But note that features 0 and 2 are forbidden to interact.The following depicts a tree and the possible splits of the tree:

   1      <- Both constraint groups could be applied from now on  / \ 1   2    <- Left split still fulfills both constraint groups./ \ / \      Right split at feature 2 has only group {1, 2} from now on.

LightGBM uses the same logic for overlapping groups.

Note that features not listed ininteraction_cst are automaticallyassigned an interaction group for themselves. With again 3 features, thismeans that[{0}] is equivalent to[{0},{1,2}].

Examples

• Partial Dependence and Individual Conditional Expectation Plots

References

1.11.1.1.7.Low-level parallelism#

HistGradientBoostingClassifier andHistGradientBoostingRegressor use OpenMPfor parallelization through Cython. For more details on how to control thenumber of threads, please refer to ourParallelism notes.

The following parts are parallelized:

• mapping samples from real values to integer-valued bins (finding the binthresholds is however sequential)

• building histograms is parallelized over features

• finding the best split point at a node is parallelized over features

• during fit, mapping samples into the left and right children isparallelized over samples

• gradient and hessians computations are parallelized over samples

• predicting is parallelized over samples

1.11.1.1.8.Why it’s faster#

The bottleneck of a gradient boosting procedure is building the decisiontrees. Building a traditional decision tree (as in the other GBDTsGradientBoostingClassifier andGradientBoostingRegressor)requires sorting the samples at each node (foreach feature). Sorting is needed so that the potential gain of a split pointcan be computed efficiently. Splitting a single node has thus a complexityof\(\mathcal{O}(n_\text{features} \times n \log(n))\) where\(n\)is the number of samples at the node.

HistGradientBoostingClassifier andHistGradientBoostingRegressor, in contrast, do not require sorting thefeature values and instead use a data-structure called a histogram, where thesamples are implicitly ordered. Building a histogram has a\(\mathcal{O}(n)\) complexity, so the node splitting procedure has a\(\mathcal{O}(n_\text{features} \times n)\) complexity, much smallerthan the previous one. In addition, instead of considering\(n\) splitpoints, we consider onlymax_bins split points, which might be muchsmaller.

In order to build histograms, the input dataX needs to be binned intointeger-valued bins. This binning procedure does require sorting the featurevalues, but it only happens once at the very beginning of the boosting process(not at each node, like inGradientBoostingClassifier andGradientBoostingRegressor).

Finally, many parts of the implementation ofHistGradientBoostingClassifier andHistGradientBoostingRegressor are parallelized.

References

1.11.1.2.1.Fitting additional weak-learners#

BothGradientBoostingRegressor andGradientBoostingClassifiersupportwarm_start=True which allows you to add more estimators to an alreadyfitted model.

>>>importnumpyasnp>>>fromsklearn.metricsimportmean_squared_error>>>fromsklearn.datasetsimportmake_friedman1>>>fromsklearn.ensembleimportGradientBoostingRegressor>>>X,y=make_friedman1(n_samples=1200,random_state=0,noise=1.0)>>>X_train,X_test=X[:200],X[200:]>>>y_train,y_test=y[:200],y[200:]>>>est=GradientBoostingRegressor(...n_estimators=100,learning_rate=0.1,max_depth=1,random_state=0,...loss='squared_error'...)>>>est=est.fit(X_train,y_train)# fit with 100 trees>>>mean_squared_error(y_test,est.predict(X_test))5.00>>>_=est.set_params(n_estimators=200,warm_start=True)# set warm_start and increase num of trees>>>_=est.fit(X_train,y_train)# fit additional 100 trees to est>>>mean_squared_error(y_test,est.predict(X_test))3.84

1.11.1.2.2.Controlling the tree size#

The size of the regression tree base learners defines the level of variableinteractions that can be captured by the gradient boosting model. In general,a tree of depthh can capture interactions of orderh .There are two ways in which the size of the individual regression trees canbe controlled.

If you specifymax_depth=h then complete binary treesof depthh will be grown. Such trees will have (at most)2**h leaf nodesand2**h-1 split nodes.

Alternatively, you can control the tree size by specifying the number ofleaf nodes via the parametermax_leaf_nodes. In this case,trees will be grown using best-first search where nodes with the highest improvementin impurity will be expanded first.A tree withmax_leaf_nodes=k hask-1 split nodes and thus canmodel interactions of up to ordermax_leaf_nodes-1 .

We found thatmax_leaf_nodes=k gives comparable results tomax_depth=k-1but is significantly faster to train at the expense of a slightly highertraining error.The parametermax_leaf_nodes corresponds to the variableJ in thechapter on gradient boosting in[Friedman2001] and is related to the parameterinteraction.depth in R’s gbm package wheremax_leaf_nodes==interaction.depth+1 .

1.11.1.2.3.Mathematical formulation#

We first present GBRT for regression, and then detail the classificationcase.

1.11.1.2.4.Loss Functions#

The following loss functions are supported and can be specified usingthe parameterloss:

1.11.1.2.5.Shrinkage via learning rate#

[Friedman2001] proposed a simple regularization strategy that scalesthe contribution of each weak learner by a constant factor\(\nu\):

The parameter\(\nu\) is also called thelearning rate becauseit scales the step length of the gradient descent procedure; it canbe set via thelearning_rate parameter.

The parameterlearning_rate strongly interacts with the parametern_estimators, the number of weak learners to fit. Smaller valuesoflearning_rate require larger numbers of weak learners to maintaina constant training error. Empirical evidence suggests that smallvalues oflearning_rate favor better test error.[HTF]recommend to set the learning rate to a small constant(e.g.learning_rate<=0.1) and choosen_estimators large enoughthat early stopping applies,seeEarly stopping in Gradient Boostingfor a more detailed discussion of the interaction betweenlearning_rate andn_estimators see[R2007].

1.11.1.2.6.Subsampling#

[Friedman2002] proposed stochastic gradient boosting, which combines gradientboosting with bootstrap averaging (bagging). At each iterationthe base classifier is trained on a fractionsubsample ofthe available training data. The subsample is drawn without replacement.A typical value ofsubsample is 0.5.

The figure below illustrates the effect of shrinkage and subsamplingon the goodness-of-fit of the model. We can clearly see that shrinkageoutperforms no-shrinkage. Subsampling with shrinkage can further increasethe accuracy of the model. Subsampling without shrinkage, on the other hand,does poorly.

Another strategy to reduce the variance is by subsampling the featuresanalogous to the random splits inRandomForestClassifier.The number of subsampled features can be controlled via themax_featuresparameter.

Stochastic gradient boosting allows to compute out-of-bag estimates of thetest deviance by computing the improvement in deviance on the examples that arenot included in the bootstrap sample (i.e. the out-of-bag examples).The improvements are stored in the attributeoob_improvement_.oob_improvement_[i] holds the improvement in terms of the loss on the OOB samplesif you add the i-th stage to the current predictions.Out-of-bag estimates can be used for model selection, for example to determinethe optimal number of iterations. OOB estimates are usually very pessimistic thuswe recommend to use cross-validation instead and only use OOB if cross-validationis too time consuming.

Examples

• Gradient Boosting regularization

• Gradient Boosting Out-of-Bag estimates

• OOB Errors for Random Forests

1.11.1.2.7.Interpretation with feature importance#

Individual decision trees can be interpreted easily by simplyvisualizing the tree structure. Gradient boosting models, however,comprise hundreds of regression trees thus they cannot be easilyinterpreted by visual inspection of the individual trees. Fortunately,a number of techniques have been proposed to summarize and interpretgradient boosting models.

Often features do not contribute equally to predict the targetresponse; in many situations the majority of the features are in factirrelevant.When interpreting a model, the first question usually is: what arethose important features and how do they contribute in predictingthe target response?

Individual decision trees intrinsically perform feature selection by selectingappropriate split points. This information can be used to measure theimportance of each feature; the basic idea is: the more often afeature is used in the split points of a tree the more important thatfeature is. This notion of importance can be extended to decision treeensembles by simply averaging the impurity-based feature importance of each tree (seeFeature importance evaluation for more details).

The feature importance scores of a fit gradient boosting model can beaccessed via thefeature_importances_ property:

>>>fromsklearn.datasetsimportmake_hastie_10_2>>>fromsklearn.ensembleimportGradientBoostingClassifier>>>X,y=make_hastie_10_2(random_state=0)>>>clf=GradientBoostingClassifier(n_estimators=100,learning_rate=1.0,...max_depth=1,random_state=0).fit(X,y)>>>clf.feature_importances_array([0.107, 0.105, 0.113, 0.0987, 0.0947,       0.107, 0.0916, 0.0972, 0.0958, 0.0906])

Note that this computation of feature importance is based on entropy, and itis distinct fromsklearn.inspection.permutation_importance which isbased on permutation of the features.

Examples

• Gradient Boosting regression

References