Repository files navigation

This project hosts Python implementations of theBoruta all-relevant feature selection method.

Related blog post

Install withpip:

pip install Boruta

or withconda:

conda install -c conda-forge boruta_py

• numpy
• scipy
• scikit-learn

Download, import and do as you would with any other scikit-learn method:

• fit(X, y)
• transform(X)
• fit_transform(X, y)

Python implementations of the Boruta R package.

This implementation tries to mimic the scikit-learn interface, so use fit,transform or fit_transform, to run the feature selection.

For more, see the docs of these functions, and the examples below.

Original code and method was authored by Miron B. Kursa.

Boruta is an all relevant feature selection method, while most other areminimal optimal; this means it tries to find all features carryinginformation usable for prediction, rather than finding a possibly compactsubset of features on which some classifier has a minimal error.

Why bother with all relevant feature selection?When you try to understand the phenomenon that made your data, you shouldcare about all factors that contribute to it, not just the bluntest signsof it in context of your methodology (yes, minimal optimal set of featuresby definition depends on your classifier choice).

It is the original R package recoded in Python with a few added extra features.Some improvements include:

• Faster run times, thanks to scikit-learn

• Scikit-learn like interface

• Compatible with any ensemble method from scikit-learn

• Automatic n_estimator selection

• Ranking of features

• Feature importances are derived from Gini impurity instead of RandomForest R package's MDA.

For more details, please check the top of the docstring.

We highly recommend using pruned trees with a depth between 3-7.

Also, after playing around a lot with the original code I identified a few areaswhere the core algorithm could be improved/altered to make it less strict andmore applicable to biological data, where the Bonferroni correction might beoverly harsh.

Percentile as threshold
The original method uses the maximum of the shadow features as a threshold indeciding which real feature is doing better than the shadow ones. This could beoverly harsh.

To control this, I added the perc parameter, which sets thepercentile of the shadow features' importances, the algorithm uses as thethreshold. The default of 100 which is equivalent to taking the maximum as theR version of Boruta does, but it could be relaxed. Note, since this is thepercentile, it changes with the size of the dataset. With several thousands offeatures it isn't as stringent as with a few dozens at the end of a Boruta run.

Two step correction for multiple testing
The correction for multiple testing was relaxed by making it a two stepprocess, rather than a harsh one step Bonferroni correction.

We need to correct firstly because in each iteration we test a number offeatures against the null hypothesis (does a feature perform better thanexpected by random). For this the Bonferroni correction is used in the originalcode which is known to be too stringent in such scenarios (at least forbiological data), and also the original code corrects for n features, even ifwe are in the 50th iteration where we only have k<<n features left. For thisreason the first step of correction is the widely used Benjamini Hochberg FDR.

Following that however we also need to account for the fact that we have beentesting the same features over and over again in each iteration with thesame test. For this scenario the Bonferroni is perfect, so it is applied bydividing the p-value threshold with the current iteration index.

If this two step correction is not required, the two_step parameter has to beset to False, then (with perc=100) BorutaPy behaves exactly as the R version.

estimator : object

A supervised learning estimator, with a 'fit' method that returns thefeature_importances_ attribute. Important features must correspond tohigh absolute values in the feature_importances_.

n_estimators : int or string, default = 1000

If int sets the number of estimators in the chosen ensemble method.If 'auto' this is determined automatically based on the size of thedataset. The other parameters of the used estimators need to be setwith initialisation.

perc : int, default = 100

Instead of the max we use the percentile defined by the user, to pickour threshold for comparison between shadow and real features. The maxtends to be too stringent. This provides a finer control over this. Thelower perc is the more false positives will be picked as relevant butalso the less relevant features will be left out. The usual trade-off.The default is essentially the vanilla Boruta corresponding to the max.

alpha : float, default = 0.05

Level at which the corrected p-values will get rejected in both correctionsteps.

two_step : Boolean, default = True

If you want to use the original implementation of Boruta with Bonferronicorrection only set this to False.

max_iter : int, default = 100

The number of maximum iterations to perform.

verbose : int, default=0

Controls verbosity of output.

n_features_ : int

The number of selected features.

support_ : array of shape [n_features]

The mask of selected features - only confirmed ones are True.

support_weak_ : array of shape [n_features]

The mask of selected tentative features, which haven't gained enoughsupport during the max_iter number of iterations..

ranking_ : array of shape [n_features]

The feature ranking, such thatranking_[i] corresponds to theranking position of the i-th feature. Selected (i.e., estimatedbest) features are assigned rank 1 and tentative features are assignedrank 2.

importpandasaspdfromsklearn.ensembleimportRandomForestClassifierfromborutaimportBorutaPy# load X and y# NOTE BorutaPy accepts numpy arrays only, hence the .values attributeX=pd.read_csv('examples/test_X.csv',index_col=0).valuesy=pd.read_csv('examples/test_y.csv',header=None,index_col=0).valuesy=y.ravel()# define random forest classifier, with utilising all cores and# sampling in proportion to y labelsrf=RandomForestClassifier(n_jobs=-1,class_weight='balanced',max_depth=5)# define Boruta feature selection methodfeat_selector=BorutaPy(rf,n_estimators='auto',verbose=2,random_state=1)# find all relevant features - 5 features should be selectedfeat_selector.fit(X,y)# check selected features - first 5 features are selectedfeat_selector.support_# check ranking of featuresfeat_selector.ranking_# call transform() on X to filter it down to selected featuresX_filtered=feat_selector.transform(X)

1. Kursa M., Rudnicki W., "Feature Selection with the Boruta Package" Journal of Statistical Software, Vol. 36, Issue 11, Sep 2010