• SeeNested versus non-nested cross-validationfor an example of Grid Search within a cross validation loop on the irisdataset. This is the best practice for evaluating the performance of amodel with grid search.

• SeeDemonstration of multi-metric evaluation on cross_val_score and GridSearchCVfor an example ofGridSearchCV being used to evaluate multiplemetrics simultaneously.

• SeeBalance model complexity and cross-validated scorefor an example of usingrefit=callable interface inGridSearchCV. The example shows how this interface adds a certainamount of flexibility in identifying the “best” estimator. This interfacecan also be used in multiple metrics evaluation.

• SeeStatistical comparison of models using grid searchfor an example of how to do a statistical comparison on the outputs ofGridSearchCV.

Besidefactor, the two main parameters that influence the behaviour of asuccessive halving search are themin_resources parameter, and thenumber of candidates (or parameter combinations) that are evaluated.min_resources is the amount of resources allocated at the firstiteration for each candidate. The number of candidates is specified directlyinHalvingRandomSearchCV, and is determined from theparam_gridparameter ofHalvingGridSearchCV.

Consider a case where the resource is the number of samples, and where wehave 1000 samples. In theory, withmin_resources=10 andfactor=2, weare able to runat most 7 iterations with the following number ofsamples:[10,20,40,80,160,320,640].

But depending on the number of candidates, we might run less than 7iterations: if we start with asmall number of candidates, the lastiteration might use less than 640 samples, which means not using all theavailable resources (samples). For example if we start with 5 candidates, weonly need 2 iterations: 5 candidates for the first iteration, then5//2=2 candidates at the second iteration, after which we know whichcandidate performs the best (so we don’t need a third one). We would only beusing at most 20 samples which is a waste since we have 1000 samples at ourdisposal. On the other hand, if we start with ahigh number ofcandidates, we might end up with a lot of candidates at the last iteration,which may not always be ideal: it means that many candidates will run withthe full resources, basically reducing the procedure to standard search.

In the case ofHalvingRandomSearchCV, the number of candidates is setby default such that the last iteration uses as much of the availableresources as possible. ForHalvingGridSearchCV, the number ofcandidates is determined by theparam_grid parameter. Changing the value ofmin_resources will impact the number of possible iterations, and as aresult will also have an effect on the ideal number of candidates.

Another consideration when choosingmin_resources is whether or not itis easy to discriminate between good and bad candidates with a small amountof resources. For example, if you need a lot of samples to distinguishbetween good and bad parameters, a highmin_resources is recommended. Onthe other hand if the distinction is clear even with a small amount ofsamples, then a smallmin_resources may be preferable since it wouldspeed up the computation.

Notice in the example above that the last iteration does not use the maximumamount of resources available: 1000 samples are available, yet only 640 areused, at most. By default, bothHalvingRandomSearchCV andHalvingGridSearchCV try to use as many resources as possible in thelast iteration, with the constraint that this amount of resources must be amultiple of bothmin_resources andfactor (this constraint will be clearin the next section).HalvingRandomSearchCV achieves this bysampling the right amount of candidates, whileHalvingGridSearchCVachieves this by properly settingmin_resources.

At any iterationi, each candidate is allocated a given amount of resourceswhich we denoten_resources_i. This quantity is controlled by theparametersfactor andmin_resources as follows (factor is strictlygreater than 1):

n_resources_i=factor**i*min_resources,

or equivalently:

n_resources_{i+1}=n_resources_i*factor

wheremin_resources==n_resources_0 is the amount of resources used atthe first iteration.factor also defines the proportions of candidatesthat will be selected for the next iteration:

n_candidates_i=n_candidates//(factor**i)

or equivalently:

n_candidates_0=n_candidatesn_candidates_{i+1}=n_candidates_i//factor

So in the first iteration, we usemin_resources resourcesn_candidates times. In the second iteration, we usemin_resources*factor resourcesn_candidates//factor times. The third againmultiplies the resources per candidate and divides the number of candidates.This process stops when the maximum amount of resource per candidate isreached, or when we have identified the best candidate. The best candidateis identified at the iteration that is evaluatingfactor or less candidates(see just below for an explanation).

Here is an example withmin_resources=3 andfactor=2, starting with70 candidates:

We can note that:

• the process stops at the first iteration which evaluatesfactor=2candidates: the best candidate is the best out of these 2 candidates. Itis not necessary to run an additional iteration, since it would onlyevaluate one candidate (namely the best one, which we have alreadyidentified). For this reason, in general, we want the last iteration torun at mostfactor candidates. If the last iteration evaluates morethanfactor candidates, then this last iteration reduces to a regularsearch (as inRandomizedSearchCV orGridSearchCV).

• eachn_resources_i is a multiple of bothfactor andmin_resources (which is confirmed by its definition above).

The amount of resources that is used at each iteration can be found in then_resources_ attribute.

By default, the resource is defined in terms of number of samples. That is,each iteration will use an increasing amount of samples to train on. You canhowever manually specify a parameter to use as the resource with theresource parameter. Here is an example where the resource is defined interms of the number of estimators of a random forest:

>>>fromsklearn.datasetsimportmake_classification>>>fromsklearn.ensembleimportRandomForestClassifier>>>fromsklearn.experimentalimportenable_halving_search_cv# noqa>>>fromsklearn.model_selectionimportHalvingGridSearchCV>>>importpandasaspd>>>param_grid={'max_depth':[3,5,10],...'min_samples_split':[2,5,10]}>>>base_estimator=RandomForestClassifier(random_state=0)>>>X,y=make_classification(n_samples=1000,random_state=0)>>>sh=HalvingGridSearchCV(base_estimator,param_grid,cv=5,...factor=2,resource='n_estimators',...max_resources=30).fit(X,y)>>>sh.best_estimator_RandomForestClassifier(max_depth=5, n_estimators=24, random_state=0)

Note that it is not possible to budget on a parameter that is part of theparameter grid.

As mentioned above, the number of resources that is used at each iterationdepends on themin_resources parameter.If you have a lot of resources available but start with a low number ofresources, some of them might be wasted (i.e. not used):

>>>fromsklearn.datasetsimportmake_classification>>>fromsklearn.svmimportSVC>>>fromsklearn.experimentalimportenable_halving_search_cv# noqa>>>fromsklearn.model_selectionimportHalvingGridSearchCV>>>importpandasaspd>>>param_grid={'kernel':('linear','rbf'),...'C':[1,10,100]}>>>base_estimator=SVC(gamma='scale')>>>X,y=make_classification(n_samples=1000)>>>sh=HalvingGridSearchCV(base_estimator,param_grid,cv=5,...factor=2,min_resources=20).fit(X,y)>>>sh.n_resources_[20, 40, 80]

The search process will only use 80 resources at most, while our maximumamount of available resources isn_samples=1000. Here, we havemin_resources=r_0=20.

ForHalvingGridSearchCV, by default, themin_resources parameteris set to ‘exhaust’. This means thatmin_resources is automatically setsuch that the last iteration can use as many resources as possible, withinthemax_resources limit:

>>>sh=HalvingGridSearchCV(base_estimator,param_grid,cv=5,...factor=2,min_resources='exhaust').fit(X,y)>>>sh.n_resources_[250, 500, 1000]

min_resources was here automatically set to 250, which results in the lastiteration using all the resources. The exact value that is used depends onthe number of candidate parameters, onmax_resources and onfactor.

ForHalvingRandomSearchCV, exhausting the resources can be done in 2ways:

• by settingmin_resources='exhaust', just like forHalvingGridSearchCV;

• by settingn_candidates='exhaust'.

Both options are mutually exclusive: usingmin_resources='exhaust' requiresknowing the number of candidates, and symmetricallyn_candidates='exhaust'requires knowingmin_resources.

In general, exhausting the total number of resources leads to a better finalcandidate parameter, and is slightly more time-intensive.