Analysis of hyperparameter search results#

In the previous notebook we showed how to implement a randomized search fortuning the hyperparameters of aHistGradientBoostingClassifier to fit theadult_census dataset. In practice, a randomized hyperparameter search isusually run with a large number of iterations.

In order to avoid the computational cost and still make a decent analysis, weload the results obtained from a similar search with 500 iterations.

importpandasaspdcv_results=pd.read_csv("../figures/randomized_search_results.csv",index_col=0)cv_results

We define a function to remove the prefixes in the hyperparameters columnnames.

defshorten_param(param_name):if"__"inparam_name:returnparam_name.rsplit("__",1)[1]returnparam_namecv_results=cv_results.rename(shorten_param,axis=1)cv_results

As we have more than 2 parameters in our randomized-search, we cannotvisualize the results using a heatmap. We could still do it pair-wise, buthaving a two-dimensional projection of a multi-dimensional problem can lead toa wrong interpretation of the scores.

importseabornassnsimportnumpyasnpdf=pd.DataFrame({"max_leaf_nodes":cv_results["max_leaf_nodes"],"learning_rate":cv_results["learning_rate"],"score_bin":pd.cut(cv_results["mean_test_score"],bins=np.linspace(0.5,1.0,6)),})sns.set_palette("YlGnBu_r")ax=sns.scatterplot(data=df,x="max_leaf_nodes",y="learning_rate",hue="score_bin",s=50,color="k",edgecolor=None,)ax.set_xscale("log")ax.set_yscale("log")_=ax.legend(title="mean_test_score",loc="center left",bbox_to_anchor=(1,0.5))

In the previous plot we see that the top performing values are located in aband of learning rate between 0.01 and 1.0, but we have no control in how theother hyperparameters interact with such values for the learning rate.Instead, we can visualize all the hyperparameters at the same time using aparallel coordinates plot.

importnumpyasnpimportplotly.expressaspxfig=px.parallel_coordinates(cv_results.rename(shorten_param,axis=1).apply({"learning_rate":np.log10,"max_leaf_nodes":np.log2,"max_bins":np.log2,"min_samples_leaf":np.log10,"l2_regularization":np.log10,"mean_test_score":lambdax:x,}),color="mean_test_score",color_continuous_scale=px.colors.sequential.Viridis,)fig.show()

The parallel coordinates plot displays the values of the hyperparameters ondifferent columns while the performance metric is color coded. Thus, we areable to quickly inspect if there is a range of hyperparameters which isworking or not.

It is possible toselect a range of results by clicking and holding on anyaxis of the parallel coordinate plot. You can then slide (move) the rangeselection and cross two selections to see the intersections. You can undo aselection by clicking once again on the same axis.

In particular for this hyperparameter search, it is interesting to confirmthat the yellow lines (top performing models) all reach intermediate valuesfor the learning rate, that is, tick values between -2 and 0 which correspondto learning rate values of 0.01 to 1.0 once we invert back the log10 transformfor that axis.

But now we can also observe that it is not possible to select the highestperforming models by selecting lines of on themax_bins axis with tickvalues between 1 and 3.

The other hyperparameters are not very sensitive. We can check that if weselect thelearning_rate axis tick values between -1.5 and -0.5 andmax_bins tick values between 5 and 8, we always select top performingmodels, whatever the values of the other hyperparameters.

In this notebook, we saw how to interactively explore the results of a largerandomized search with multiple interacting hyperparameters. In particular weobserved that some hyperparameters have very little impact on thecross-validation score, while others have to be adjusted within a specificrange to get models with good predictive accuracy.