|
9 | 9 | "\n", |
10 | 10 | "_Authored by: [Parag Ekbote](https://github.com/ParagEkbote)_\n", |
11 | 11 | "\n", |
12 | | -"In this notebook, we are going\n", |
13 | | -"\n" |
| 12 | +"In this notebook, we are going to use the [optuna](https://github.com/optuna/optuna) library to perform hyperparameter optimization on a light-weight BERT model on a small subset of the IMDB dataset. To learn more about transformers' hyperparameter search, you can check the following documentation [here](https://huggingface.co/docs/transformers/en/hpo_train).\n", |
| 13 | +"\n", |
| 14 | +"Firstly, we will install the following dependencies to ensure that our code is executed:" |
14 | 15 | ] |
15 | 16 | }, |
16 | 17 | { |
17 | | -"cell_type":"markdown", |
18 | | -"id":"3612d3e9", |
| 18 | +"cell_type":"code", |
| 19 | +"execution_count":null, |
| 20 | +"id":"a309e1a0", |
19 | 21 | "metadata": {}, |
| 22 | +"outputs": [], |
20 | 23 | "source": [ |
21 | 24 | "!pip install -q datasets evaluate transformers" |
22 | 25 | ] |
|
72 | 75 | "id":"5a46fac4", |
73 | 76 | "metadata": {}, |
74 | 77 | "source": [ |
75 | | -"# Set the Metrics and define the model" |
| 78 | +"# Set the Metrics and define the Trainer class\n", |
| 79 | +"\n" |
76 | 80 | ] |
77 | 81 | }, |
78 | 82 | { |
|
116 | 120 | "id":"b10c26c6", |
117 | 121 | "metadata": {}, |
118 | 122 | "source": [ |
119 | | -"# Define the Search Space and Start the Trials" |
| 123 | +"# Define the Search Space and Start the Trials\n", |
| 124 | +"\n", |
| 125 | +"We will now define the optuna hyperparameter search space to find the best set of hyperparameters for the learning rate and batch size. We can now launch the hyperparameter search by passing the following metrics:\n", |
| 126 | +"\n", |
| 127 | +"1. direction: We aim to maxime the evaluation metric\n", |
| 128 | +"2. backend: We will use optuna for searching\n", |
| 129 | +"3. n_trials: The number of trials optuna will be executed\n", |
| 130 | +"4. compute_objective: THe objective to minimize or maximize from the metrics returned by `evaluate`" |
120 | 131 | ] |
121 | 132 | }, |
122 | 133 | { |
|
139 | 150 | " direction=\"maximize\",\n", |
140 | 151 | " backend=\"optuna\",\n", |
141 | 152 | " hp_space=optuna_hp_space,\n", |
142 | | -" n_trials=5,\n", |
| 153 | +" n_trials=20,\n", |
143 | 154 | " compute_objective=compute_objective,\n", |
144 | 155 | ")\n", |
145 | 156 | "\n", |
|