To run this example, install the following:pipinstall"ray[tune]".

In this quick-start example youminimize a simple function of the formf(x)=a**2+b, ourobjective function.The closera is to zero and the smallerb is, the smaller the total value off(x).We will define a so-calledsearchspace fora andb and let Ray Tune explore the space for good values.

fromrayimporttunedefobjective(config):# ①score=config["a"]**2+config["b"]return{"score":score}search_space={# ②"a":tune.grid_search([0.001,0.01,0.1,1.0]),"b":tune.choice([1,2,3]),}tuner=tune.Tuner(objective,param_space=search_space)# ③results=tuner.fit()print(results.get_best_result(metric="score",mode="min").config)

① Define an objective function.

② Define a search space.

③ Start a Tune run and print the best result.

To tune your Keras models with Hyperopt, you wrap your model in an objective function whoseconfig youcan access for selecting hyperparameters.In the example below we only tune theactivation parameter of the first layer of the model, but you cantune any parameter of the model you want.After defining the search space, you can simply initialize theHyperOptSearch object and pass it torun.It’s important to tell Ray Tune which metric you want to optimize and whether you want to maximize or minimize it.

fromrayimporttunefromray.tune.search.hyperoptimportHyperOptSearchimportkerasdefobjective(config):# ①model=keras.models.Sequential()model.add(keras.layers.Dense(784,activation=config["activation"]))model.add(keras.layers.Dense(10,activation="softmax"))model.compile(loss="binary_crossentropy",optimizer="adam",metrics=["accuracy"])# model.fit(...)# loss, accuracy = model.evaluate(...)return{"accuracy":accuracy}search_space={"activation":tune.choice(["relu","tanh"])}# ②algo=HyperOptSearch()tuner=tune.Tuner(# ③objective,tune_config=tune.TuneConfig(metric="accuracy",mode="max",search_alg=algo,),param_space=search_space,)results=tuner.fit()

① Wrap a Keras model in an objective function.

② Define a search space and initialize the search algorithm.

③ Start a Tune run that maximizes accuracy.

To tune your PyTorch models with Optuna, you wrap your model in an objective function whoseconfig youcan access for selecting hyperparameters.In the example below we only tune themomentum and learning rate (lr) parameters of the model’s optimizer,but you can tune any other model parameter you want.After defining the search space, you can simply initialize theOptunaSearch object and pass it torun.It’s important to tell Ray Tune which metric you want to optimize and whether you want to maximize or minimize it.We stop tuning this training run after5 iterations, but you can easily define other stopping rules as well.

importtorchfromrayimporttunefromray.tune.search.optunaimportOptunaSearchdefobjective(config):# ①train_loader,test_loader=load_data()# Load some datamodel=ConvNet().to("cpu")# Create a PyTorch conv netoptimizer=torch.optim.SGD(# Tune the optimizermodel.parameters(),lr=config["lr"],momentum=config["momentum"])whileTrue:train_epoch(model,optimizer,train_loader)# Train the modelacc=test(model,test_loader)# Compute test accuracytune.report({"mean_accuracy":acc})# Report to Tunesearch_space={"lr":tune.loguniform(1e-4,1e-2),"momentum":tune.uniform(0.1,0.9)}algo=OptunaSearch()# ②tuner=tune.Tuner(# ③objective,tune_config=tune.TuneConfig(metric="mean_accuracy",mode="max",search_alg=algo,),run_config=tune.RunConfig(stop={"training_iteration":5},),param_space=search_space,)results=tuner.fit()print("Best config is:",results.get_best_result().config)

① Wrap a PyTorch model in an objective function.

② Define a search space and initialize the search algorithm.

③ Start a Tune run that maximizes mean accuracy and stops after 5 iterations.