Keras Tuner: Hyperparameters Tuning/Optimization of Keras Models¶

When designing a deep learning model, there are many decisions that need to be made and we don't know the answer to many of them upfront. The common questions like

• Which activation function to use (relu or tanh or elu or selu?)
• How many layers to keep in the network?
• How many units per layer?
• Which weights and bias initialization to use (normal, lecun-normal, he-initialization, etc)?
• Whether to use regularization?
• Which regularization to use (l1, l2, elasticnet, etc)?
• Whether to use bias or not?
• How much learning rate to use?
• Which optimizer to use (SGD, Adam, Adagrad, etc)?
• and many more.

There are some common standards like using relu activation and Adam optimizer gives good results. But this is not always true. There are not 100% right answers to the above questions for any given problem. These are commonly referred to as hyperparameters for which we need to make decisions. One of the solutions is to try all possible combinations of these hyperparameters to see which one works best. Though this solution seems viable, in reality, deep learning models have a lot of data and can require a lot of time to train hence grid searching through all possible combinations might not be a feasible solution. There are other algorithms like random search, hyperband, and Bayesian optimization which we have covered in this tutorial.

As a part of this tutorial, we'll be explaining how we can use Keras Tuner library to optimize the hyperparameters of networks designed by Python deep learning library keras. The keras tuner library provides an implementation of algorithms like random search, hyperband, and bayesian optimization for hyperparameters tuning. These algorithms find good hyperparameters settings in less number of trials without trying all possible combinations. They search for hyperparameters in the direction that is giving good results. We have explained step by step guide to hyperparameters optimization with simple examples using a keras tuner.

Below, we have listed important sections of tutorial to give an overview of the material covered.

Important Sections Of Tutorial¶

1. Regression Example (Random Hyperparameters Search)
2. Classification Example (Random Hyperparameters Search)
3. Override Compile Arguments
4. Override Existing Hyperparameters Search Space
5. Fixing Few Hyperparameters
6. Hyperband Algorithm
7. Bayesian Optimization

Installation¶

• pip install -U keras_tuner

Below, we have imported the necessary libraries and printed the versions that we have used in our tutorial.

1. Regression Example (Random Hyperparameters Search) ¶

In our first example, we'll explain how we can use a keras tuner for regression tasks. We have loaded the Boston housing dataset available from datasets module of keras below.

In order to use the keras tuner, we need to design a function that takes as input a single parameter and returns a compiled keras model. The single input parameter is an instance of HyperParameters that has information about values of various hyperparameters that we want to tune. The HyperParameters instance has various methods that can be used to try different values for a particular type of hyperparameter. These methods let us provide values of different types like boolean, integer, list of strings, etc.

In our case below, we have created a neural network of 3 dense layers. For the first two dense layers, we want to try different values of units, use_bias, and activation hyperparameters. The last dense layer has one output unit which will be a prediction of our network. Then, we also want to try different optimizers to select the one that gives the best results. In order to try different values of hyperparameters, we have used various methods available from HyperParameters class. Below, we have explained the commonly used methods.

Important Methods of 'Hyperparameters' Object¶

• Boolean(name, default=False) - This method let us provide boolean values for hyperparameter. It accept name as first parameter and default value is False. It'll try True and False values.
• Choice(name, values, ordered=None, default=None) - This method let us provide list of values to choose from for hyperparameter. The first value is the name of the hyperparameter and the second value is a list of values from which to try different values. The ordered parameter accepts boolean value specifying whether
• Float(name,min_value,max_value,step=None,sampling=None,default=None) - This method let us try different float values for the hyperparameter. It let us try values in the range [min value, max value]. It has few optional parameters as mentioned below which can be useful.
  • step - We can specify another small float value which is the minimum distance between two float values. Let's say that if we provide a value of 0.1 then different float values tried for the parameter will be at least 0.1 apart from one another.
  • sampling - This parameter accepts one of the below string specifying value sampling strategy.
    • 'linear'
    • 'log'
    • 'reverse_log'
• Int(name,min_value,max_value,step=1,sampling=None,default=None) - This method let us try different integer values for given hyperparameter. It has same parameters as that of Float() method with only difference that it'll try integer values instead.

In our case below, we have asked to try various units of dense layers using Int() method. We have asked to try values in the range [16,50] with a value increment of 16 hence it'll try values 16,32 and 48. We have given different names for hyperparameters of both layers. For use_bias hyperparameter, we have used Boolean() method to try boolean values. For activation hyperparameter, we have used Choice() method to select 'relu' or 'tanh' activation function. We have also used Choice() method to try different optimizers ('sgd','rmsprop' and 'adam').

After creating a model with hyperparameters, we have compiled it and returned it from the function. This function will be used by the hyperparameters optimization algorithm. The algorithm will provide one set of hyperparameters settings to this function to create a model and then it'll run this model recording its performance of it. The algorithm will also keep track of the performance of various hyperparameters settings to select the best one.

Below, we have performed hyperparameters tunning using a random search algorithm. We can create an instance of random search algorithm using RandomSearch() constructor available from keras tuner. The constructor takes the below-mentioned important parameters that are required for finding the best hyperparameters for the model.

• hypermodel - This parameter accepts an instance of HyperModel class or a callable that takes hyperparameters and returns a compiled model. We have created a callable above that we'll provide to this parameter. We can also create a class that extends HyperModel class and has build() method that works exactly like our callable (takes hyperparameters and returns a compiled model) above. Then, we can give that class instance to this parameter.
• objective - This parameter takes either string value of Objective instance specifying an objective function that we want to optimizer. Keras tuner will look for hyperparameters setting in a way that this objective function is minimized. To give an example, let's say, we give 'accuracy' as the value of this parameter then it'll try to maximize training accuracy, if we give 'val_loss' as the value of this parameter then it'll try to minimize validation loss.
  • We can provide any string metric or loss name to this parameter like 'val_loss', 'loss', 'mean_squared_error', etc. If we provide a string value then whether this objective function should be minimized or maximized will be inferred by the keras tuner.
  • We can also provide an instance of Objective. This is useful when we are using custom metric or loss and the keras tuner can't figure by itself whether to minimize or maximize it. The Objective() constructor takes two arguments.
    • name - This is the name of the metric to monitor.
    • direction - This is string specifying direction of optimization. The 'min' value for minimizing and 'max' value for maximizing.
• max_trials - This parameter accepts integer values specifying the number of different hyperparameter settings to try.
• seed - It's a random seed to reproduce the same results.
• hyperparameters - This parameter is optional and accepts instances of HyperParameters that can be used to override what is already set when creating a function that creates the network. We have explained the usage of this later.
• optimizer - This parameter is optional and accepts optimizers that can be used to override what we provided in a method that creates a model.
• loss - This parameter is optional and accepts loss that can be used to override what we have already specified inside the method that creates a model.
• metrics - This parameter is optional and accepts a list of metrics that can be used to override what we have already specified inside the method that creates a model.
• directory - This parameter accepts relative path specifying where work should be saved when the tuner tries various hyperparameters settings. By default, it's the current directory.
• project_name - This is the name of the project. The folder by this name will be created and all progress during the optimization process will be stored there. We need to give a different value to this parameter if we are trying more than one optimization at a time. We have given a different name for all our tuners.
• overwrite - This parameter accepts boolean value specifying whether to reload an existing project of the same name if there is one. Default is False.

In our case below, we have created RandomSearch instance by giving the function we designed earlier. We have asked it to minimize validation mean squared error and try 5 different combinations of hyperparameters. It'll call the function 5 times with different hyperparameters settings and create 5 different models to try.

After we have created an instance of RandomSearch tuner, we need to call search() function on it to actually try different hyperparameters settings. The search() function accepts same parameters as fit() method of model instance. We have given train data and validation data to the method. The call to search() function will try 5 different hyperparameters settings by creating 5 different models using them. It'll run all models for 10 epochs and record all metrics for them. It'll then sort models that have less validation mean squared error to more than one.

The tuner prints the best validation mean squared error of 63.61 at the end of the tuning process.

After hyperparameters tuning process has completed, we can call get_best_hyperparameters() method on instance of RandomSearch tuner. Below, we have printed the best hyperparameters combination.

We can also retrieve a model instance that gave the best results. We can use the same model for making predictions as well. It'll be loaded with trained parameters. We can save the best model for later use as well.

We can print the results of trials by calling results_summary() function. It'll print results in order of best to worst performing models. We can provide num_trials parameter to it specifying to print only that many best entries.

In our case, we have asked to print 3 best-performing models. We can notice that the first best performing network has a validation mean squared error of around 63.6.

2. Classification Example (Random Hyperparameters Search) ¶

As a part of our second example, we have explained how we can use a random search tuner for classification tasks. We have loaded the Fashion MNIST dataset below for our task. The dataset has grayscale images of shape (28,28) pixels for 10 different fashion items. The dataset is already divided into the train (60k images) and test (10k images) sets. We'll be trying various convolutional neural networks on this dataset to check which one is giving the best results.

In the below cell, we have created a new class that extends HyperModel class. The class has build() method that takes HyperParameters instance as input and returns a compiled keras model. It sets various hyperparameters using methods of HyperParameters instance. We'll be giving an instance of this class to RandomSearch() constructor later.

The function has first hyperparameter ('ConvNetType') that is choice between two values ('Conv1' and 'Conv2'). Based on the value of this parameter, we'll add two ('Conv1') or three ('Conv2') convolution layers to the network. For the convolution layer, we are trying different output channels using Int() method. For 2 layer convolution option ('Conv1'), both layers try 16 and 32 as output channel values. For 3 layer convolution option ('Conv2'), the first two convolution layer tries 16 and 32 output channel values and the third convolution layer tries 8 and 16 output channel values.

Apart from this, we have also asked to try different values of activation ('relu' and 'tanh') and kernel initialization ('random_normal', 'lecun_normal' and 'he_normal') using Choice() method.

Once, convolution layers are added to the network based on 'ConvNetType' hyperparameter value, we add a dense layer to the network that has 10 output units (same as the number of target classes) and softmax activation function.

Then, we have compiled the network and returned it.

This time, we have used another method of HyperParameters class named conditional_scope(). This method is used to create a scope that will be only active during specified values of particular hyperparameters. In our case, we have used it when 'ConNetType' has values 'Conv1' or 'Conv2'. This can be useful when we have to make decisions based on many values and we want a scope for a small list of values from all original values. To explain it with simple example, lets say that 'ConNetType' has 5 values ('Conv1', 'Conv2', 'Conv3', 'Conv4' and 'Conv5') and we want scope for 3 values ('Conv1', 'Conv3' and 'Conv5') and 2 values ('Conv2' and 'Conv4') separately.

In the below cell, we have created a random search tuner and executed it for 5 trials. We have given our instance of HyperModel to it and have asked it to maximize validation accuracy using Objective instance.

We have executed the tuning process by calling search() function giving it train data validation data, batch size (512), and epochs (10).

The tuner prints the best validation accuracy of 0.903 at the end of the tuning process.

In the below cell, we have printed the best hyperparameters settings that gave 0.903 accuracy.

In the next cells, we have retrieved the best model and used it to evaluate performance on the test dataset which we had used as a validation dataset. Then, we have printed the tuning summary as well.

3. Override Compile Arguments ¶

In this section, we have explained how we can override default arguments like an optimizer, loss function, and metrics that we gave when we compile the model inside of the model creation function.

The RandomSearch tuner lets us provide arguments like an optimizer, loss, and metrics that will override whatever we had provided when compiling the model. Below, we have explained for example how we can override default arguments. We have overridden the optimizer from Adam to RMSProp. Though we have overridden loss and metrics as well, we have provided the same values again.

4. Override Existing Hyperparameters Search Space ¶

As a part of this example, we have explained how we can override the existing setting of any hyperparameter by providing our new HyperParameters instance to RandomSearch() constructor.

There can be situations when we want to just override a few hyperparameters of the model and we don't want to modify settings done inside of the function building model. In those situations, we can define our own HyperParameters instance and set those hyperparameters that we want to modify inside it. Then, we provide this HyperParameters instance to hyperparameters argument of RandomSearch() constructor. This will override hyperparameters defined inside the function with those we provided through HyperParameters instance.

In our example below, we have again used the function from Regression section. We have override values that we try for activation functions of both dense layers. The function by default tries values relu and tanh. We have defined HyperParameters instance that replaces those values with selu and elu activations. Then, we have provided HyperParameters instance to RandomSearch tuner. We have then performed hyperparameters tunning by calling search() method on the tuner.

Later on, in the next few cells, we have printed the best hyperparameters, best model, and tuning results for verification purposes. We can notice from the results that the tuner now tries activation functions selu and elu instead of relu and tanh. This confirms that our settings are working as expected.

5. Fixing Few Hyperparameters ¶

In this example, we have explained how we can fix the values of some of the hyperparameters that we are tunning. We can do this by defining our own HyperParameters instance and calling Fixed() method to fix values of hyperparameters. Then, we need to give this HyperParameters instance to RandomSearch() constructor.

Below, we are again using the model building function from the regression section. We have fixed a few hyperparameters that we don't want to tune. We have set activation of the first dense layer to relu, a number of units of the first dense layer to 32, and optimizer to adam. These 3 hyperparameters inside of the build function won't be tuned and these fixed values will be used. All other hyperparameters defined inside the function will still be tuned.

After fixing hyperparameters, we have created RandomSearch tuner with HyperParameters instance and called search() method on it to perform hyperparameters tuning.

In the next few cells after tuning, we have also printed the best hyperparameters found by the tuner, best model, and tuning summary results.

We can notice from the results that for the first dense layer units are set at 32 and an activation value of relu is used. The adam optimizer is used for optimization. This confirms that our settings are working as expected.

6. Hyperband Algorithm ¶

In this section, we have performed hyperparameters optimization using Hyperband algorithm. It is a variation of random search with explore-exploit theory to find good hyperparameters settings. It focuses on speeding up random search through adaptive resource allocation and early stopping. It randomly allocates resources like iterations, data samples, and features to different hyperparameters settings and tries to solve stochastic bandit problems where it keeps on eliminating underperforming settings. The keras tuner provides an implementation of Hyperband algorithm tuner through Hyperband() constructor. It has the majority of the parameters same as random search with a few additional parameters as listed below.

• max_epochs - It accepts integers specifying a maximum number of epochs to train one model.
• factor - This parameter accepts integer value specifying reduction factor for a number of epochs and number of models for each bracket. Default is 3.
• hyperband_iterations - This parameter accepts integer specifying number of times to iterate over full hyperband algorithm. The default is 1. One iteration runs approximately max_epochs * (math.log(max_epochs, factor) ** 2) cumulative epochs across all trials.

In our case below, we have used Hyperband tuner for our classification task involving CNN. We have initialized it with the convolutional neural network with hyper band iterations set to 1. We have asked it to maximize validation accuracy.

After initializing the tuner, we have called search() method as usual to perform the tuning process. We have printed the best hyperparameters settings as well as the best model after completion of the tuning process. We have also printed the tuning process summary. We have got the best accuracy of 0.896. We can set hyperband_iterations to greater than 1 and it might improve results further.

7. Bayesian Optimization Algorithm ¶

In this example, we have explained bayesian optimization tuner available from keras tuner. Bayesian optimization uses Bayes theorem to find the best hyperparameters settings. We can use the Bayesian optimization tuner by BayesianOptimization() constructor of the keras tuner. It has almost the same parameters as a random search tuner with a few additional parameters listed below.

• num_initial_points - This parameter accepts integer values specifying the number of randomly generated samples for the initial training of the network. The default is 2.
• alpha - This parameter accepts float value added to the diagonal of kernel matrix during fitting. It is the expected amount of noise in the observed performances in the Bayesian optimization process. The default value is 1e-4.
• beta - This parameter accepts float value specifying balancing factor of exploration and exploitation. The larger value means more exploration. The default value is 2.6.

Below, we have initialized the bayesian optimization tuner and tried to find good hyperparameters settings for our classification task network (CNN). As usual, we have performed a search by calling search() method on the tuner object.

We have printed the best hyperparameters settings and best model after completion of the process, as well as a summary of various settings, tried.

This ends our small tutorial explaining how we can use various tuners available from keras tuner to find the best hyperparameters for the given model. We have explained all hyperparameters tuning algorithms available from keras tuner. Please feel free to let us know your views in the comments section.

References¶

• Keras Tuner API
• Simple Guide to Optuna for Hyperparameters Optimization/Tuning
• bayes_opt: Bayesian Optimization for Hyperparameters Tuning
• Scikit-Optimize: Simple Guide to Hyperparameters Optimization/Tunning
• Hyperopt - Simple Guide to Hyperparameters Optimization/Tunning
• Scikit-Learn - Cross-Validation & Hyperparameter Tuning Using GridSearch