Scikeras : Give Scikit-Learn like API to your Keras Networks¶

Scikit-learn is the most famous machine learning library in the python community. It's been used by the majority of developers worldwide. It provides an implementation of the majority of machine learning algorithms in its API. One of the main reasons scikit-learn is preferred by many developers is the simplicity of the API. It let us train ML models with one function call, make a prediction using one function call and even evaluation of dataset can be done with just one function call. This easy-to-use API has made the scikit-learn widely accepted library of ML as it does not require a steep learning curve.

But latest ML problems (object detection, image classification, speech recognition, etc) are quite complicated and can not be solved using simple ML algorithms available from scikit-learn. It requires creating complicated neural networks like convolutional neural networks, recurrent neural networks, etc. One famous library for creating such a complicated neural network is keras. Keras like scikit-learn has been accepted by many developers worldwide to create deep neural networks. Like scikit-learn provides easy API in the machine learning domain, keras provides easy to use API for the deep learning domain. This is the reason keras has been favored by many developers worldwide to create deep neural networks though it requires little learning to get things right.

As a part of this tutorial, we are going to introduce a new library named scikeras which lets us use keras deep neural networks with simple API like that of scikit-learn. Scikeras let us wrap our keras models into classes available from scikeras. We can then use this wrapped instances like scikit-learn ML model instances and call methods like fit(), predict() and score() on them. In short, scikeras let us use keras model like they are scikit-learn models. We'll explain the API of scikeras with simple examples using toy datasets available from scikit-learn.

Below we have highlighted important sections of the tutorial to give an overview of the material that we'll be covering.

Important Sections of Tutorial¶

1. Regression
   • Load Dataset
   • Create Keras Model
   • Wrap Keras Neural Network into Scikeras KerasRegressor
   • Train Wrapped Model
   • Make Predictions
   • Evaluate Model Performance on Test Data
   • Explore Training History
2. Classification
   • Load Dataset
   • Create Keras Classifier Neural Network
   • Wrap Keras Model into Scikeras KerasClassifier
   • Train Model
   • Make Predictions
   • Evaluate Model Performance
   • Analyze Training History
3. Warm Start
4. Machine Learning Pipeline
5. Grid Search
6. ML Pipeline + Grid Search
7. Saving and Loading Model

Below we have imported the necessary libraries that we'll be using in this tutorial and printed the versions of each of them.

1. Regression ¶

In this section, we'll explain how we can solve a simple regression problem using keras neural net by wrapping it using scikeras so that we can use it like scikit-learn for training and evaluation. We'll be creating a very simple neural network for explanation purposes. The dataset used for example is a simple Boston housing toy dataset available from scikit-learn.

Load Dataset¶

We'll start by loading the Boston housing dataset available from scikit-learn. It has information houses in Boston like the number of bedrooms, the crime rate in the area, tax rate, etc. The target variable of the dataset is the median value of homes in 1000 dollars. As the target variable is a continuous variable, this will be a regression problem.

We have divided the dataset into the train (80%) and test (20%) sets as well.

Create Keras Model¶

In this section, we have created a simple keras neural network that will be used for our regression problem. The network has an input layer with the same shape as that of a number of features of data which is 13 in our case. The second layer has 26 units, the third layer has 52 units and the final layer has 1 unit. The final layer has only 1 unit as it'll output prediction. The activation for the second and third layers is relu.

Please make a NOTE that we have not covered technical detail about model creation as we expect that reader has a bit background of simple neural network creation using keras.

Wrap Keras Neural Network into Scikeras KerasRegressor¶

In this section, we have explained how we can wrap the keras neural network into scikeras model so that it can be used like scikit-learn models. We'll be wrapping our network into KerasRegressor class from scikeras. It provides an API for regression tasks. Below we have highlighted the definition of KerasRegressor class.

• KerasRegressor(model,optimizer="rmsprop",loss=None,metrics=None,epochs=1,batch_size=None,callbacks=None,validation_split=0.0, verbose=1, warm_start=False, random_state=None)
  • This class constructor takes as input keras neural network and returns an instance of KerasRegressor which will behave like regression estimator from scikit-learn. We can call methods like fit(), predict(), and score() on instance of KerasRegressor.
  • The model parameter takes as input instance of keras.Model.
  • The optimizer parameter takes as input optimizer name specified as a string. It can also accept instances of optimizers created from keras.optimizers module. The default value is 'rmsprop'.
  • The loss function takes as input loss function name specified as a string. It can also accept instances of loss created from keras.losses module or callback from the same module.
  • The metrics parameter accepts a list of strings specifying metrics that need to be evaluated on each epoch.
  • The epochs parameter accepts integer specifying number of passes through train data.
  • The batch_size parameter specifies the batch size to use during training.
  • The callbacks parameter accepts a list of callable that needs to be executed at various stages of training like start/end of an epoch, start/end of the batch, etc.
  • The validation_split parameter accepts float in the range 0-1 which will divide the training dataset into train and validation sets based on the proportion specified using this parameter value.
  • The warm_start parameter accepts boolean values specifying whether to perform warm start or not. It is set to False by default which will reinitialize neural network weights each time we call fit() method on KerasRegressor. If we set it to True then it won't reinitialize network weights and training will start with weights after the last call to fit() method.

Below we have wrapped our keras neural network inside of KerasRegressor class. We have asked to use adam as optimizer and mean squared error as a loss. We have asked to use a batch size of 8 and run the training process for 100 epochs. We have set the verbose parameter to 0 to silent output as we don't want to flood output with messages of each epoch.

Train Wrapped Model¶

Now, we have simply trained our KerasRegressor using fit() method by giving it train features and target values.

Make Predictions¶

In this section, we have made predictions on test data using predict() method of KerasRegressor instance.

Evaluate Model Performance on Test Data¶

At last, we have calculated mean squared error and R^2 score on both train and test datasets to evaluate the performance of our neural network. We can notice from the results that it seems to have done a good job at the task.

The score() method will calculate the R^2 score for regression tasks by default.

If you are interested in learning about model evaluation metrics using scikit-learn then please feel free to check our tutorial on the same which explains the topic with simple and easy-to-understand examples.

• Scikit-Learn - Model Evaluation & Scoring Metrics

Explore Training History¶

The history object is available through history_ attribute of KerasRegressor instance. We can use it to access loss and metric values for both train and validation sets. Those values can later be used for plotting purposes as well.

The metrics that we have set in metrics parameter of KerasRegressor will also have entries here for each epoch.

2. Classification ¶

In this section, we'll explain how we can solve a simple classification problem using keras neural net by wrapping it using scikeras so that we can use it like scikit-learn estimator for training and evaluation. We'll be creating a very simple neural network for explanation purposes. The dataset used for example is a simple wine classification toy dataset available from scikit-learn.

Load Dataset¶

In this section, we have loaded the wine dataset available from scikit-learn. The wine dataset has the measurement of ingredients used in the creation of three different types of wine. The measurement of ingredients is the features of our dataset and wine type is the target variable.

After loading, We have divided the dataset into the train (80%) and test (20%) sets.

Create Keras Classifier Neural Network¶

In this section, we have created a simple classification neural network which we'll use to solve our classification task. The input layer of the network is of shape 13 which is the same size as the number of features. The second layer has 13 units, the third layer has 26 units and the final layer has 3 units (same as the number of wine classes). The second and third layer has relu as activation function. The final layer has softmax as an activation function.

Wrap Keras Model into Scikeras KerasClassifier¶

In this section, we have wrapped the keras neural network we created in the previous step into scikeras KerasClassifier. The KerasClassifier has an API for classification tasks. Below we have highlighted the definition of it which is almost the same as that of KerasRegressor.

• KerasClassifier(model,optimizer="rmsprop",loss=None,metrics=None,epochs=1,batch_size=None,callbacks=None,validation_split=0.0, class_weight=None,verbose=1, warm_start=False, random_state=None)
  • This class constructor takes as input keras neural network and returns an instance of KerasClassifier which will behave like regression estimator from scikit-learn. We can call methods like fit(), predict(), score() and predict_proba() on instance of KerasClassifier.
  • The model parameter takes as input instance of keras.Model.
  • The optimizer parameter takes as input optimizer name specified as a string. It can also accept instances of optimizers created from keras.optimizers module. The default value is 'rmsprop'.
  • The loss function takes as input loss function name specified as a string. It can also accept instances of loss created from keras.losses module or callback from the same module.
  • The metrics parameter accepts a list of strings specifying metrics that need to be evaluated on each epoch.
  • The epochs parameter accepts integer specifying number of passes through train data.
  • The batch_size parameter specifies the batch size to use during training.
  • The callbacks parameter accepts a list of callable that needs to be executed at various stages of training like start/end of an epoch, start/end of the batch, etc.
  • The validation_split parameter accepts float in the range 0-1 which will divide the training dataset into train and validation sets based on the proportion specified using this parameter value.
  • The warm_start parameter accepts boolean values specifying whether to perform warm start or not. It is set to False by default which will reinitialize neural network weights each time we call fit() method on KerasClassifier. If we set it to True then it won't reinitialize network weights and training will start with weights after the last call to fit() method.

Below we have wrapped our keras classifier into an instance of KerasClassifier. We have asked to use adam optimizer and categorical cross entropy as a loss. We have set epochs to 100 so that training will make 100 passes through data. We have set validation_split to 0.1 which will instruct the model to use 10% of training data for validation purposes.

Train Model¶

In this section, we have performed actual training by calling fit() method on an instance of KerasClassifier.

Make Predictions¶

In this section, we have made predictions on test data using predict() method. We can also make model output probabilities by calling predict_proba() method.

Evaluate Model Performance¶

In this section, we have evaluated model performance by calculating accuracy on test and train datasets using score() method. It'll calculate accuracy for classification models.

Analyze Training History¶

Here, we have shown a few entries of training and validation losses using the history object of the model. The metrics that we have set in metrics parameter of KerasClassifier will also have entries here for each epoch.

3. Warm Start ¶

When running the keras model to improve performance, we generally run it for a few epochs, check performance and then run it again for a few epochs to check whether performance is improving or not. We generally perform these trials until we find good accuracy. The process will update weights that were already updated last time.

When we wrap our keras model inside of scikeras model, by default it'll reset weights of the model each time we call fit() method on them which is referred to as the cold start. If we want to call fit() method more than once and update weights of the model from the last call to fit() then we should set parameter warm_start to True when creating scikeras model. This will inform the model to set model weights only before the first call to fit() method and all subsequent calls should update already updated weights through previous calls to fit().

By default, warm_start parameter is set to False which will reset model weights before each call to fit(). We can change this default behavior by setting parameter warm_start to True.

Load Data, Create Neural Network and Wrap It Inside of KerasClassifier¶

Our code for this example starts by loading the wine dataset and divides it into train/test sets. It then creates a keras model which is the same as that of the classification section. We have then wrapped our keras model inside of KerasClassifier scikeras model. The code in this part is almost the same as the code from the classification section.

Train Model¶

Below we are calling fit() method the first time with train data and the target variable. We have kept statistics getting displayed at the end of each epoch for this example. It's displaying train data loss at end of each epoch.

Below we have again called fit() method with train data to run the training process for another 5 epochs. This call to fit() method won't start with fresh model weights. Instead, it'll update weights from last call to fit() because we have set warm_start to True. We can notice from loss getting displayed after each epoch that it's decreasing from last call to fit().

Below we have called fit() method again to run the training process for another 5 epochs.

Below we have printed model accuracy on train and test datasets after completion of the training process. The accuracy is quite less because we have run the training process for only 15 epochs. If we run it for like 100 epochs then accuracy will improve significantly which we have done in the classification section.

4. Machine Learning Pipeline ¶

In this section, we'll explain how we can create a machine learning pipeline where we perform a list of steps on data before feeding it to the model. We'll explain how we can create a pipeline using scikit-learn and use our scikeras model in it. The pipeline will be simple and will have two steps only. The first step will scale the data and the second step will fit keras model to it. We'll be using the Boston housing dataset for our purpose.

If you are interested in learning about how to create a machine learning pipeline using scikit-learn then please feel free to check our tutorial on the same which tries to explain the topic with simple and easy-to-understand examples.

• Scikit-Learn - Creating ML Processing Pipeline

Load Dataset¶

Below we have loaded the Boston housing dataset available from scikit-learn and divided it into train/test sets. The code is exactly the same as the one from the regression section.

Create Keras Model and Wrap It Inside of Scikeras Model¶

Below we have created a simple keras model for performing regression task on our Boston housing dataset and wrapped it inside of scikeras KerasRegressor model. The code for this part is exactly the same as that of the regression section hence we have not included a detailed explanation.

Create and Train ML Pipeline¶

In this section, we have created our machine learning pipeline using Pipeline class of scikit-learn. It accepts a list of scikit-learn estimators which will be applied to data in sequence in which they are specified. We have a pipeline with two steps.

1. Data Scaling
2. Scikeras Regression Model

After creating the pipeline, we have trained the pipeline by calling fit() method on it giving train data and target variables to it.

If you want to learn about scaling the data for machine learning tasks then please feel free to check our tutorial on the same which covers the topic with simple and easy-to-understand examples.

• Scikit-Learn - Data Preprocessing [Scaling, Imputation & One-Hot Encoding]

Evaluate Performance of ML Pipeline¶

In this section, we have evaluated the performance of the ML pipeline by calculating MSE and R^2 score metrics on both train and test datasets. We can notice from the metrics output that the performance seems a little better due to scaling if we compare them with metrics results from the regression section.

5. Grid Search ¶

In this section, we'll explain how we can perform a grid search on hyperparameters to tune the model for good performance. We'll be creating a simple keras model, wrapping it inside of the scikeras model, and grid searching different hyperparameters of the model to find parameters setting which gives the best results. We'll be using the Boston housing dataset for our purpose.

If you are interested in learning about hyperparameters grid search using scikit-learn then please feel free to check our tutorial on the same which covers the topic with simple and easy-to-understand examples.

• Scikit-Learn - Cross-Validation & Hyperparameter Tuning Using GridSearch

Load Data, Create Keras Model and Wrap It Inside of KerasRegressor¶

In this section, we have loaded the Boston housing dataset and divided it into train/test sets. We have then created a simple keras model for the regression task and wrapped it inside of scikeras model. We have not provided optimizer parameter this time as we'll be trying different optimizers in a grid search.

Grid Search Model Hyperparameters¶

In this section, we have first declared a hyperparameters search dictionary with three different hyperparameters.

1. batch_size
2. optimizer - This will try two different optimizers.
3. optimizer__learning_rate - This will try 3 different values of learning rate for optimizer.

After creating a dictionary, we have created an instance of GridSearchCV by giving it scikeras model and hyperparameters dictionary. We have then called fit() method on an instance of GridSearchCV which will perform grid search by trying different combinations of those three hyperparameters to find the combination which gives the best result.

Print Best Score¶

Below we have printed hyperparameters setting that gave the best result. We have also printed the best score.

Print Results¶

Below we have evaluated MSE and R^2 score metrics on both train and test datasets to check the performance of the model with the best hyperparameters setting.

6. ML Pipeline + Grid Search ¶

In this section, we have explained how we can perform a grid search on a machine learning pipeline. This way we can tune earlier components of the ML pipeline as well along with the ML model. We'll be using the same ML pipeline which we had used in the ML pipeline section. We'll be using the Boston housing dataset for this example.

Load Data, Create Keras Model and Wrap It Inside of KerasRegressor¶

In this section, we have loaded the Boston housing dataset and divided it into train/test sets. We have then created a simple keras model for the regression task and wrapped it inside of scikeras model. The code for this part is exactly the same as our code from the previous grid search section.

Grid Search Model Hyperparameters¶

In this section, we have first declared a hyperparameters search dictionary with three hyperparameters to be tuned with different values of them. We have prefixed each hyperparameter name with string 'Model__' to specify that those hyperparameters are of scikeras model. The reason behind adding this prefix is that because we have specified scikeras model name as string 'Model' inside of ML pipeline.

After creating a dictionary, we have created an ML pipeline as we had created in the ML pipeline section. We have then created an instance of GridSearchCV by giving ML pipeline and hyperparameters dictionary to it. We have then called fit() method on an instance of GridSearchCV to perform grid search on hyperparameters.

Print Best Score¶

Below we have printed the performance of the model which gave the best result and the hyperparameters setting which was responsible for that result.

Print Results¶

Below we have printed MSE and R^2 scores evaluated on train and test datasets using the above ML pipeline with the best hyperparameters setting.

7. Saving and Loading Model ¶

In this section, we'll explain how we can save keras model wrapped inside of scikeras model to a file and then load it again.

We can access keras model underlying scikeras model anytime by just accessing model attribute of scikeras model. Below we have called model attribute on scikeras model from the regression section.

Save Model¶

Keras model has a method named save() which accepts pathname as input and will save keras model in that path.

Below we have called save() method on keras model present inside of scikeras model from regression section. We can notice a logging message informing us that model is saved inside of keras_regressor path.

Load Model¶

We can load the keras model from the saved path by calling load_model() method available from keras.models module. Below we have reloaded our keras model from keras_regressor directory.

After loading the keras model, we have wrapped it again inside of KerasRegressor to create a new scikeras model. We have set other parameters exactly the same way as we had set earlier during the regression section.

After creating scikeras model, we need to initialize it as well so that it can be used to make predictions.

Below we have calculated MSE and R^2 score metrics on both train and test datasets using both the original scikeras model from the regression section and the one we loaded from a file. We can notice that both have given the same results hence we have correctly loaded the model from the file.

This ends our small tutorial explaining how we can wrap keras model inside of scikeras so that the resulting model can be used like scikit-learn estimator with simple API. Please feel free to let us know your views in the comments section.

References¶

• Scikeras Docs