Skorch: Give Scikit-Learn like API to your PyTorch Networks¶

Scikit-Learn is the most famous ML library out there. It's been the most preferred ML library for a long time. It has an implementation of the majority of ML algorithms related to any problems (regression, classification, clustering, anomaly detection, dimensionality reduction, etc.). One of the main reasons why it's most preferred is because it's easy to use API. It let us perform training of our data and evaluation of it on test sets with just a few function calls. Though it has good API and implementation of the majority of ML algorithms, it does not have support for deep neural networks (convolutional neural network, recurrent neural network, etc.) which are commonly getting used to solving complicated problems (image classification, speech recognition, etc) nowadays. Apart from this, Scikit-Learn does not have support for running code on GPU as well.

The commonly preferred library when creating a deep neural network is PyTorch. It let us create complicated networks like convolutional and recurrent neural networks as well as it let us run the code on GPU as well. But one of the drawbacks of Pytorch is that it's a lower-level library that requires us to design training and evaluation code for our problem. This can take time to get it working correctly and can result in errors as well sometimes. This can be a little hard for the person with a background in Scikit-Learn who wants to use PyTorch to solve their problem.

To eliminate the drawbacks of both Scikit-Learn and PyTorch, a new library named Skorch was created. It let us use PyTorch for creating complicated neural network models and then use Scikit-Learn like API for training and evaluating that model. This frees developers from the burden of writing code for the training and evaluation of models. Skorch also makes the life of a developer who has good experience with Scikit-Learn and lets him easily use PyTorch to solve complicated problems using neural networks without worrying too much about code of training and evaluation.

As a part of this tutorial, we'll explain with simple examples how we can use Skorch to train and evaluate PyTorch models. We'll be creating simple neural networks to make things easy to understand and will be trying them on small toy datasets. Below we have listed important sections of the tutorial for giving an overview of what we'll be covering.

Important Sections of Tutorial¶

1. Regression
   • Load Dataset
   • Create Neural Network Model using PyTorch
   • Wrap Neural Network into NeuralNetRegressor Object of Skorch
   • Train Model
   • Predict Using Trained Model
   • Evaluate Model Performance
   • Analyze History Object of Training
2. Classification
   • Load Dataset
   • Create Neural Network Model using PyTorch
   • Wrap Neural Network into NeuralNetClassifier Object of Skorch
   • Train Model
   • Predict Using Trained Model
   • Evaluate Model Performance
   • Analyze History Object of Training
3. Machine Learning Pipeline
4. Grid Search Hyperparameters
5. ML Pipeline + Grid Search
6. Saving and Loading ML Model

Below we have imported the necessary libraries that we'll use in our tutorial and printed the version of each.

1. Regression ¶

In this section, we'll explain how we can use skorch for a regression problem. We'll be designing a simple PyTorch neural network and using it to solve a regression problem. We'll be using the Boston housing dataset available from scikit-learn for our purpose.

Load Dataset¶

In this section, we have loaded the Boston housing dataset available from scikit-learn. It has information houses in Boston like the number of bedrooms, the crime rate in the area, tax rate, etc. The target variable of the dataset is the median value of homes in 1000 dollars. As the target variable is a continuous variable, this will be a regression problem.

We have divided the dataset into the train (80%) and test (20%) sets as well. After that, we have converted the train and test dataset into PyTorch tensor as all PyTorch models require input as tensors.

Create Neural Network using Pytorch¶

In this section, we have created a simple neural network that takes as input our dataset features and predicts the median house price.

The design of a neural network is simple. The first layer is the input layer with 13 inputs (one for each feature of data). The second layer has 26 neurons, the third layer has 52 neurons and the final layer has just one neuron for predicting the house price. We have initialized all layers in init method of the class. The forward method has actual logic about going through the network. It applies relu activation function after each layer and returns output at last.

Please make a NOTE that we have not explained neural network creation using PyTorch in detail as we expect that reader has a background in simple model creation using it.

Wrap Neural Network into NeuralNetRegressor Object of Skorch¶

In this section, we have included logic that will create an ML model by wrapping PyTorch neural network which will behave like scikit-learn models and have API-like scikit-learn models (methods like fit(), predict(), etc).

In order to make our PyTorch neural network behave like scikit-learn ML models, we need to wrap them into NeuralNetRegressor instance of skorch. Below we have given the definition of NeuralNetRegressor for explanation purposes.

• NeuralNetRegressor(module,optimizer=torch.optim.SGD,criterion=torch.nn.MSELoss,lr=0.01,max_epochs=10,batch_size=128,train_split=skorch.dataset.CVSplit(5),warm_start=False,verbose=1,device="cpu",**kwargs)
  • This class constructor takes as input pytorch neural network and returns an instance of NeuralNetRegressor which will behave like scikit-learn ML models. We can call methods like fit(), score() and predict() on instance of NeuralNetRegressor.
  • The module parameter takes an instance of torch.nn.Module class which represents neural network designed using PyTorch.
  • The optimizer parameter takes as input instance of any class from torch.optim module. The commonly used optimizers are SGD, Adam, Adagrad, etc. The default value of optimizer for NeuralNetRegressor is torch.optim.SGD. The optimizer is used to update the parameters of the neural network after calculating gradients.
  • The criterion parameter accepts reference to loss function available from torch.nn module. The default value of loss function for NeuralNetRegressor is torch.nn.MSELoss. The gradients of parameters is calculated with respect to loss function.
  • The lr parameter represents the learning rate of the neural network and accepts float value. The default value is 0.01.
  • The max_epochs parameter accepts integer value specifying the number of epochs to try.
  • The train_split parameter accepts instance of skorch.dataset.CVSplit specifying how to divide train dataset to create train and validation sets. The validation set will be used to evaluate the performance of the model for each validation set. The default value is skorch.dataset.CVSplit(5) which will divide the dataset into 5 sets, use 4 sets for training and 1 set for validation purposes.
  • The warm_start parameter accepts boolean value. If the value of the parameter is set to True then each call to fit() won't reinitialize the parameter. It'll start with the last trained model. If set to False, it'll reinitialize model parameter for each call to fit() method.
  • Apart from this, if we want to set the value of parameters of optimizer then we can give that by using special notation. We need to give an optimizer parameter prefixed by optimizer__ string. E.g - optimizer__momentum.

Below we have created an instance of NeuralNetRegressor by giving it our neural network which we had created in an earlier cell. We have given an instance of Adam optimizer to be used in the model. We have asked the model to be trained for 500 epochs for each call to fit() method. We have set verbose parameter to 0 which will have silent updates printed at each epoch. By default, the output result of each epoch is printed. As we have nearly 500 epochs, it'll flood output hence we have silent it.

Train Model¶

In this section, we have simply trained the model by calling fit() method giving it train dataset features and the target variable.

Predict Using Trained Model¶

We can make predictions using predict() method by giving it dataset features.

Evaluate Model Performance¶

In this section, we are evaluating model performance by calculating mean squared error and R2 score on both train and test datasets. We can notice that our model seems to have good performance based on evaluation results.

The score() method will calculate R^2 score for regression problems.

If you are interested in learning about model evaluation metrics using scikit-learn then please feel free to check our tutorial on the same which explains the topic with simple and easy-to-understand examples.

• Scikit-Learn - Model Evaluation & Scoring Metrics

Analyze History Object of Training¶

We can access training history using history attribute of NeuralNetRegressor instance. It has information about train loss, validation loss, epoch number, etc.

Below we have retrieved a few details from the history of training and printed them.

2. Classification ¶

In this section, we have explained how we can use PyTorch classification models like scikit-learn models. We’ll be designing a simple PyTorch classification neural network as a part of this example. We'll be using the wine dataset available from scikit-learn for our purpose.

Load Dataset¶

In this section, we have loaded the wine dataset available from scikit-learn. The wine dataset has the measurement of ingredients used in the creation of three different types of wine. The measurement of ingredients is the features of our dataset and wine type is the target variable.

After loading, We have divided the dataset into the train (80%) and test (20%) sets. We have then converted all train and test array into tensor as all PyTorch models requires input as tensors.

Create Neural Network Model using PyTorch¶

In this section, we have designed a simple PyTorch model of a few layers. The first layer is the input layer with 13 inputs (13 features). The second layer has 5 neurons and the last layer has 13 neurons. We have initialized each layer inside of init method of the model class. We have then included logic to take a dataset and make predictions inside of forward() method. We have also applied relu activation for each intermediate layer. At the final layer, we have applied softmax activation.

Wrap Neural Network into NeuralNetClassifier Object of Skorch¶

In order to make our PyTorch classification neural net behave like scikit-learn models, we need to wrap them inside of NeuralNetClassifier class. It has the almost same signature as that of NeuralNetRegressor. Below we have highlighted the definition.

• NeuralNetClassifier(module,optimizer=torch.optim.SGD,criterion=torch.nn.modules.loss.NLLLoss,classes=None,lr=0.01,max_epochs=10,batch_size=128,train_split=skorch.dataset.CVSplit(5),warm_start=False,verbose=1,device="cpu",**kwargs)
  • This class constructor takes as input pytorch neural network and returns an instance of NeuralNetClassifier which will behave like scikit-learn ML models. We can call methods like fit(), predict(), score() and predict_proba() on instance of NeuralNetClassifier.
  • The module parameter takes an instance of torch.nn.Module class which represents neural network designed using PyTorch.
  • The optimizer parameter takes as input instance of any class from torch.optim module. The commonly used optimizers are SGD, Adam, Adagrad, etc. The default value of optimizer for NeuralNetClassifier is torch.optim.SGD. The optimizer is used to update the parameters of the neural network after calculating gradients.
  • The criterion parameter accepts reference to loss function available from torch.nn module. The default value of loss function for NeuralNetClassifier is torch.nn.modules.loss.NLLLoss. The gradients of parameters is calculated with respect to loss function.
  • The classes parameter takes as input list of classes of the task.
  • The lr parameter represents the learning rate of the neural network and accepts float value. The default value is 0.01.
  • The max_epochs parameter accepts integer value specifying the number of epochs to try.
  • The train_split parameter accepts instance of skorch.dataset.CVSplit specifying how to divide train dataset to create train and validation sets. The validation set will be used to evaluate the performance of the model for each validation set. The default value is skorch.dataset.CVSplit(5) which will divide the dataset into 5 sets, use 4 sets for training and 1 set for validation purposes.
  • The warm_start parameter accepts boolean value. If the value of the parameter is set to True then each call to fit() won't reinitialize the parameter. It'll start with the last trained model. If set to False, it'll reinitialize model parameter for each call to fit() method.
  • Apart from this, if we want to set the value of parameters of optimizer then we can give that by using special notation. We need to give an optimizer parameter prefixed by optimizer__ string. E.g - optimizer__momentum.

Below we have wrapped our PyTorch classifier inside of NeuralNetClassifier. We have also instructed to use nn.CrossEntropyLoss as loss function and optim.Adam as optimizer. We have also instructed train_split parameter to do a stratified split of the dataset as its classification dataset and we want the proportion of classes to be maintained across different datasets.

Train Model¶

Now, we have trained the model by calling fit() method and giving it the train dataset and the target variable.

Predict Using Trained Model¶

Below we have made prediction on test dataset using predict() and predict_proba() methods. The predict() method will return actual class and predict_proba() will return probabilities of each class.

Evaluate Model Performance¶

In this section, we have printed the accuracy of the model on train and test datasets. The score() method will calculate accuracy by default.

Analyze History Object of Training¶

Below we have printed a few entries of the history of the training process for analysis purposes.

3. Machine Learning Pipeline ¶

In this section, we'll explain how we can create a machine learning pipeline using scikit-learn by treating our PyTorch model as a sklearn estimator using skorch. We'll be creating a simple ML pipeline with only two steps. The first step of the pipeline will scale the data and the second step will apply our PyTorch model wrapped inside of skorch class. We'll be using the Boston housing dataset for this example hence our example will be solving the regression task.

If you are interested in learning about how to create a machine learning pipeline using scikit-learn then please feel free to check our tutorial on the same which tries to explain the topic with simple and easy-to-understand examples.

• Scikit-Learn - Creating ML Processing Pipeline

Load Dataset¶

In this section, we have loaded the Boston housing dataset from sklearn, divided it into train/test sets, and converted datasets into Pytorch tensor. The code in this part is almost exactly the same as our code from the first example data load.

Create Neural Network and Wrap it in NeuralNetRegressor Object¶

In this section, we have created a simple PyTorch model and wrapped it into NeuralNetRegressor class of skorch so that it can be used like sklearn estimator. The code of this part is almost the same as that of the code from the regression section.

Create ML Pipeline and Train Pipeline¶

In this section, we have created a machine learning pipeline using Pipeline class of sklearn. Our ML pipeline consists of two steps.

1. The first step applies RobustScaler to data to scale it.
2. The second step applies an actual model to scaled data.

After creating the pipeline, we have called fit() method on it to train the pipeline using train data.

If you want to learn about scaling the data for machine learning tasks then please feel free to check our tutorial on the same which covers the topic with simple and easy-to-understand examples.

• Scikit-Learn - Data Preprocessing [Scaling, Imputation & One-Hot Encoding]

Evaluate Pipeline Performance¶

In this section, we have evaluated the performance of the pipeline by calculating MSE and R^2 scores on train and test datasets. We can compare the output with that of output from the regression section and can notice that metrics results have improved significantly by just applying simple scaling to data.

4. Grid Search ¶

In this section, we'll explain how we can perform hyperparameters tunning by grid searching through different values of hyperparameters. We'll be designing a simple PyTorch neural network, wrapping it inside of skorch class, and grid search through different hyperparameters of the model to find the best hyperparameter settings that give the best results. We'll be using the Boston housing dataset from the previous section.

If you are interested in learning about hyperparameters grid search using scikit-learn then please feel free to check our tutorial on the same which covers the topic with simple and easy-to-understand examples.

• Scikit-Learn - Cross-Validation & Hyperparameter Tuning Using GridSearch

Create Neural Network and Wrap it in NeuralNetRegressor Object¶

In this section, we have created a simple PyTorch neural network for the regression task and wrapped it inside of NeuralNetRegressor class of skorch to make it behave like sklearn estimator. The code for this part is almost the same as the code from the regression section.

Grid Search Model Hyperparameters¶

In this section, we have first declared a hyperparameters dictionary with a list of hyperparameters and their different values to try. The GridSearchCV from sklearn will try all combinations of these hyperparameters values with our data and keep track of results. We'll be trying different values of the below 3 hyperparameters.

1. lr - learning rate
2. max_epochs - Number of pass through training data
3. optimizer__weight_decay - This hyperparameter value is that of optimizer hence we have specified hyperparameter name after optimizer__.

After creating an instance of GridSearchCV by giving skorch regressor and hyperparameters dictionary, we have also performed hyperparameters search by calling fit() method on grid search instance. The call to fit() method will try different combinations of hyperparameters on a model with given data.

Print Best Score¶

In this section, we have printed the best hyperparameters settings which gave the best score.

Print Results¶

At last, we have printed MSE and R^2 scores for train/test datasets using grid instance which will use a model with best hyperparameters settings.

5. ML Pipeline + Grid Search ¶

In this section, we have explained how we can perform a grid search for hyperparameters tunning on a machine learning pipeline. We can tune various parameters of individual parts of the pipeline. We'll be creating a pipeline using scikit-learn and performing a grid search on it. We'll be using the Boston housing dataset which we had loaded earlier. We'll also be reusing the skorch wrapped PyTorch model for the task which we had created in the previous section.

Grid Search ML Pipeline Hyperparameters¶

Below we have first declared a hyperparameters search dictionary. As we'll be tunning hyperparameters of our skorch model, we have prefixed all hyperparameter names with string 'Model__'. The reason behind adding this prefix is because we have given the name 'Model' to our skorch model inside of ML pipeline which we have declared next.

We have then created an instance of GridSearchCV by giving it ML pipeline and hyperparameters dictionary. At last, we have performed grid search on ML pipeline by calling fit() method on GridSearchCV instance by giving it training data.

Print Best Score¶

In this section, we have printed the best hyperparameters settings which gave the best score.

Print Results¶

At last, we have printed MSE and R^2 scores for train/test datasets using grid instance which will use a model with best hyperparameters settings.

6. Saving and Loading Model ¶

In this section, we have explained how we can save the trained skorch model and then load it again from saved files.

The skorch model provides a method named save_params() which lets us save model weights, optimizer, loss function, and training history to different files. We can then load the model using these files and resume training or make direct predictions.

Save Model¶

Below we have called save_params() method on skorch model from classification section. We have provided it with four different file names for saving different details of the model as well.

Create Empty Model¶

Below we have created a new instance of NeuralNetClassifier using our PyTorch model. We have given all other parameter values which we had used in the classification section.

After creating the model, we need to call initialize() method on it in order to make any prediction using it. This is needed because we haven't called fit() method a single time on it. If we have called fit() method then weights and other things will be initialized and we don't need to call initialize().

We have also evaluated model performance after initializing it and we can notice that as the model is initialized with random weights, the performance is not good.

Load Model¶

We can load the model with previously saved weights using load_params() method of skorch model.

Below we have called load_params() method on the newly created skorch model by giving it various file names to which we had saved model weights and other details.

After loading the model from files, below we have evaluated the performance of the model on train and test datasets. We can see that it's giving the same results as that of the classification section model.

This ends our small tutorial explaining how we can wrap PyTorch model inside of scikeras model so that the resulting model can be used like scikit-learn estimator. Please feel free to let us know your views in the comments section.

References¶

• Skorch Docs