Keras: LSTM Networks For Text Classification Tasks¶

Recurrent Neural Networks (RNNs) is the preferred network when working with data that has sequences in it like time-series data, text data, etc. These kinds of datasets have an internal sequence that can not be captured by a neural network consisting of dense layers because it does not take previous examples into consideration when making predictions of current examples. RNNs can capture the hidden sequence of such datasets by taking past examples into consideration when making decisions as they have an influence on the current example which needs to be captured. Over the years, different versions of RNNs like LSTM and GRU have been implemented, which perform better than vanilla RNN networks. Below, we have included an image of one cell of the LSTM network. Many such cells are laid out next to each other in one LSTM layer to capture the sequence.

Please feel free to check the below link for a theoretical overview of Recurrent Neural Networks.

• Recurrent Neural Networks (RNNs): An In-depth Guide

As a part of this tutorial, we have explained various ways to design LSTM networks using Python deep learning library Keras for solving text classification tasks. The tutorial explains different approaches to using LSTM layers in a network. To encode text data, we have used word embeddings approach. We have used AG NEWS dataset available from torchtext for our purpose. After training the network, we have also evaluated their performance by calculating various ML metrics and explained prediction using LIME algorithm. Please feel free to check the below tutorial if you are looking for a guide to using vanilla RNN layers.

• Keras: RNNs For Text Classification Tasks

Below, we have listed essential sections of tutorial to give an overview of the material covered.

Important Sections Of Tutorial¶

1. Prepare Data
   • 1.1 Load Datasets
   • 1.2 Tokenize Text Data
2. Approach 1: Single LSTM Layer Network (Max Tokens=25, Embedding Length=25, LSTM Output=50)
   • Truncate Data to Max Tokens
   • Define Network
   • Compile Network
   • Train Network
   • Evaluate Network Performance
   • Explain Network Predictions using LIME Algorithm
3. Approach 2: Single LSTM Layer Network (Max Tokens=50, Embedding Length=25, LSTM Output=50)
4. Approach 3: Single Bidirectional LSTM Layer Network (Max Tokens=50, Embedding Length=25, LSTM Output=50)
5. Approach 4: Multiple LSTM Layers Network (Max Tokens=50, Embedding Length=25, LSTM Output=50,60,75)
6. Approach 5: Multiple Bidirectional LSTM Layers Network (Max Tokens=50, Embedding Length=30, LSTM Output=50,60,75)
7. Results Summary And Further Suggestions

Below, we have imported the necessary Python libraries and printed the versions we used in our tutorial.

1. Prepare Data ¶

In this section, we are preparing data to be given to the neural network. We are going to use word embeddings approach to encode text data where we break text into tokens (words, punctuation marks, etc.) and each token is assigned a real-valued vector. This encoding approach will be completed in three steps.

1. Create vocabulary of all unique tokens. The vocabulary is a simple mapping from token to integer index. Each token is assigned a unique integer index starting from 0.
2. Tokenize each text example into tokens and assign indexes to each token using vocabulary.
3. Map indexes to real-valued vectors.

The first two steps will be completed in this section whereas the third step will be implemented as an embedding layer in the neural network which will map token indexes to their real-valued vectors.

Below, we have included an image of word embeddings for explanation purposes.

1.1 Load Datasets¶

In this section, we have simply loaded AG NEWS dataset available from torchtext library. The dataset has text documents for 4 different categories (["World", "Sports", "Business", "Sci/Tech"]) of news. We'll be using this dataset for our text classification task.

1.2 Tokenize And Vectorize Text Data¶

In this section, we have completed the first two steps of our word embedding encoding process explained earlier.

First, we have populated a vocabulary of all unique tokens present in datasets (train and test) by creating an instance of Tokenizer and calling fit_on_texts() function on it with train and test text examples.

Then, we have called texts_to_sequences() method on Tokenizer object with train and text examples. This step will tokenize each text example into a list of tokens and retrieve their respective indexes from the vocabulary. The output of this step will be a list of indexes per text example.

Below, we have explained with one simple example how vectorization works.

text = "Hello, How are you? Where are you planning to go?"

tokens = ['hello', ',', 'how', 'are', 'you', '?', 'where',
            'are', 'you', 'planning', 'to', 'go', '?']

vocab = {
    'hello': 0,
    'bye': 1,
    'how': 2,
    'the': 3,
    'welcome': 4,
    'are': 5,
    'you': 6,
    'to': 7,
    '<unk>': 8,
}

vector = [0,8,2,4,6,8,8,5,6,8,7,8,8]

Approach 1: Single LSTM Layer Network (Max Tokens=25, Embedding Length=25, LSTM Output=50) ¶

The first approach that we'll try for our text classification task consists of a neural network that has only one LSTM layer. Apart from the LSTM layer, the network has an embedding layer for encoding purposes and a dense layer for generating probabilities of target classes. We have trained this network and evaluated its performance by calculating various ML metrics. We have also tried to explain network predictions using LIME algorithm.

Truncate Data to Max Tokens¶

We have decided that for our approach in this section, we'll keep 25 tokens per text example. By default, a text document can be of any length hence we have used pad_sequences() method to truncate all text examples that have more than 25 tokens and pad examples that has less than 25 tokens with 0s to bring all to the length of 25. The output of this step will be used during the training of the neural network.

Define Network¶

In this section, we have defined the first neural network that we'll be using for our text classification task. The network consists of 3 layers.

1. Embedding Layer
2. LSTM Layer
3. Dense Layer

The first layer of the network is Embedding layer which is responsible for mapping token indexes to their respective real-valued vectors. We have defined the embedding layer using Embedding() constructor available from layers sub-module of keras. We have given a number of tokens as vocabulary length and embedding length of 25 to the constructor. The initialization of this layer will create an internal weight matrix of shape (vocab_len, embed_len) which has embeddings for each token of vocabulary. When we give a list of indexes to the network, this layer will simply map embeddings of token indexes using this matrix. The input shape to this will be (batch_size, max_tokens) = (batch_size, 25) and the output shape will be (batch_size, max_tokens, embed_len) = (batch_size, 25, 25).

The output of the embedding layer will be given to LSTM layer. The LSTM layer is created with output units of size 50. The LSTM layer will takes input of shape (batch_size, max_tokens, embed_len) and returns output of shape (batch_size, max_tokens, lstm_out) = (batch_size, 25, 50). The LSTM layer goes through tokens of each text example one by one to capture the sequence.

The output of the LSTM layer is given to Dense layer that has 4 output units (same as a number of target classes) and softmax activation** function.

Compile Network¶

Below, we have compiled our network to use rmsprop optimizer, cross entropy loss, and accuracy metric.

Train Network¶

Here, we have trained our network by calling fit() method. We have trained the network for 5 epochs with a batch size of 512. We have provided train data for training and test data for validation purposes. We can notice from the loss and accuracy values getting printed at the end of each epoch that our model is doing a good job.

Evaluate Network Performance¶

In this section, we have evaluated the performance of our trained network by calculating accuracy score, classification report (precision, recall and f1-score per target class) and confusion matrix metrics on test predictions. We can notice from the accuracy score that our model has done a good job at the text classification task. We have calculated ML metrics using various functions available from scikit-learn.

Please feel free to check the below link if you are interested in learning about various ML metrics available from sklearn.

• Scikit-Learn - Model Evaluation & Scoring Metrics

Apart from the calculation, we have also plotted the confusion matrix. We can notice from the plot that our model is doing a good job at text classification tasks for all categories except Business where accuracy is a little low compared to other categories.

We have created a confusion matrix plot using Python library scikit-plot. Please feel free to check the below link if you want to learn about it as it provides plots for many ML metrics.

• Scikit-Plot: Visualizing Machine Learning Algorithm Results & Performance Metrics

Explain Network Predictions using LIME Algorithm¶

In this section, we have tried to explain predictions made by our network using LIME algorithm. The python library lime has an implementation of the algorithm. It let us generate visualization which highlights words contributing to predicting a particular target label.

Please feel free to check the below links if you are new to the LIME and want to learn about it in depth.

• How to Use LIME to Understand sklearn Models Predictions?
• LIME: Interpret Predictions Of Keras Text Classification Networks

In order to explain predictions using lime, we first need to create an instance of LimeTextExplainer which we have done below.

Here, we have first declared a function that takes a list of text examples as input and returns their prediction probabilities using our trained model. It tokenizes and vectorizes text examples before giving them to the network to generate probabilities. We'll use this function in the next cell when generating an explanation.

Then, we randomly selected one text example from the test dataset. We have then made predictions on that example using our trained network. Our network correctly predicts the target category as Business for the selected example.

Below, we have generated an explanation visualization for the selected text example. To do that, we have first called explain_instance() method on LimeTextExplainer object. We have provided selected the text example, prediction function, and target label for the method. The method returns an Explanation instance.

At last, we have called show_in_notebook() method on the explanation instance to generate the visualization.

We can notice from the visualization that words like 'concessions', 'airlines', 'pensions', 'financing', 'bankruptcy', 'cuts', etc are used for predicting the target category Business which makes sense as they are commonly used words in the business world.

Approach 2: Single LSTM Layer Network (Max Tokens=50, Embedding Length=25, LSTM Output=50) ¶

Our approach in this section is exactly the same as our approach in the previous section the only difference is that we have increased the number of tokens per example to 50 from 25. We have increased to tokens to see whether it improves accuracy or not. The majority of the code is exactly the same as in our previous section.

Truncate Data to Max Tokens¶

Below, we have truncated our text examples to maintain 50 tokens per example by calling pad_sequences() method.

Define Network¶

Below, we have defined a network that we'll use for our text classification task in this section. The network definition is exactly the same as in the previous section.

Compile Network¶

Here, we have compiled our network to use rmsprop optimizer, cross entropy loss, and accuracy metric.

Train Network¶

Below, we have trained our network using exactly the same settings that we had used in the previous section. We can notice from the loss and accuracy values getting printed after each epoch that our network is doing a quite good job at the given classification task.

Evaluate Network Performance¶

In this section, we have evaluated the performance of our trained network as usual by calculating accuracy score, classification report and confusion matrix metrics on test predictions. We can notice from the accuracy score that it is a little better compared to our previous approach. We have also created a confusion matrix plot for reference purposes.

Explain Network Predictions using LIME Algorithm¶

In this section, we have tried to explain predictions made by our trained network using a LIME algorithm. Our network correctly predicts the target label as Business for randomly selected a text example from the test dataset. The visualization shows that words like 'concessions', 'employees', 'airlines', 'financing', 'bankruptcy', 'pensions', 'million', etc are contributing to predicting target label as Business.

Approach 3: Single Bidirectional LSTM Layer Network (Max Tokens=50, Embedding Length=25, LSTM Output=50) ¶

Our approach in this section again uses a single LSTM layer like our previous approach but this time bidirectional LSTM layer is used. By default, LSTM layers are unidirectional which means that they process sequences in the forward direction to find out patterns. In the case of bidirectional LSTM layers, it processes sequences in both forward and backward directions to find out patterns. This can sometimes help improve network performance. The majority of the code is the same as our earlier approaches with only a change in network architecture.

Define Network¶

Below, we have defined the network that we'll use for our text classification task in this section. We can create bidirectional LSTM layer by wrapping normal LSTM layer in Bidirectional() constructor. The output of bidirectional LSTM layer will be of shape (batch_size, max_tokens, 2 x lstm_out) = (batch_size, 25, 100) because it traverses sequence in both forward and backward directions.

Compile Network¶

Below, we have complied our network to use rmsprop optimizer, cross entropy, and accuracy metrics.

Train Network¶

Here, we have trained our network using the same settings we have been for all our approaches. We can notice from the loss and accuracy values getting printed after each epoch that our model is doing a good job at the task.

Evaluate Network Performance¶

In this section, we have evaluated the performance of our trained network by calculating accuracy score, classification report and confusion matrix metrics on test predictions. We can notice from the accuracy score that it is the highest of all our approaches. We have also plotted the confusion matrix for reference purposes.

Explain Network Predictions using LIME Algorithm¶

In this section, we have explained predictions made by our trained network using LIME algorithm. The network correctly predicts the target label as Business for the selected test example. The visualization created using LIME shows that words like 'airlines', 'concessions', 'bankruptcy', 'cuts', 'pensions', 'labor', 'financing', etc are contributing to predicting target label as Business.

Approach 4: Multiple LSTM Layers Network (Max Tokens=50, Embedding Length=25, LSTM Output=50,60,75) ¶

Our approach in this section creates a recurrent neural network consisting of multiple LSTM layers. The network uses 3 LSTM layers of different output units. The majority of the code in this section is the same as earlier with the only difference being network architecture.

Define Network¶

Below, we have defined the network that we have used for our text classification task. The network consists of 5 layers. The embedding layer is followed by 3 LSTM layers with output unit sizes of 50, 60, and 75 respectively. The output of the last LSTM layer is given to the Dense layer whose output is the prediction of network.

Compile Network¶

Here, we have compiled our network to use RMSProp optimizer, cross entropy loss, and accuracy metric.

Train Network¶

In this section, we have trained our network using the same settings that we have been using for all our approaches. We can notice from the loss and accuracy values that the network is doing a good job at the classification task.

Evaluate Network Performance¶

In this section, we have evaluated the performance of the network by calculating the accuracy score, classification report and confusion matrix metrics. We can notice from the accuracy score that it's a little less compared to our single LSTM layer networks which is surprising as we had expected that stacking more layers could improve accuracy further.

Explain Network Predictions using LIME¶

In this section, we have explained predictions of the network using LIME algorithm. The network correctly predicts the target label as Business for the selected text example from the test dataset. The visualization created using LIME highlights that words like 'concessions', 'airlines', 'cuts', 'bankruptcy', 'pensions', 'financing', 'labor', etc are contributing to predicting target label as Business.

Approach 5: Multiple Bidirectional LSTM Layers Network (Max Tokens=50, Embedding Length=30, LSTM Output=50,60,75) ¶

Our approach in this section is exactly the same as our approach in the previous section with minor changes. Our approach in this section again uses a recurrent neural network with multiple LSTM layers but all layers are bidirectional this time. The majority of the code is the same as our earlier sections with the only difference being network architecture.

Define Network¶

Below, we have defined the network that we'll use for our text classification task in this section. The network architecture is the same as in the previous section with the only difference being that LSTM layers are wrapped in Bidirectional() constructor to make them bidirectional.

Compile Network¶

Here, we have compiled our network to use RMSProp optimizer, cross entropy loss, and accuracy metric.

Train Network¶

In this section, we have trained our network using the same settings that we have used for all our previous approaches. We can notice from the loss and accuracy values getting printed after each epoch that our model is doing a good job at the text classification task.

Evaluate Network Performance¶

In this section, we have evaluated the performance of our trained network by calculating accuracy score, classification report and confusion matrix metrics on test predictions. We can notice from the accuracy that it is doing a good job at the classification task. We have also plotted the confusion matrix for reference purposes.

Explain Network Predictions using LIME Algorithm¶

In this section, we have explained the prediction made by our network on a random test example using LIME algorithm. The network correctly predicts the target label as Business for the selected sample. The visualization shows that words like 'concessions', 'financing', 'pensions', 'labor', 'cuts', etc are contributing to predicting target label as Business.

7. Results Summary And Further Suggestions ¶

Further Recommendations¶

• Try different max tokens.
• Try different embedding lengths.
• Try different LSTM layer output units.
• Train network for more epochs.
• Try different optimizers (Adam).
• Try different weight initialization for networks.
• Try to stack the different number of LSTM layers.
• Try learning rate schedulers

References¶

• Keras: RNNs For Text Classification Tasks
• Text Classification using Keras Networks
• Keras: Word Embeddings for Text Classification