Keras: GloVe Embeddings for Text Classification Tasks¶

Word embeddings is one of the most preferred approaches when encoding text data to real-valued data for Machine learning tasks. When using the word embeddings approach, we first break text into a list of tokens (words) and then assign a real-valued vector (embeddings) of pre-decided length to each token. Other encoding approaches like one-hot, word frequency, Tf-IDF, etc assign just a single scalar to each token. This limits the representation power of the tokens. With a single scalar, you can only represent a single meaning. With the word embeddings approach where we assign a real-valued vector to the token, it gives more representation power to it. The same word can have different meanings in a different context and can have a different relationship with other words. These can be better captured by embeddings.

GloVe (Global Vectors) is an unsupervised learning algorithm that is trained on a big corpus of data to capture the meaning of the words by generating word embeddings for them. These word embeddings can be then used by other ML tasks that have different small datasets. The trained token embeddings can be taken from GloVe Embeddings. This saves us time to generate and train our own embeddings which might not be possible with small datasets. There are different versions of Glove embeddings generated by Stanford professors with different embedding lengths. Please feel free to check the below link if you want to learn more about GloVe.

• GloVe: Global Vectors for Word Representation

As a part of this tutorial, we have designed neural networks using Python deep learning library Keras (Tensorflow) that uses GloVe Word Embeddings (840B.300d) for text classification tasks. We have tried different approaches to using embeddings and recorded their results for comparison purposes. We have also calculated various ML metrics for evaluating the performance of our networks. Apart from calculations, we have also explained predictions made by our networks using LIME (Local Interpretable Model-Agnostic Explanations) algorithm.

Below, we have listed important sections of tutorial to give an overview of the material covered.

Important Sections Of Tutorial¶

1. Prepare Data
   • 1.1 Download And Unzip GloVe Embeddings (840B.300d)
   • 1.2 Load GloVe Word Embeddings in Memory
   • 1.3 Load Dataset
   • 1.4 Tokenize Examples, Populate Vocabulary And Vecorize Text Examples
   • 1.5 Create Matrix Of GloVe Embeddings for Vocabulary Tokens
2. Approach 1: GloVe Embeddings Flattened (Max Tokens=50, Embedding Length=300)
   • Define Network
   • Compile Network
   • Train Network
   • Evaluate Network Performance
   • Explain Network Predictions using LIME Algorithm
3. Approach 2: GloVe Embeddings Averaged (Max Tokens=50, Embedding Length=300)
4. Approach 3: GloVe Embeddings Summed (Max Tokens=50, Embedding Length=300)
5. Results Summary And Further Suggestions

Below, we have imported Python deep learning library Keras (tensorflow) and printed the version we used in our tutorial.

1. Prepare Data ¶

In this section, we are preparing data to be given to the neural network for training purposes. We'll follow the below steps in order to use GloVe word embeddings for text classification tasks.

1. Loop through all text examples, tokenize them and populate the vocabulary of all tokens. A vocabulary is a simple mapping from token to integer index. Each token is assigned a unique index starting from 0. This unique index identifies tokens uniquely.
2. Loop through each text example, tokenize them and retrieve indexes of tokens of examples from the vocabulary. After this step, we'll have a list of token indexes for each text example.
3. Loop through each token of vocabulary and retrieve GloVe embeddings for tokens. Wrap embeddings of all tokens in a single matrix.
4. Retrieve glove embeddings for tokens by integer indexing embedding matrix created in the third step.

So basically, we first tokenize text examples, populate vocabulary, and retrieve token indexes for tokens of text examples. We can then retrieve glove embeddings using these token indexes to index the embeddings matrix.

The first three steps mentioned above will be completed in this section whereas the fourth step will be implemented in the neural network as the Embedding layer. The Glove embedding matrix will be set as the weight matrix of the first layer of the network which is the embedding layer and this layer will retrieve embeddings from the weight matrix based on input token indexes.

Below, we have included an image that explains word embeddings.

1.1 Download And Unzip GloVe Embeddings (840B.300d)¶

In this step, we have simply downloaded GloVe 840B.300d word embeddings from the Stanford website as a zip file and then unzipped it. This word embedding has embeddings of length 300 for 2.2 Million tokens.

1.2 Load GloVe Word Embeddings in Memory¶

In this section, we are simply loading Glove embeddings in memory from the file. We have created a simple dictionary whose keys are tokens (words) and values are embeddings.

1.3 Load Dataset¶

In this section, we have loaded the dataset that we are going to use for our text classification task. We'll be using the newsgroups dataset available from scikit-learn. The dataset has ~20k text documents for 20 different categories of news. We have loaded text documents for four categories for our purpose. The dataset is already divided into train and test sets for our convenience.

1.4 Tokenize Examples, Populate Vocabulary, And Vecorize Text Examples¶

In this section, we have implemented the first two steps of our encoding process which we had explained at the beginning.

First, we have created an instance of Tokenizer and called fit_on_texts() method on it. We have provided the method train and test examples. A call to this method will internally populate a vocabulary of all unique tokens in the tokenizer object.

Next, we have called texts_to_sequences() method on Tokenizer object with train and test text examples. This method will tokenize each text example into tokens and then retrieve indexes of those tokens from the vocabulary.

Now, each of our text examples is of a different length and hence has a different number of tokens (words). We have decided to keep maximum of 50 tokens per text example. To do this, we have called pad_sequences() function on the list of token indexes. This method will make sure that each example has exactly 50 token indexes. The examples that have more than 50 tokens will be truncated to 50 tokens and those who have less than 50 tokens will be padded with 0s to bring it to length 50.

So we have turned each text example into a list of token indexes. We'll later use these indexes to integer index embedding matrix and retrieve embeddings of tokens.

In the next cells after the below cell, we have printed the number of tokens in vocabulary.

Below, we have explained with a simple example how text example is vectorized.

text = "Hello, How are you? Where are you planning to go?"

tokens = ['hello', ',', 'how', 'are', 'you', '?', 'where',
            'are', 'you', 'planning', 'to', 'go', '?']

vocab = {
    'hello': 0,
    'bye': 1,
    'how': 2,
    'the': 3,
    'welcome': 4,
    'are': 5,
    'you': 6,
    'to': 7,
    '<unk>': 8,
}

vector = [0,8,2,4,6,8,8,5,6,8,7,8,8]

1.5 Create Matrix Of GloVe Embeddings for Vocabulary Tokens¶

In this section, we have implemented the third step of our encoding process that we had explained earlier. We are simply looping through our vocabulary and retrieving GloVe embeddings for each token. We have then stacked embeddings of all tokens of our vocabulary in one big matrix. The matrix have shape (vocab_len, embed_len). The embedding length in our case is 300 as we had said earlier.

We can now retrieve glove embedding from this matrix by integer indexing it using the token index of that token. To explain it with an example, let's say that the index of 'the' token in our vocabulary is '1' then we can simply index embedding matrix like 'embedding_matrix[1]' to retrieve the embedding of the token 'the'.

Approach 1: GloVe Embeddings Flattened (Max Tokens=50, Embedding Length=300) ¶

Our first approach flattens GloVe embeddings and processes them through dense layers to make predictions. We have defined a simple network of one embedding layer and 3 dense layers for our text classification task. After training the network, we have also calculated ML metrics for evaluating the performance of the network.

Define Network¶

In this section, we have defined a network that we'll use for our text classification task. The network consists of one embedding layer and 3 dense layers.

The first layer of the network is the embedding layer. We have created an embedding layer using Embedding() constructor. We have provided the length of vocabulary as a number of tokens to the constructor. The embedding length is set at 300. We have also set the glove embedding matrix we created earlier as the weight of the layer. We don't want to update these embeddings hence we have set trainable parameter to False which will prevent updating of weights. This layer is responsible for mapping token indexes to their respective embeddings. The shape of input data to embedding layer is (batch_size, max_tokens) = (batch_size, 50) and output data shape is (batch_size, max_tokens, embed_len) = (batch_size, 50, 300).

The output of embedding layer is flattened which transforms data shape from (batch_size, 50, 300) to (batch_size, 50 x 300) = (batch_size, 15000).

The flattened output is given to a dense layer with 128 output units. This will change data from shape (batch_size, 15000) to (batch_size, 128). The relu activation is applied to the output of the dense layer.

The output of the first dense layer is given to the second dense layer which has 64 output units. This transforms shape from (batch_size, 128) to (batch_size, 64). It also applies relu activation to the output.

The output of second layer is given to the third and last dense layer of the network which has 4 output units (same as no of target classes) for processing. This transforms shape from (batch_size, 64) to (batch_size, 4). The last layer also applies softmax activation to convert outputs to probabilities.

After defining the network, we have also printed a summary of layers' output shapes and count of parameters.

Compile Network¶

Here, we have compiled our network to use Adam optimizer, cross entropy loss, and accuracy metric.

Train Network¶

In this section, we have trained our network for 8 epochs with a batch size of 1024 using fit() method. We have provided a test dataset as a validation dataset for validation purposes. We can notice from the loss and accuracy values getting printed after each epoch that our network is doing a good job at the text classification task.

Evaluate Network Performance¶

In this section, we have evaluated the performance of our network by calculating accuracy score, classification report (precision, recall, and f1-score per target class) and confusion matrix metrics on test predictions. We can notice from the accuracy score that our network has done a good job at the classification task.

Here, we have calculated ML metrics using functions available from the Python library Scikit-learn. If you are interested in learning about various ML metrics available from sklearn then please check the below link which covers the majority of them in detail.

• Scikit-Learn: Model Evaluation and Scoring Metrics

Apart from calculations, we have also created a visualization of the confusion matrix. This lets us better understand which target class is doing better and which are worse. We can notice from the visualization that the accuracy of all 4 categories is almost the same.

We have created confusion matrix visualization using Python library scikit-plot. It provides visualization for many ML metrics. Please feel free to check the below link if you are interested in it.

• Scikit-Plot: Visualizing Machine Learning Algorithm Results & Performance Metrics

Explain Network Predictions using LIME Algorithm¶

In this section, we have explained the prediction made by our network using LIME (Local Interpretable Model-Agnostic Explanations) algorithm. The algorithm is commonly used to explain black-box ML models like deep neural networks. Currently, the Python library lime provides an implementation of the algorithm. We'll use it to create a visualization that highlights words that contributed to predicting a particular target label on a given text example.

If you are someone who is new to the concept of LIME and want to understand it better then we would recommend that you go through the below links in your spare time.

• How to Use LIME to Understand sklearn Models Predictions?
• LIME: Interpret Predictions Of Keras Text Classification Networks

In order to create explanation visualization using lime, we need to create an instance of LimeTextExplainer which we had done below. We'll use it later.

Below, we have first created a prediction function. The function takes a batch of text examples and returns their probabilities predicted by the network. We'll use this function later to create an explanation.

Next, we randomly selected a text example from test data and made predictions on it using our trained network. We can notice that our network correctly predicts the target label as 'rec.autos' for the selected text example. Now, we'll create an explanation visualization for this text example.

Below, we have first created Explanation object by calling explain_instance() method on LimeTextExplainer instance. We provided the method with a selected text example, a prediction function we had defined earlier, and a target label. Then, we have called show_in_notebook() method on the explanation object to create visualizations.

We can notice from the visualization that words like 'cars', 'Datsun', 'Opel', 'owners', etc are contributing to predicting the target label as 'rec.autos'. This makes sense as these are commonly used words in the auto industry.

Approach 2: GloVe Embeddings Averaged (Max Tokens=50, Embedding Length=300) ¶

Our approach in this section has a minor change in network architecture compared to our previous approach. Our network in this section takes the average of embeddings of tokens per text example instead of flattening them like in the previous section. The majority of the code is the same as in the previous section with only a change in network architecture.

Define Network¶

Below, we have defined the network that we'll use for our text classification task. The network as usual consists of one embedding layer and three dense layers. The only major difference from the previous section is that the output of the embedding layer is averaged at the token level. This step transforms data shape from (batch_size, max_tokens, embed_len) = (batch_size, 50, 300) to (batch_size, embed_len) = (batch_size, 300). This output is then given to the first dense layer and forward.

After defining the network, we have printed a summary of layers output shapes and layer parameter counts.

Compile Network¶

Here, we have compiled a network to use Adam embedding, cross entropy loss, and accuracy metric.

Train Network¶

Now, we have trained our network for 8 epochs with a batch size of 32 using fit() method. We have provided test data as validation data for validation purposes. We can notice from the loss and accuracy values that our network is doing a good job at classification tasks.

Evaluate Network Performance¶

In this section, we have evaluated the performance of our trained network by calculating accuracy score, classification report and confusion matrix metrics on test predictions. We can notice from the accuracy score that it is quite better compared to our previous approach. We have also plotted the confusion matrix which is also showing improvement in classifying text documents into individual categories.

Explain Network Predictions using LIME Algorithm¶

In this section, we have explained network predictions using LIME algorithm. Our trained network correctly predicts the target label as 'rec.autos' for randomly selected text example from test data. The visualization shows that words like 'cars', 'Datsun', 'Nissan', 'Opel', 'mileage', etc contributes to predicting target category as 'rec.autos'.

Approach 3: GloVe Embeddings Summed (Max Tokens=50, Embedding Length=300) ¶

Our approach in this section is the same as our previous approach the only minor difference is that we are taking the sum of embeddings instead of the average. The majority of the code is the same as earlier with only differences in network architecture.

Define Network¶

Below, we have defined a network that we'll use for our task in this section. The network has the same architecture as the previous section with the only difference being that we have used sum() function on the output of the embedding layer instead. The rest of the code is exactly the same.

Compile Network¶

Here, we have compiled a network to use Adam optimizer, cross entropy loss, and accuracy metric.

Train Network¶

Below, we have trained our network using the same settings that we have been using for all our approaches. The loss and accuracy values getting printed after each epoch hint that the network is doing a good job at the classification task.

Evaluate Network Performance¶

In this section, we have evaluated the performance of the network by calculating ML metrics like accuracy score, classification report and confusion matrix on test predictions. We can notice from the accuracy score that it is the highest of all our approaches. We have also plotted the confusion matrix metric for reference purposes.

Explain Network Predictions using LIME Algorithm¶

In this section, we have explained predictions made by our network using LIME algorithm. The network correctly predicts target label as 'rec.autos' for selected text example from test dataset. The visualization shows that words like 'cars', 'datsun', 'nissan', 'mileage', 'opel', 'sell', 'rust', '240Z', '260Z', '280Z', '280ZX' etc are contributing to predicting target category as 'rec.autos'.

5. Results Summary And Further Suggestions ¶

Further Recommendations¶

• Try different token sizes per text example.
• Try different embedding lengths.
• Try other GloVe embeddings like 42B, 27B, 6B, etc.
• Try different functions on embeddings other than average and sum.
• Try different combinations of linear/dense layers.
• Try different weight initialization methods.
• Try different activation functions.
• Train network for more epochs.
• Try different optimizers.
• Try learning rate schedulers

References¶

• Keras: Word Embeddings for Text Classification
• Text Classification Using Keras Networks
• Keras: CNNs With Conv1D For Text Classification Tasks
• Keras: LSTM Networks For Text Classification Tasks
• Keras: RNNs For Text Classification Tasks
• LIME: Interpret Predictions Of Keras Text Classification Networks
• Explain Text Classification Models Using SHAP Values (Keras + Vectorized Data)
• SHAP Values for Text Classification Tasks (Keras NLP)