MXNet: Text Generation using LSTM Networks (Character-based RNNs)¶

Text Generation is an NLP task of generating new text using some trained network. The networks used for text generation tasks are referred to as Language Models and the process of training such networks is referred to as Language Modeling. Other tasks related to NLP like translation, text summarization, speech-to-text, conversational systems (chatbots), etc are also language modeling tasks. Nowadays deep neural networks are developed for language modeling. Generally, The Recurrent Neural Networks and their variants (LSTM, GRU, etc) outperforms other neural network architectures for text generation tasks. The reason behind this is that language sentences have structure. The word sequence is important for correct grammar and forming sentences. The RNNs are quite good at capturing and remembering those sequences hence commonly used for language modeling tasks.

As a part of this tutorial, we have explained how we can create Recurrent Neural Networks (RNNs) consisting of LSTM Layers using Python Deep Learning library MXNet for text generation tasks. We have used character-based approach for text generation which means we'll give the network a specified number of characters of a sentence and will make it predict the next character after them. The text data is encoded using bag of words approach. We have used Wikipedia dataset available from torchtext for our task. It has the text of well-curated Wikipedia articles. We have another tutorial on text generation using MXnet that uses character embeddings for encoding text data. Please check the below link to take a look at it.

• MXNet: Text Generation using LSTM and Character Embeddings

Please make a NOTE those language models are quite hard to train on CPU hence we recommend using GPU for the task. We have used GPU for training in this tutorial (see below after library imports).

Below, we have listed essential sections of the tutorial to give an overview of the material covered.

Important Sections Of Tutorial¶

1. Prepare Data
   • 1.1 Load Data
   • 1.2 Populate Vocabulary
   • 1.3 Reorganize Data for Task
   • 1.4 Create Dataset and Data Loader
2. Define Network
3. Train Network
4. Generate Text
5. Train Network More
6. Generate Text
7. Train Even More
8. Generate Text
9. Further Recommendations

Below, we have imported the necessary libraries and printed the versions that we have used in our tutorials.

1. Prepare Data ¶

In this section, we are preparing data for our text generation task. As discussed earlier, we'll be using character-based approach for the task. We'll be designing a network that takes 100 characters of data as input and predicts the next character. In order to train the network, we'll organize data following the below steps.

1. Load dataset.
2. Loop through each text example of the dataset and create a vocabulary of unique characters. A vocabulary is a simple mapping from a character to an integer index. Each character is assigned a unique index starting from 0.
3. Move the window of size 100 through text examples of data to create data features (X) and target values (Y). To explain with an example,
   • Characters 1-100 will be data features (X) and character 101 will be the target value (Y).
   • Move the window by one character.
   • Characters 2-101 will be data features (X) and character 102 will be target value (Y).
   • Move the window by one character.
   • Characters 3-102 will be data features (X) and character 103 will be target value (Y).
   • Move the window by one character.
   • and so on.
4. After creating data features and target values, retrieve character index for characters present in them using vocabulary.
5. Create data loaders to train the network.

The output of the 5th step will be used to train the network. It's okay if you don't understand the steps exactly, they will become clear as we implement them below.

1.1 Load Data¶

In this section, we have loaded our Wikipedia dataset from torchtext library. The dataset is already divided into train, test, and validation sets. We'll be using only train dataset for our purpose. It has nearly ~36k text documents each representing a unique Wikipedia article.

1.2 Populate Vocabulary¶

In this section, we have populated the vocabulary of unique characters. In order to populate vocabulary, we have used count_tokens() helper functions available from gluonnlp helper Python library from MXnet. We have first created a Counter object available from collections Python library. This object is a kind of dictionary object that maintains characters and their count. After defining the counter object, we are looping through each text example of the dataset calling count_tokens() method with a list of characters and counter object. This method will keep updating the counter object with frequency of characters. After completion of the loop, Counter object will have all characters and their counts in the dataset.

Then, we have created a vocabulary by calling Vocab() constructor available from gluonnlp module with Counter object. This Vocab object holds our vocabulary of characters. We have printed a number of entries present in vocabulary as well as vocabulary contents itself.

1.3 Reorganize Data for Task¶

In this section, we have organized our dataset for training purposes. We are looping through text examples of train dataset one by one and moving window of size 100 through each example as we had discussed earlier. We have created data features (X_train) and target values (Y_train) that have characters. Then, we have retrieved the index of characters from the vocabulary as well for data features (X_train) and target values (Y_train). Now our dataset consists of character indexes which will be given to the network for training. We have also converted arrays to mxnet ndarrays as required by MXNet networks.

Please make a NOTE that we have used the first few text examples from the dataset for training and not the whole dataset. The reason behind this is that it'll take a lot of time to train the network if we use all examples.

Below, we have tried to explain the process with one simple example.

vocab = {
'h':1,
'e':2,
'l':3,
'o':4,
' ':5,
',':6,
'w',7,
'a':8,
'r':9,
'y':10,
'u':11,
'?':12,
'c':13,
'm':14,
't':15,
'd':16,
'z':17,
'n':18
}

text_example = "Hello, How are you? Welcome to coderzcolumn?"
seq_length = 10

X_train = [
            ['h','e','l','l','o',',',' ', 'h','o','w'],
            [,'e','l','l','o',',',' ', 'h','o','w',' '],
            ['l','l','o',',',' ', 'h','o','w', ' ', 'a'],
            ['l','o',',',' ', 'h','o','w',' ', 'a', 'r'],
            ...
            ['d','e','r','z','c','o','l', 'u','m','n']
            ]
Y_train = ['e','l','l','o',',',' ', 'h','o','w',' ',..., '?']

X_train_vectorized = [
                        [1,2,3,4,5,6,1,4,7],
                        [2,3,4,5,6,1,4,7,5],
                        [3,4,5,6,1,4,7,5,1],
                        ...
                        [16,2,9,17,13,4,3,11,14,18]
                     ]
Y_train_vectorized = [1,2,3,4,5,6,1,4,7,5,1,...., 12]

1.4 Create Dataset and Data Loader¶

In this section, we have created the dataset and data loader using data features and target values ndarrays we created in the previous step. The data loader will help us loop through training data in batches. We have set shuffle to False to prevent shuffling of examples because character sequence is important. The batch size is set at 1024.

2. Define Network ¶

In this section, we have defined our LSTM network that we'll use for our task. As we'll be predicting one of the vocabulary characters as output, our task will be considered classification task. The network is simple and consists of 3 layers.

1. LSTM Layer
2. LSTM Layer
3. Dense Layer

The first two layers of our network are LSTM layers. We have created LSTM layers using LSTM() constructor available from rnn sub-module of gluon sub-module of MXnet. We have set the output of the size of lstm layers at 256. By setting num_layers parameter to 2, we have asked the constructor to stack two LSTM layers. The input shape to first LSTM layer will be (batch_size, seq_length, 1) = (batch_size, 100, 1) and output shape will be (batch_size, seq_length, hidden_size) = (batch_size, 100, 256). The output of the first LSTM layer will be given to the second LSTM layer for processing and it'll also produce an output of shape (batch_size, 100, 256). The LSTM layers process a sequence of character indexes internally. For each example, it goes through 100 characters and produces a final output that in some way remembers something about this 100 characters sequence.

The output of the second LSTM layer will be given to the dense layer. The dense layer has the same output units as the length of vocabulary. The output of the dense layer will be of shape (batch_size, vocab_len) and will be a prediction of the network.

After defining the network, we initialized it and performed a forward pass through it for verification purposes. We have also printed the shape of weights/biases of layers of network for information purposes.

We have not discussed LSTM layer in detail over here. If you are someone new to it then we recommend that you go through the below link in your free time where we have explained it. The tutorial uses the LSTM network for text classification task and it'll help you understand LSTM layers a little better.

• MXNet: LSTM Networks for Text Classification Tasks

If you are new to MXNet and want to learn how to create neural networks using it then please check the below link in your free time.

• Simple Guide to Create Neural Networks using MXNet

3. Train Network ¶

In this section, we are training our network. In order to train the network, we have designed a function that will help us perform the training process. The function takes the trainer object, train data loader, and a number of epochs as input. It then executes a training loop number of epochs time. During each epoch, it loops through training data in batches. For each batch of data, it performs a forward pass to make predictions, calculates loss, calculates gradients, and updates network parameters. The function records the loss of each batch and prints the average loss of all batches at the end of the epoch.

Below, we are actually training the network using the function designed in the previous cell. We have initialized a number of epochs to 50 and the learning rate to 0.001. Then, we have initialized our model, cross-entropy loss, Adam optimizer, and Trainer object (with network parameters). At last, we have called our training routine with the necessary parameters to perform training. We can notice from the decreasing loss value at the end of each epoch that our model seems to be improving. Next, we'll generate some text using it.

4. Generate Text ¶

In this section, we are generating text using our trained model. To start with, we have randomly selected an example from our dataset. We have printed the characters of our example. Then, we have executed a loop of generating new characters. The loop generates 100 new characters. For the first iteration, our selected example is given to the model to make the prediction. The predicted character is added at the end of the sequence and the first character is removed from the sequence to keep the sequence length of 100 as required by our model. This modified sequence with a predicted character added at the end of the sequence becomes an input to the model for the next iteration. This process is repeated for each iteration where we generate a new character, add it at the end of the sequence and remove the existing first character from the sequence. After generating 100 new characters, we have also printed them.

We can notice from the generated text that our model has learned to form words and there are no spelling mistakes. The generated text seems to be in the English language though it does not make much sense. The prediction made by the network seems a little deterministic as it is repeating words. We'll train the network for more epochs to see whether we can further improve results.

5. Train Network More ¶

In this section, we have trained the network for another 50 epochs. We have set the learning rate to 0.0003. We can notice from the loss values that our network is improving further because loss is decreasing at every epoch. Next, we'll test it by generating new text.

6. Generate Text ¶

In this section, we have again generated text using our more trained model. The code for text generation is almost the same as earlier. We have started with the same random example. We can notice from the generated text that it is a little better compared to last time. It is generating more words. There are no spelling mistakes as usual. Though the model still seems deterministic. We can train it for a few more epochs to check whether it helps.

7. Train Even More ¶

In this section, we have trained the network for another 50 epochs with a learning rate of 0.0001. Please make a note that we have reduced the learning rate a second time. The loss values indicate that our network has improved further. Next, we'll test it by generating new text.

8. Generate Text ¶

Here, we have again generated new text using our trained network. Our logic to generate text is the same as earlier and it starts with the same example. We can notice from the generated text that it is quite different compared to earlier. It is generating punctuation marks as well this time. It is correctly spelling words. Next, we have given a few recommendations on how we can improve text generation models further.

9. Further Recommendations ¶

1. Train the network for more epochs.
2. Try different sequence lengths. We tried a sequence length of 100.
3. Try different encoding approaches to encode text like character embeddings, etc.
4. Try the n-gram/word-based model instead of the character-based model.
5. Try different LSTM layer output sizes.
6. Try adding more LSTM layers. This can increase training time.
7. Try learning rate schedulers
8. Add little randomness to the prediction of the next character. REFERENCE

References¶

• MXNet: LSTM Networks for Text Classification Tasks
• Simple Guide to Create Neural Networks using MXNet