LightRNN: Memory and Computation-Efficient Recurrent Neural Networks

Introduction

The study of natural language processing has been around for more than fifty years. It begins in 1950s which the specific field of natural language processing (NLP) is still embedded in the subject of linguistics [(Hirschberg & Manning, 2015)]. After the emergence of strong computational power, computational linguistics began to evolve and gradually branch out to various applications of NLP, such as text classification, speech recognition and question answering (https://machinelearningmastery.com/applications-of-deep-learning-for-natural-language-processing/). Computational linguistics or natural language processing is usually defined as “subfield of computer science concerned with using computational techniques to learn, understand, and produce human language content” (Hirschberg & Manning, 2015, p. 261).

With the development in deep neural networks, one type of neural network, namely recurrent neural networks (RNN) have preformed significantly well in many natural language processing tasks. The reason why such specific networks works better compared to others is that the nature of RNN takes into account of the past input as well as the current input. More detail of how RNN works will be discussed in the section of recurrent neural network. However, one challenge faced by RNN used in NLP is the enormous size of its input vocabulary. This will result in a very complex RNN model with too many parameters to train and makes the training process both time and memory-consuming. This is the motivation for this paper’s authors to develop a new model of RNN that is particularly efficient at processing large size of vocabulary in many NLP tasks.

LightRNN Structure

Part II: Word Allocation

1. one
2. two