why lstm over rnn?

You can describe a recurrent neural network (RNN) or a long short-term memory (LSTM), depending on the context, at different levels of abstraction. For example, you could say that an RNN is any neural network that contains one or more recurrent (or cyclic) connections. Or you could say that layer $l$ of neural network $N$ is a recurrent layer, given that it contains units (or neurons) with recurrent connections, but $N$ may not contain only recurrent layers (for example, it may also be composed of feedforward layers, i.e. layers with units that contain only feedforward connections).

In any case, a recurrent neural network is almost always described as a neural network (NN) and not as a layer (this should also be obvious from the name).

On the other hand, depending on the context, the term "LSTM" alone can refer to an

People may also refer to neural networks with LSTM units as LSTMs (plural version of LSTM).

An LSTM unit is a recurrent unit, that is, a unit (or neuron) that contains cyclic connections, so an LSTM neural network is a recurrent neural network (RNN).

The main difference between an LSTM unit and a standard RNN unit is that the LSTM unit is more sophisticated. More precisely, it is composed of the so-called gates that supposedly regulate better the flow of information through the unit.

Here's a typical representation (or diagram) of an LSTM (more precisely, an LSTM with a so-called peephole connection).

This can actually represent both an LSTM unit (and, in that case, the variables are scalars) or an LSTM layer (and, in that case, the variables are vectors or matrices).

You can see from this diagram that an LSTM unit (or layer) is composed of gates, denoted by

and recurrent connections (e.g. the connection from the cell into the forget gate and vice-versa).

It's also composed of a cell, which is the only thing that a neuron of a "vanilla" RNN contains.

To understand the details (i.e. the purpose of all these components, such as the gates), you could read the paper that originally proposed the LSTM by S. Hochreiter and J. Schmidhuber. However, there may be other more accessible and understandable papers, articles or video lessons on the topic, which you can find on the web.

Source

Long short-term memory (LSTM) is the artificial recurrent neural network (RNN) architecture used in the field of deep learning. Unlike standard RNNs, LSTM has "memory cells" that can remember information for long periods of time. It also has three gates that control the flow of information into and out of the memory cells: the input gate, the forget gate, and the output gate.

LSTM networks have been used on a variety of tasks, including speech recognition, language modeling, and machine translation. In recent years, they have also been used for more general sequence learning tasks such as activity recognition and music transcription.

So, as we are now through with the basic question, “what is long short term memory” let us move on to the ideology behind Long short term memory networks. Humans can remember memories from the distant past, as well as recent events, and we can also easily recall sequences of events. LSTMs are designed to mimic this ability, and they have been shown to be successful in a variety of tasks, such as machine translation, image captioning, and even handwriting recognition. Does this bring an easy undertaking to “What is the long-term memory”?

But we are now here with the question, how do Long Short Term Memory networks work? As quoted everywhere in the basic Database Courses, the key difference between LSTMs and other types of neural networks is the way that they deal with information over time. Traditional neural networks process information in a “feedforward” way, meaning that they take in input at one-time step and produce an output at the next time step.

LSTMs, on the other hand, can process information in a “recurrent” way, meaning that they can take in input at one-time step and use it to influence their output at future time steps. This recurrent processing is what allows LSTMs to learn from sequences of data.

There are four main components to an LSTM network: the forget gate, the input gate, the output gate, and the cell state. The forget gate controls how much information from the previous time step is retained in the current time step. The input gate controls how much new information from the current time step is added to the cell state. The output gate controls how much information from the cell state is used to produce an output at the current time step. And finally, the cell state is a vector that represents the “memory” of the LSTM network; it contains information from both the previous time step and the current time step.

Recurrent neural networks (RNNs) are a type of artificial neural network that is well-suited for processing sequential data such as text, audio, or video. RNNs have a recurrent connection between the hidden neurons in adjacent layers, which allows them to retain information about the previous input while processing the current input.

This makes RNNs particularly useful for tasks such as language translation or speech recognition, where understanding the context is essential. A long short term memory neural network is designed to overcome the vanishing gradient problem, which can occur when training traditional RNNs on long sequences of data. LSTMs have been shown to be effective for a variety of tasks, including machine translation and image captioning.

Long Short Term Memory networks are a type of recurrent neural network designed to model complex, sequential data. Unlike traditional RNNs, which are limited by the vanishing gradient problem, LSTMs can learn long-term dependencies by using a method known as gated recurrent units (GRUs). GRUs contain a "forget" gate, which allows them to selectively forget information from the previous timestep, and an "update" gate, which allows them to control how much information from the current timestep is passed on to the next time step.

This makes LSTMs well-suited for tasks such as machine translation, where it is important to be able to remember and interpret information from long sequences. In addition, LSTMs can be trained using a variety of different methods, including backpropagation, through time and reinforcement learning.

Long Short Term Memory neural networks are types of recurrent neural networks (RNN) that are well-suited for modeling sequence data. In contrast to RNNs, which tend to struggle with long-term dependencies, LSTMs can remember information for extended periods of time. This makes them ideal for tasks such as language modeling, where it is important to be able to capture the context of a sentence to predict the next word. LSTMs are also commonly used in machine translation and speech recognition applications.

There are a number of advantages that LSTMs have over traditional RNNs.

Despite these advantages, LSTMs do have some drawbacks.

Bidirectional LSTMs are a type of recurrent neural network that is often used for natural language processing tasks. Unlike traditional LSTMs, which read input sequentially from left to right, bidirectional LSTMs are able to read input in both directions, allowing them to capture context from both the past and the future.

This makes them well-suited for tasks such as named entity recognition, where it is important to be able to identify entities based on their surrounding context. Bidirectional LSTMs are also sometimes used for machine translation, where they can help to improve the accuracy of the translation by taking into account words that appear later in the sentence.

For a comprehensive look into the world of LSTM, it is advisable to get enrolled in a MongoDB Certification course and learn everything you need to know about these neural networks.

LSTM has been used to achieve state-of-the-art results in a wide range of tasks such as language modeling, machine translation, image captioning, and more.

One of the most common applications of LSTM is language modeling. Language modeling is the task of assigning a probability to a sequence of words. In order to do this, LSTM must learn the statistical properties of language so that it can predict the next word in a sentence.

Another common application of LSTM is a machine translation. Machine translation is the process of translating one natural language into another. LSTM has been shown to be effective for this task because it can learn the long-term dependencies that are required for accurate translations.

Handwriting recognition is the task of automatically recognizing handwritten text from images or scanned documents. This is a difficult task because handwritten text can vary greatly in terms of style and quality, and there are often multiple ways to write the same word. However, because LSTMs can remember long-term dependencies between strokes, they have been shown to be effective for handwriting recognition tasks.

LSTM can also be used for image captioning. Image captioning is the task of generating a textual description of an image. This is a difficult task because it requires understanding both the visual content of an image and the linguistic rules for describing images. However, LSTM works well at image captioning by learning how to interpret images and generate appropriate descriptions.

Attention models are a type of neural network that can learn to focus on relevant parts of an input when generating an output. This is especially useful for tasks like image generation, where the model needs to focus on different parts of the image at different times. LSTMs can be used together with attention models to generate images from textual descriptions.

LSTMs can also be used for question-answering tasks. Given a question and a set of documents, an LSTM can learn to select passages from the documents that are relevant to the question and use them to generate an answer. This task is known as reading comprehension and is an important testbed for artificial intelligence systems.

Recently, Google released the SQuAD dataset, which contains 100,000+ questions answered by crowd workers on a set of Wikipedia articles. A number of different neural networks have been proposed for tackling this challenge, and many of them use LSTMs in some way or another.

Video-to-text conversion is the task of converting videos into transcripts or summaries in natural language text. This is a difficult task because it requires understanding both the audio and visual components of the video in order to generate accurate text descriptions. LSTMs have been used to develop successful video-to-text conversion systems.

Polyphonic music presents a particular challenge for music generation systems because each note must be generated independently while still sounding harmonious with all the other notes being played simultaneously. One way to tackle this problem is to use an LSTM network trained on polyphonic music data. This approach has been shown to generate convincing polyphonic music samples that sound similar to human performances.

Speech synthesis systems typically use some form of acoustic modeling in order to generate speech waveforms from text input. Recurrent neural networks are well suited for this task due to their ability to model sequential data such as speech signals effectively.

Protein secondary structure prediction is another important application of machine learning in biology. Proteins are often described by their primary structure (the sequence of amino acids) and their secondary structure (the three-dimensional shape).

Secondary structure prediction can be viewed as a sequence labeling task, where each residue in the protein sequence is assigned one of three labels (helix, strand, or coil). Long Short Term Memory networks have been shown to be effective at protein secondary structure prediction, both when used alone and when used in combination with other methods such as support vector machines.

LSTMs are not perfect, however, and there are certain limitations to their abilities. Here, we'll explore some of those limitations and what they mean for the future of artificial intelligence.

One of the biggest limitations of LSTMs is their inability to handle temporal dependencies that are longer than a few steps. This was demonstrated in a paper published by Google Brain researchers in 2016. The researchers found that when they trained an LSTM on a dataset with long-term dependencies (e.g., 100 steps), the network struggled to learn the task and generalize to new examples.

This limitation arises because LSTMs use a forget gate to control what information is kept in the cell state and what is forgotten. However, this gate can only forget information that is a few steps back; anything further back is forgotten completely. As a result, LSTMs struggle to remember dependencies that are many steps removed from the current input.

There are two possible ways to address this limitation: either train a larger LSTM with more cells (which requires more data) or use a different type of neural network altogether. Researchers from DeepMind recently proposed a new type of recurrent neural network called the Neural Stack Machine, which they claim can learn temporal dependencies of arbitrary length.

However, it remains to be seen whether this model will be able to scale to large datasets and complex tasks like machine translation and automatic question answering.

Another limitation of LSTMs is their limited context window size. A context window is the set of inputs that the network uses to predict the next output; for instance, in a language model, an input might be a sequence of words while the output is the next word in the sentence. The size of the context window is determined by the number of recurrent units in the LSTM; typically, this number is between 2 and 4.

This means that an LSTM can only consider a limited number of inputs when making predictions; anything outside of the context window is ignored completely. This can be problematic for tasks like machine translation, where it's important to consider the entire input sentence (not just the last few words) in order to produce an accurate translation.

There are two possible ways to address this limitation as well: either train a larger LSTM with more cells (which requires more data) or use Attention-based models instead, which have been shown to be better at handling long input sequences. However, both of these methods come with their own trade-offs and challenges (e.g., Attention models usually require more training data). In case you feel these limitations are still in your way, then get in touch with the experts of KnowledgeHut’s Database Courses and solve all your problems with their professional expertise.

The advantage of the Long Short-Term Memory (LSTM) network over other recurrent networks back in 1997 came from an improved method of back propagating the error. Hochreiter and Schmidhuber called it “constant error back propagation” .

But what does it mean to be “constant”? We’ll go through the architecture of the LSTM and understand how it forward and back propagates to answer the question. We will make some comparisons to the Recurrent Neural Network (RNN) along the way. If you are not familiar with the RNN, you may want to read about it here.

However, we should first understand what is the issue of the RNN that demanded for the solution the LSTM presents. That issue is the exploding and vanishing of the gradients that comes from the backward propagation step.

Back propagation is the propagation of the error from its prediction up until the weights and biases. In recurrent networks like the RNN and the LSTM this term was also coined Back Propagation Through Time (BPTT) since it propagates through all time steps even though the weight and bias matrices are always the same.

The figure above depicts a portion of a typical RNN with two inputs. The green rectangle represents the feed forward calculation of net inputs and their hidden state activations using an hyperbolic tangent (tanh) function. The feed forward calculations use the same set of parameters (weight and bias) in all time steps.

In red we see the BPTT path. For large sequences one can see that the calculations stack. This is important because it creates an exponential factor that depends greatly on the values of our weights. Everytime we go back a time step, we need to make an inner product between our current gradient and the weight matrix.

We can imagine our weight matrix to be a scalar and let’s say that the absolute scalar is either around 0.9 or 1.1. Also, we have a sequence as big as 100 time steps. The exponential factor created by multiplying these values one hundred times would raise a vanishing gradient issue for 0.9:

and an exploding gradient issue for 1.1:

Essencially, the BPTT calculation at the last time step would be similar to the following:

Note that although the representation is not completely accurate, it gives a good idea of the exponential stacking of the weight matrices in the BPTT of an RNN with n inputs. W_h is the weight matrix of the last linear layer of the RNN.

Next, we would be adding a portion of these values to the weight and bias matrices. You can see that we either barely improve the parameters, or try to improve so much that it backfires.

Now that we understand these concepts of vanishing and exploding gradients, we can move on to learn the LSTM. Let’s start by its forward pass.

Despite the differences that make the LSTM a more powerful network than RNN, there are still some similarities. It mantains the input and output configurations of one-to-one, many-to-one, one-to-many and many-to many. Also, one may choose to use a stacked configuration.

Above we can see the forward propagation inside an LSTM cell. It is considerably more complicated than the simple RNN. It contains four networks activated by either the sigmoid function (σ) or the tanh function, all with their own different set of parameters.

Each of these networks, also refered to as gates, have a different purpose. They will transform the cell state for time step t (c^t) with the relevant information that should be passed to the next time step. The orange circles/elipse are element-wise transformations of the matrices that preceed them. Here’s what the gates do:

Forget gate layer (f): Decides which information to forget from the cell state using a σ function that modulates the information between 0 and 1. It forgets everything that is 0, remembers all that is 1 and everything in the middle are possible candidates.

Input gate layer (i): This could also be a remember gate. It decides which of the new candidates are relevant for this time step also with the help of a σ function.

New candidate gate layer (n): Creates a new set of candidates to be stored in the cell state. The relevancy of these new candidates will be modulated by the element-wise multiplication with the input gate layer.

Output gate layer (o): Determines which parts of the cell state are output. The cell state is normalized through a tanh function and is multiplied element-wise by the output gate that decides which relevant new candidate should be output by the hidden state.

On the left you can see the calculations performed inside an LSTM cell. The last two calculations are an external feed forward layer to obtain a prediction and some loss function that takes the prediction and the true value.

The entire LSTM network’s architecture is built to deal with a three time step input sequence and to forecast a time step into the future like shown in the following figure:

Putting the inputs and parameters into vector and matrix form may help understand the dimansionality of the calculations. Note that we are using four weight and bias matrices with their own values.

This is the forward propagation of the LSTM. Now it is time to understand how the network back propagates and how it shines compared to the RNN.

The improved learning of the LSTM allows the user to train models using sequences with several hundreds of time steps, something the RNN struggles to do.

Something that wasn’t mentioned when explaining the gates is that it is their job to decide the relevancy of information that is stored in the cell and hidden states so that, when back propagating from cell to cell, the passed error is as close to 1 as possible. This ensures that there is no vanishing or exploding of gradients.

Another simpler way of understanding the process is that the cell state connects the layers inside the cell with information that stabilizes the propagation of error somewhat like a ResNet does.

Let’s see how the error is kept constant by going through the back propagation calculations. We’ll start with the linear output layer.

Now we’ll go about the LSTM cell’s back propagation. But first let’s get a visual of the path we must take within a cell.

As you can see, the path is quite complicated, which makes for computationally heavier operations than the RNN.

Bellow you can see the back propagation of both outputs of an LSTM cell, the cell state and the hidden state. You can refer to the equations I showed above for the forward pass to get a better understanding of which equations we are going through.

You can see that the information that travelled forward through the cell state is now going backwards modulated by the tanh’. Note that the prime (‘) in σ’ and tanh’ represent the first derivative of both these functions.

In the next steps, we are going back to the parameters in each gate.

Output gate:

New candidate gate:

Input gate:

Forget gate:

We have calculated the gradient for all the parameters inside the cell. However, we need to keep back propagating until the last cell. Let’s see the last steps:

You may see that the information that travels from cell state c³ to c² largelly depends on the outputs of the output gate and the forget gate. At the same time, the output and forget gradients depend on the information that was previously stored in the cell states. These interactions should provide the constant error back propagation.

Going further back into the global input (X³), we add what is coming from all four gates together.

Finally we deconcatenate the hidden state from the global input vector, go through the remaining cells and add all the gradients with respect to the parameters from all cells together.

This story’s goal was to understand why the LSTM is capable of dealing with more complex problems than the RNN by keeping a constant flow of error throughout the backpropagation from cell to cell.

We explored the resulting issues from the poor handling of complex sequences from the RNN giving raise to the exploding and vanishing gradients.

Then we saw how these issues come to happen by exploring the flow of gradients in the RNN.

Finally we introduced the LSTM, its forward pass and, by deconstructing its backward pass, we understood that the cell state is influenced by two gate units that are responsible for ensuring a constant back flow of the error.

It is important to mention that as more experiments were performed with the LSTM there is a certain degree of complexity where this network stops being able to learn. Generally it goes to the thousand time steps before it happens which is already pretty good.

This is leading to a gradual phase out of the LSTM as problems become more ambitious in favour of a newer network called the Transformer or BERT. You may have also heard of the GTP-3 for Natural Language Processing. These are very powerful networks with a great potential.

However, the LSTM sure had its impact and was created with ingenuity and still is usefull today.

Artificial neural networks (ANN) are feedforward networks that take inputs and produce outputs, whereas RNNs learn from previous outputs to provide better results the following time. Apple's Siri and Google's voice search algorithm are exemplary applications of RNNs in machine learning.

The input and output of standard ANNs are interdependent. However, the output of an RNN is reliant on the previous nodes in the sequence.

Each neuron in a feed-forward network or multi-layer perceptron executes its function with inputs and feeds the result to the next node.

As the name implies, recurrent neural networks have a recurrent connection in which the output is transmitted back to the RNN neuron rather than only passing it to the next node.

Each node in the RNN model functions as a memory cell, continuing calculation and operation implementation.

If the network's forecast is inaccurate, the system self-learns and performs backpropagation toward the correct prediction.

An RNN remembers every piece of information throughout time. It is only effective in time series prediction because of the ability to recall past inputs. This is referred to as long short-term memory (LSTM, explained later in this blog).

Recurrent neural networks combine with convolutional layers to widen the effective pixel neighborhood.

Convolutional neural networks (CNNs) are close to feedforward networks in that they are used to recognize images and patterns.

These networks use linear algebra concepts, namely matrix multiplication, to find patterns in images. However, they have some drawbacks, like

CNN does not encode spatial object arrangement. Its inability to be spatially invariant to incoming data

Besides, here’s a brief comparison of RNN and CNN.

CNN analyses the image data. The sequence data is processed using RNN.

In CNN, the input length is fixed. RNN input length is never set in machine learning.

CNN has more characteristics than other neural networks in terms of performance. When compared to CNN, RNN has fewer features.

CNN has no repetitive/recurrent connections. RNN uses recurrent connections to generate output.

Some of the downsides of RNN in machine learning include gradient vanishing and explosion difficulties. To tackle this problem LSTM neural network is used.

LSTM is a type of RNN with higher memory power to remember the outputs of each node for a more extended period to produce the outcome for the next node efficiently.

LSTM networks combat the RNN's vanishing gradients or long-term dependence issue.

Gradient vanishing refers to the loss of information in a neural network as connections recur over a longer period.

In simple words, LSTM tackles gradient vanishing by ignoring useless data/information in the network.

For example, if an RNN is asked to predict the following word in this phrase, "have a pleasant _______," it will readily anticipate "day."

The input data is very limited in this case, and there are only a few possible output results.

What if the sentence is stretched a little further, which can confuse the network - "I am going to buy a table that is large in size, it’ll cost more, which means I have to ______ down my budget for the chair," now a human brain can quickly fill this sentence with one or two of the possible words.

But we are talking about artificial intelligence here. As there are many inputs, the RNN will probably overlook some critical input data necessary to achieve the results.

Here – the critical data/input is – “______ down the budget” now, the machine has to predict which word is suitable before this phrase and look into previous words in the sentence to find any bias for the prediction.

If there is no valuable data from other inputs (previous words of the sentence), LSTM will forget that data and produce the result “Cut down the budget.”

The forget gate, input gate, and output gate are the three gates that update and regulate the cell states in an LSTM network.

Given new information that has entered the network, the forget gate determines which information in the cell state should be overlooked.

As a result, LSTM assists RNN in remembering the critical inputs needed to generate the correct output.

RNNs are categorized based on the four network sequences, namely,

The one-to-one RNN is a typical sequence in neural networks, with only one input and one output. Application – Image classification

One to Many network has a single input feed into the node, producing multiple outputs. Application – Music generation, image captioning, etc.

Many to One architecture of RNN is utilized when there are several inputs for generating a single output. Application – Sentiment analysis, rating model, etc.

LSTM networks combat the RNN's vanishing gradients or long-term dependence issue. Gradient vanishing refers to the loss of information in a neural network as connections recur over a longer period. In simple words, LSTM tackles gradient vanishing by ignoring useless data/information in the network.