Recurrent neural networks

Recurrent Neural Networks (Neural Networks) are a type of neural networks that are primarily used to process sequential data. Sequential data or sequences are made up of elements that follow one another. Such are, for example, textual data (elements are individual words), audio recordings (elements are individual samples), time series (elements are individual measurements), genetic sequences (elements are individual nucleotides), and many others. Recurrent networks process sequence element by element. In order to be able to process an element at position t , all the elements that precede it must be processed, and in order for the elements of the sequence to be connected into a single whole, the values of the hidden layers are divided between the processing of successive input elements. This is usually shown graphically as in the figure below.

Recurrent neural network

(image taken from https://d2l.ai/chapter_recurrent-neural-networks/index.html )

Recurrent neural networks are hypothetically capable of processing infinitely long sequences: element by element. However, when training such networks, it has been observed that they forget. If the sequences are too long, the network begins to forget what it saw at the beginning and stores the recently seen information at the level of hidden layers. This observation led to the design of special neurons called LSTM (LSTM). Long Short Term Memory ) and GRU (eng. Gated Recurrent Unit ), and we will not go into details due to its complexity. A solution to this problem is two-way recurrent neural networks (neural networks). Bidirectional Recurrent Neural Networks ): In these networks, on the one hand, the sequence is processed from beginning to end, and on the other hand, from end to beginning. The input representation of the individual elements represents the contiguous representations of these passages, illustrated as in the figure below.

Two-way recurrent neural network - sequential elements

(image taken from https://www.arxiv-vanity.com/papers/1506.00019/ )

There are several popular architectures of recurrent neural networks. In the table below, we'll briefly go through some of the most popular examples by showing them graphically in the left column and describing the network and application areas in the right column.

ARCHITECTURE

CLARIFICATION AND EXAMPLES OF APPLICATION

This type of network corresponds to tasks in which the input is a sequence and the output is a vector representation of a fixed length. Networks of this type are called encoders . encoders ) and the obtained vectors of fixed lengths by context . The tasks in which we encounter this type of networks are various classification tasks such as the classification of audio tracks or the classification of text.

Unlike the previous example, the input for this type of network is a vector representation of a fixed length and the output is a sequence. This type of network is called decoders . decoders). The tasks in which we encounter decoders are the generation of image titles.

This type of network is a combination of the previous two types and is called encoder-decoder architecture. The task of the encoder is to create a representation (context) based on the input sequence that the decoder can use to generate a new output sequence. This type of network is encountered in machine translation or abstract generation tasks.

This type of network allows the generation of outputs for each element of the input. As you can see, there are sequences at both the entrance and the exit. The tasks in which we encounter this type of mesh are, for example, the tasks of tagging (marking) individual elements.

One major drawback of recurrent neural networks is the inability to parallelize: in order to process an element at position t , all the elements that precede it must be processed. That is why training neural networks requires much more time and resources than training the convolutional neural networks that we got to know in the previous lesson. These circumstances have led to the emergence of the attention mechanism and transformers, Neural networks that will be discussed in more detail in the next lesson.

Audio recordings can also be processed using convolutional neural networks. Namely, an audio recording can be divided into fragments, shorter pieces that last a few seconds, and then spectrograms can be created for each part. A spectrogram is a graphical representation of all the frequencies of sound present in an audio recording. The resulting images can then be passed as inputs to convolutional neural networks and used for audio analysis.

An example of a spectrogram

Last modified: Saturday, 18 January 2025, 2:09 PM