4.2 Training of neural networks

As we have seen, each neuron of a neural network is connected in some way to other neurons in the network. These connections are described by weights w, which actually represent the parameters of the neural network that need to be learned during training. The number of parameters in a neural network is generally large. Let's say , for a fully connected neural network that has 5 neurons in the input layer, one hidden layer with 10 neurons and an output layer with 3 neurons, the number of parameters to be learned is 93. In practice, neural networks have thousands and millions of parameters, even billions! That is why a large amount of data is needed to train them.

Show answer

Neural networks, like other models, are trained on a training set and evaluated on a test set. Because neural networks are complex models that can learn complex relationships between attributes and outputs, they can easily be readapted to data. That is why we always use a validation set during the training of the network. It helps us to follow the course of training more finely and notice overadjustments and other undesirable properties of the model earlier.

 In the introduction of the course, we said that the unknown parameters of the model are determined by defining the error function, and then applying some optimization techniques (which include gradient descent) with the aim of finding those parameter values for which the error function is the smallest. This protocol follows the story of neural networks, but the error values are not calculated for individual instances, but for groups of instances. The motivation for this design is, First of all, working with a large amount of data and the need to parallelize and speed up the whole process. That is why all the data in the training set is first divided into packets (batch) of equal sizes. The packets are then passed through the network, one by one, and the value of the error function is calculated for them by comparing the expected and obtained values of the target variable. Then, in proportion to their contributions, the error values of the neural network are updated by going backwards through the network. The described process of updating network parameters is called backpropagation and allows us to refine the parameter values in iterations and reach the optimal parameter values. Otherwise, we're going to start at the beginning.

One passage through the entire data set, i.e. one processing of all packets of the training set, is called  an epoch. Neural networks are trained in several epochs. After one epoch is completed, the data is "shuffled", then divided into packets again and passed through the network. In which epoch the model will be trained, depends on the success of the training and the available resources. Due to working with large amounts of data, Networks need specialized hardware that can parallelize computations (for example, graphics cards or tensor cards), so training networks is often both expensive and time-consuming.

 This way of training the network through epochs allows us to follow the course of training more finely. At the end of each epoch, the model error on the training set and the model error on the validation set are calculated. These two values are then displayed on a graph that shows the epoch ordinal number along the x-axis and the error value along the y-axis. You can see one such graphic in the picture below. Good training is characterized by a comparative decrease in these values to a satisfactory error value - the closer we are to zero, the better the model. Recall that this conclusion is based on the fact that the validation set contains data that is separate from the training set and that the network sees for the first time.

If we notice that the values of the error function on the training set are decreasing and on the validation set are increasing, we conclude that the model is refitting and we stop training. Next, we have two options. If the values of the model error function in the epoch before the observed model refitting were satisfactory, we can keep that version of the model for further testing on the test set (usually, during the training of the network, several versions of the model are saved with the idea of using them for this purpose, or to be used if it is necessary to stop and continue the training process). Otherwise, we have to try a slightly different network architecture or a slightly different set of its hyperparameters. Given that each layer of the network has its own settings (number of neurons, activation function, initial set of parameters), that the layers can be connected in different ways, that we have to simultaneously monitor all the settings of the optimization algorithm, for example the gradient descent and its learning step, and that some expectations in terms of quality measures need to be met, training the network is a challenging and complex task. That is why it is said to represent  the art of coaching .

Monitoring of neural network readaptation based on error function value graphs on the training set and validation set