3.11 Validation, cross-validation

We've said many times that before applying a machine learning algorithm, data is divided into a training set and a testing set (we include the validation set when we need it). We also mentioned that the division process is random. You may have wondered if some different divisions, compared to the ones we have chosen, would lead to different results of the model's work. Perhaps it is for a specific division of the data set that we get more optimistic results or drastically worse. And it's a kind of adjustment.

Whenever the sizes of the datasets and the selected algorithms allow, it is desirable to actually perform multiple divisions of the initial dataset into a training set and a testing set so that each instance in the dataset gets the opportunity to be found in both sets. One such procedure that we will describe is called cross-validation. In the example, we will use a linear regression algorithm, but the story is general and applies to all algorithms.

Let's divide the dataset into 10 parts as in the figure below. In the first step, we extract the first part of the test set and keep the remaining nine parts for training. To make it easier for you to follow, the test set is colored yellow in the image, and the training sets are colored blue. Now let's train the first linear regression model on the training set and calculate the value of its mean square error on the test set. The resulting value can be marked with MSE₁ . In step two, we separate the second part of the test set, and the remaining nine parts for the training set. Now in the picture the second part is painted yellow, and the remaining parts are painted blue. Let's retrain the linear regression model on the training set (that's the second model now) and calculate the value of its mean square error on the test set. Now let's mark this value with MSE₂ . Let's continue this process until we get to the last, tenth, part: now we will keep it as a test set, and we will use the remaining parts to train the model. We will train the tenth linear regression model on it and then calculate the mean square error MSE₁₀ on the test set.

Cross-validation with 10 layers

Since we have 10 different divisions of the data set, we also have 10 different values of the mean square error. The average of the obtained values (MSE₁ + MSE₂ + ... + MSE₁₀ )/10 actually best indicates how our model behaves and helps us solve the dilemmas we had at the beginning regarding the impact of division on the success of the model. What is not very clear is which of the 10 different models we have at our disposal should we choose. The slightest mistake or some other? In fact, we should now train a new model over the entire dataset and continue to use it we approximate its behavior and rate it with the behaviors of each of the 10 trained models.

 This process is called 10-layer cross-validation10-fold cross validation. In practice, 3- and 5-layer divisions are also used, and the choices depend on the size of the dataset and the type of algorithms used. Also, there is a division in which the number of layers corresponds to the number of instances in the data set, called leave-one-out cross validation.