2.9 Training, validation and testing sets

After the data is analyzed and the appropriate instances and attributes are selected, the set of all data is divided into a Training set (trainable) and Testing set. As you can guess from the name of the set, the training set is used to train the machine learning model itself. A selected algorithm is applied to it and creates the model itself. A test suite is used to test the model, i.e. calculation of suitable measures of model quality. Thanks to him, one can objectively assess how well the model has learned the necessary task. Typically, we also use a portion of the baseline dataset data to create a validation set. A validation kit is used to track the process of training a model and determine some model configurations that lead to better quality measures. These topics will be discussed later in the course.

Dividing a dataset into a training set, a validation set, and a testing set

Training, validation, and testing sets are typically created by randomly dividing an underlying dataset. First, we define how large these sets should be, and then randomly select the instances that will be found in each of them. Typically, the training set is the largest, while the testing and validation sets are smaller because we want to have enough data to train the model, but also enough data to adequately evaluate its performance. The practice is that the size ratios of these sets are expressed in proportion. For example, you will often find the ratio of the training set to the test set expressed as 2:1, which would mean that two-thirds of the starting set is a training set and one-third is a test set. Similarly, a ratio of 2:1:1 would mean that two-quarters (i.e., one-half) of the baseline set would be used as a training kit and one-quarter each as a validation and testing set.

While it's convenient that train, validation, and testing sets are created randomly, it would still mean some insight into how this division was done. For example, when we want to replicate an experiment or allow others to run it on their own (this is an important property of experiments and is called reproducibility), it's preferable to use the same training sets, validation and testing. Similarly, when we solve a problem, we are not immediately sure what is best to do, so we try a larger number of algorithms and create a larger number of models. For the sake of fairness of comparison, it would mean that we create all models over the same training set and evaluate over the same test set. That's why it's a good idea to set a parameter at the level of the library that affects the randomness of division (usually called random seed and has the same purpose as setting seeds in a random number generator) or simply split the data from scratch and continue to use it consistently. Some commonly used datasets have these predefined divisions into a set for training, validation, and testing (for example, you can look at the MNIST set).

 An important property that train, validation, and testing suites need to meet is that they be disjointed. This means that each instance of the source dataset when creating training, validation, and testing sets must belong to exactly one of these sets, and there must be no intersections and shared instances. It should be remembered that machine learning models are expected to generalize well, i.e. that they behave well for new instances that the model has not had the opportunity to meet in the training set. If the training sets and test sets overlap, we will not be able to objectively assess whether the model actually learns or memorizes. remembers information from the training set. The same is true of the relationship between the training set and the validation set: the function of the validation set is to help select the configurations that will make the learning as successful as possible. If these sets overlap, we will not be able to objectively and impartially evaluate the behavior of the model and select suitable configurations.

 The requirement that training, validation and testing sets should be disjoint also means that information from one of the sets must not spill over into the other sets. This is especially important when applying preprocessing and set preparation techniques. Let's consider the following example. We mentioned that due to the sensitivity of the machine learning model to the value of attributes, standardization of numerical attributes is often performed. One can approach this transformation by calculating the mean and standard deviation based on the entire set, and then using it, in turn, to standardize the training and testing sets. In order not to overflow information, the correct sequence of these steps is actually as follows:

• dividing the dataset into a training and testing suite,
• calculation of the mean and standard deviation only on the training set,
• transformation of the training set using calculated values,
• Transform the test set using the values computed over the training set.

Because of the desire not to make mistakes, one may also think as follows: after dividing the initial dataset into a training set and a test set, I will perform a separate standardization of the training set and the test set. And this approach, although more cautious, is not correct because it leads to the modification of the test set. In the lower left image, the yellow triangles represent the instances of the training set, and the blue circles represent the instances of the test set. The image in the middle represents these instances after correct standardization (you can carefully compare the images and the arrangement of the points - the scale along the x-axis has changed due to standardization, everything else has remained the same). In the image on the right you can see the instances after the standardization has been done separately over the training set and the test set - the spatial arrangement has changed quite a bit now.

Examples of Right and Wrong Standardization

When dividing the baseline dataset, it would be ideal to preserve the proportions relative to the attribute values and the value of the target variable. For example, if the ratio of male to female patients in the medical dataset is 4:5, it would be ideal that, after the division, the ratio of patients in the training set and in the test set should be approximately 4:5. Techniques that allow this type of division are called stratification techniques. However, due to the number of attributes and their combinations, in practice this is often not a realistic requirement, so most often it is insisted on proportionality in relation to the values of the target variable. We will discuss this topic separately in the context of the task of classification.

Stratified Training and Testing Assemblies

Show answer

Show answer