3.9 Hyperparameters

We have seen that in the x-nearest neighbor algorithm it is necessary to fix the value of the number k in advance, and that different choices lead to different conclusions. How do we know which value to choose? This question follows all other machine learning algorithms in which some values appear that we need to define in advance. Such values are called hyperparameters or metaparameters .

 We mentioned that when dividing a dataset, we always single out a training set, a testing set, and a validation set. We have not used the validation set so far. In fact, we need it whenever there are some hyperparameters in our learning algorithm whose best value we need to determine. The story we're going to share applies to all algorithms, but we're going to continue to use the k-nearest neighbor algorithm.

 Let's go back to the question of how to choose the best value of the hyperparameter k. It is natural to think: we will try multiple values, for example, all numbers from 1 to 10, and then we will choose the best value! We're going to do that, but we're going to be very careful about where we try how good our choice is. If we do this on a test suite, we will be breaking the golden rule of machine learning about the strict separation of the test suite and model development: we will use the test suite to decide what the best value of the hyperparameter k is, and then, when we train the model, we will again use the test suite to evaluate how good it is! You'll agree that it doesn't make much sense!

 It is correct to do the following: we will test which hyperparameter values are best on the validation set. This set does not share information with either the training set or the test set, so it will contribute to the objectivity of our conclusions. Now that we have established this, we can get to work determining the best value of the hyperparameter k.

 For each of the values of the hyperparameter k that we want to test, we will separately train the model on the training set and calculate its quality measure on the validation set. In this case, it's accurate. The obtained values can be displayed graphically by placing different values of the parameter k along the x-axis, and accuracy values along the y-axis. The value of the hyperparameter k for which we get the best value of the quality measure on the validation set is the value of the hyperparameter we are looking for. This is usually seen in the graph as the region where the values are the highest.

Demonstrating the accuracy of the model on the validation set

Based on the previous graph, we can see that the optimal values of the hyperparameter k are actually 9, 10, 11, 12, and 13 because they all result in the same, highest accuracy of the model.

 Similar graphics can be drawn for hyperparameter values and error functions. Then we set different values of the hyperparameter along the x-axis, and the values of the error function along the y-axis. Now it is important to note the values of the hyperparameter for which the error function is smallest.

 When multiple hyperparameters are present in a learning algorithm, the goal is to find the best combination of hyperparameters. We also determine it based on the validation set by tracking the success of the model and hunting for the combination that gives the best value of the quality measure (or, equally, tracking the error of the model and hunting for the combination that gives the least value of error). The trouble is that this process can be quite slow and computationally demanding for a large number of hyperparameters: for example, if we want to examine 10 different values of k and 3 different distance functions, we actually have 10 x 3 = 30 different combinations, so we have to train and evaluate 30 different models.