3.12 Regularization

Regularizations are another set of techniques that can be used to control model refits. Their main goal is to prevent complex models, which help us learn a richer set of dependencies in data, from becoming over-adapted.

 We will introduce regularization using the example of a linear regression model. Suppose we have trained the model and we have obtained the values of the parameters, the graphical representation of which looks like in the figure.

 The parameters that are the largest (absolute) in terms of their value are also the most important for model predictions. In the figure, these are the parameters that correspond to attributes 3 and 5 and their values, as we can see, are significantly higher than the values of the other parameters. In this sense, these attributes can ignore the impact of the remaining attributes on the prediction values, so we can interpret this behavior of the model as a form of data readjustment.

 That is why it is desirable, to some extent, to limit the values of the parameters - we want the model to learn the parameters and that they reflect the properties of the data, but we also want to monitor their value in order to prevent overadjustment. This technique is called regularization. In the context of linear regression, we can do this by adding the sum of the squares of the parameters to the mean square error of the model:

\( \frac{1}{N}\sum\limits_{i = 1}^{N}(y_i −(\beta_0 + \beta_1x_1 + \beta_2x_2 +...+ \beta_n x_n))^2 + λ(\beta_1^2 + \beta_2^2 +...+ \beta_n^2) \)

The value of \(λ\) in the expression is a hyperparameter that affects the strength of the regularization. If its value is 0, the regularization will have no effect. By giving some non-zero values, we balance the learning determined by the mean square error and the readjustment measured by the values of the sum of the squares of the parameters. Squares are there for technical reasons, first to prevent the values of the coefficients from being suppressed against each other, and then to preserve the properties of the error function for the application of the optimization algorithm. Such an extended form of linear regression supplemented by a regularization term is called ridge regression.

 A little later, we will return to the story of regularization when we introduce neural networks. They are very complex models, so they can often be re-adapted to the data. We'll also see how we can follow it.