Page Summary

outlined_flag

• L2 regularization is a technique used to reduce model complexity and prevent overfitting by penalizing large weights.

• A regularization rate (lambda) controls the strength of regularization, with higher values leading to simpler models and lower values increasing the risk of overfitting.

• Early stopping is an alternative regularization method that involves ending training before the model fully converges to prevent overfitting.

• Finding the right balance between learning rate and regularization rate is crucial for optimal model performance, as they influence weights in opposite directions.

L₂ regularizationis a popular regularization metric, which uses the following formula:

For example, the following table shows the calculation of L₂regularization for a model with six weights:

Notice that weights close to zero don't affect L₂ regularizationmuch, but large weights can have a huge impact. For example, in thepreceding calculation:

• A single weight (w₃) contributes about 93% of thetotal complexity.
• The other five weights collectively contribute only about 7% of thetotal complexity.

L₂ regularization encourages weightstoward 0, but never pushesweights all the way to zero.

Exercises: Check your understanding

Regularization rate (lambda)

As noted, training attempts to minimize some combination of loss and complexity:

Model developers tune the overall impact of complexity on model trainingby multiplying its value by a scalar called theregularization rate.The Greek character lambda typically symbolizes the regularization rate.

That is, model developers aim to do the following:

A high regularization rate:

• Strengthens the influence of regularization, thereby reducing the chances ofoverfitting.
• Tends to produce a histogram of model weights having the followingcharacteristics:
  • a normal distribution
  • a mean weight of 0.

A low regularization rate:

• Lowers the influence of regularization, thereby increasing the chances ofoverfitting.
• Tends to produce a histogram of model weights with a flat distribution.

For example, the histogram of model weights for a high regularization ratemight look as shown in Figure 18.

In contrast, a low regularization rate tends to yield a flatter histogram, asshown in Figure 19.

Early stopping is aregularization method that doesn't involve a calculation of complexity.Instead, early stopping simply means ending training before the modelfully converges. For example, you end training when the loss curvefor the validation set starts to increase (slope becomes positive).

Although early stopping usually increases training loss, it can decreasetest loss.

Early stopping is a quick, but rarely optimal, form of regularization.The resulting model is very unlikely to be as good as a model trainedthoroughly on the ideal regularization rate.

Finding equilibrium between learning rate and regularization rate

Learning rate andregularization rate tend to pull weights in oppositedirections. A high learning rate often pulls weightsaway from zero;a high regularization rate pulls weightstowards zero.

If the regularization rate is high with respect to the learning rate,the weak weights tend to produce a model that makes poor predictions.Conversely, if the learning rate is high with respect to the regularizationrate, the strong weights tend to produce an overfit model.

Your goal is to find the equilibrium between learning rate andregularization rate. This can be challenging. Worst of all, once you findthat elusive balance, you may have to ultimately change the learning rate.And, when you change the learning rate, you'll again have to find the idealregularization rate.

• Early stopping
• L₁ regularization
• L₂ regularization
• Learning rate
• Regularization
• Regularization rate