Inmathematics,statistics,finance,^[1] andcomputer science, particularly inmachine learning andinverse problems,regularization is a process that converts theanswer to a problem to a simpler one. It is often used in solvingill-posed problems or to preventoverfitting.^[2]

Although regularization procedures can be divided in many ways, the following delineation is particularly helpful:

• Explicit regularization is regularization whenever one explicitly adds a term to the optimization problem. These terms could bepriors, penalties, or constraints. Explicit regularization is commonly employed with ill-posed optimization problems. The regularization term, or penalty, imposes a cost on the optimization function to make the optimal solution unique.
• Implicit regularization is all other forms of regularization. This includes, for example, early stopping, using a robust loss function, and discarding outliers. Implicit regularization is essentially ubiquitous in modern machine learning approaches, includingstochastic gradient descent for trainingdeep neural networks, andensemble methods (such asrandom forests andgradient boosted trees).

In explicit regularization, independent of the problem or model, there is always a data term, that corresponds to a likelihood of the measurement, and a regularization term that corresponds to a prior. By combining both usingBayesian statistics, one can compute a posterior, that includes both information sources and therefore stabilizes the estimation process. By trading off both objectives, one chooses to be more aligned to the data or to enforce regularization (to prevent overfitting). There is a whole research branch dealing with all possible regularizations. In practice, one usually tries a specific regularization and then figures out the probability density that corresponds to that regularization to justify the choice. It can also be physically motivated by common sense or intuition.

Inmachine learning, the data term corresponds to the training data and the regularization is either the choice of the model or modifications to the algorithm. It is always intended to reduce thegeneralization error, i.e. the error score with the trained model on the evaluation set (testing data) and not the training data.^[3]

One of the earliest uses of regularization isTikhonov regularization (ridge regression), related to the method of least squares.

Inmachine learning, a key challenge is enabling models to accurately predict outcomes on unseen data, not just on familiar training data. Regularization is crucial for addressingoverfitting—where a model memorizes training data details but cannot generalize to new data. The goal of regularization is to encourage models to learn the broader patterns within the data rather than memorizing it. Techniques likeearly stopping, L1 andL2 regularization, anddropout are designed to prevent overfitting and underfitting, thereby enhancing the model's ability to adapt to and perform well with new data, thus improving model generalization.^[4]

Stops training when validation performance deteriorates, preventing overfitting by halting before the model memorizes training data.^[4]

Adds penalty terms to the cost function to discourage complex models:

• L1 regularization (also calledLASSO) leads to sparse models by adding a penalty based on the absolute value of coefficients.
• L2 regularization (also calledridge regression) encourages smaller, more evenly distributed weights by adding a penalty based on the square of the coefficients.^[4]

In the context of neural networks, the Dropout technique repeatedly ignores random subsets of neurons during training, which simulates the training of multiple neural network architectures at once to improve generalization.^[4]

Empirical learning of classifiers (from a finite data set) is always anunderdetermined problem, because it attempts to infer a function of any${\displaystyle x}$ given only examples${\displaystyle x_1,x_2,\dots ,x_n}$.

A regularization term (or regularizer)${\displaystyle R(f)}$ is added to aloss function:$${\displaystyle \underset{f}{min}\sum _{i=1}^{n}V(f(x_i),y_i)+\lambda R(f)}$$where${\displaystyle V}$ is an underlying loss function that describes the cost of predicting${\displaystyle f(x)}$ when the label is${\displaystyle y}$, such as thesquare loss orhinge loss; and${\displaystyle \lambda }$ is a parameter which controls the importance of the regularization term.${\displaystyle R(f)}$ is typically chosen to impose a penalty on the complexity of${\displaystyle f}$. Concrete notions of complexity used include restrictions forsmoothness and bounds on thevector space norm.^{[5][page needed]}

A theoretical justification for regularization is that it attempts to imposeOccam's razor on the solution (as depicted in the figure above, where the green function, the simpler one, may be preferred). From aBayesian point of view, many regularization techniques correspond to imposing certainprior distributions on model parameters.^[6]

Regularization can serve multiple purposes, including learning simpler models, inducing models to be sparse and introducing group structure^{[clarification needed]} into the learning problem.

The same idea arose in many fields ofscience. A simple form of regularization applied tointegral equations (Tikhonov regularization) is essentially a trade-off between fitting the data and reducing a norm of the solution. More recently, non-linear regularization methods, includingtotal variation regularization, have become popular.

Regularization can be motivated as a technique to improve the generalizability of a learned model.

The goal of this learning problem is to find a function that fits or predicts the outcome (label) that minimizes the expected error over all possible inputs and labels. The expected error of a function${\displaystyle f_n}$ is:$${\displaystyle I[f_n]={\int }_{X\times Y}V(f_n(x),y)\rho (x,y)dxdy}$$where${\displaystyle X}$ and${\displaystyle Y}$ are the domains of input data${\displaystyle x}$ and their labels${\displaystyle y}$ respectively.

Typically in learning problems, only a subset of input data and labels are available, measured with some noise. Therefore, the expected error is unmeasurable, and the best surrogate available is the empirical error over the${\displaystyle N}$ available samples:$${\displaystyle I_S[f_n]=\frac{1}{n}\sum _{i=1}^{N}V(f_n({\hat{x}}_i),{\hat{y}}_i)}$$Without bounds on the complexity of the function space (formally, thereproducing kernel Hilbert space) available, a model will be learned that incurs zero loss on the surrogate empirical error. If measurements (e.g. of${\displaystyle x_i}$) were made with noise, this model may suffer fromoverfitting and display poor expected error. Regularization introduces a penalty for exploring certain regions of the function space used to build the model, which can improve generalization.

These techniques are named forAndrey Nikolayevich Tikhonov, who applied regularization tointegral equations and made important contributions in many other areas.

When learning a linear function${\displaystyle f}$, characterized by an unknownvector${\displaystyle w}$ such that${\displaystyle f(x)=w\cdot x}$, one can add the${\displaystyle L_2}$-norm of the vector${\displaystyle w}$ to the loss expression in order to prefer solutions with smaller norms. Tikhonov regularization is one of the most common forms. It is also known as ridge regression. It is expressed as:$${\displaystyle \underset{w}{min}\sum _{i=1}^{n}V({\hat{x}}_i\cdot w,{\hat{y}}_i)+\lambda {\Vert w\Vert }_2^2,}$$where${\displaystyle ({\hat{x}}_i,{\hat{y}}_i),1\le i\le n,}$ would represent samples used for training.

In the case of a general function, the norm of the function in itsreproducing kernel Hilbert space is:$${\displaystyle \underset{f}{min}\sum _{i=1}^{n}V(f({\hat{x}}_i),{\hat{y}}_i)+\lambda {\Vert f\Vert }_{\mathcal{H}}^2}$$

As the${\displaystyle L_2}$ norm isdifferentiable, learning can be advanced bygradient descent.

The learning problem with theleast squares loss function and Tikhonov regularization can be solved analytically. Written in matrix form, the optimal${\displaystyle w}$ is the one for which the gradient of the loss function with respect to${\displaystyle w}$ is 0.$${\displaystyle \underset{w}{min}\frac{1}{n}{\left(\hat{X}w-Y\right)}^{\mathsf{T}}\left(\hat{X}w-Y\right)+\lambda {\Vert w\Vert }_2^2}$$$${\displaystyle {\mathrm{\nabla }}_w=\frac{2}{n}{\hat{X}}^{\mathsf{T}}\left(\hat{X}w-Y\right)+2\lambda w}$$$${\displaystyle 0={\hat{X}}^{\mathsf{T}}\left(\hat{X}w-Y\right)+n\lambda w}$$$${\displaystyle w={\left({\hat{X}}^{\mathsf{T}}\hat{X}+\lambda nI\right)}^{-1}\left({\hat{X}}^{\mathsf{T}}Y\right)}$$where the third statement is afirst-order condition.

By construction of the optimization problem, other values of${\displaystyle w}$ give larger values for the loss function. This can be verified by examining thesecond derivative${\displaystyle {\mathrm{\nabla }}_{ww}}$.

During training, this algorithm takes${\displaystyle O(d^3+n{d}^2)}$time. The terms correspond to the matrix inversion and calculating${\displaystyle X^{\mathsf{T}}X}$, respectively. Testing takes${\displaystyle O(nd)}$ time.

Early stopping can be viewed as regularization in time. Intuitively, a training procedure such as gradient descent tends to learn more and more complex functions with increasing iterations. By regularizing for time, model complexity can be controlled, improving generalization.

Early stopping is implemented using one data set for training, one statistically independent data set for validation and another for testing. The model is trained until performance on the validation set no longer improves and then applied to the test set.

Consider the finite approximation ofNeumann series for an invertible matrixA where:$${\displaystyle \sum _{i=0}^{T-1}{\left(I-A\right)}^i\approx A^{-1}}$$

This can be used to approximate the analytical solution of unregularized least squares, ifγ is introduced to ensure the norm is less than one.$${\displaystyle w_T=\frac{\gamma }{n}\sum _{i=0}^{T-1}{\left(I-\frac{\gamma }{n}{\hat{X}}^{\mathsf{T}}\hat{X}\right)}^i{\hat{X}}^{\mathsf{T}}\hat{Y}}$$

The exact solution to the unregularized least squares learning problem minimizes the empirical error, but may fail. By limitingT, the only free parameter in the algorithm above, the problem is regularized for time, which may improve its generalization.

The algorithm above is equivalent to restricting the number of gradient descent iterations for the empirical risk$${\displaystyle I_s[w]=\frac{1}{2n}{\Vert \hat{X}w-\hat{Y}\Vert }_{{\mathbb{R}}^n}^2}$$with the gradient descent update:

The base case is trivial. The inductive case is proved as follows:

Assume that a dictionary${\displaystyle {\varphi }_j}$ with dimension${\displaystyle p}$ is given such that a function in the function space can be expressed as:$${\displaystyle f(x)=\sum _{j=1}^p{\varphi }_j(x)w_j}$$

Enforcing a sparsity constraint on${\displaystyle w}$ can lead to simpler and more interpretable models. This is useful in many real-life applications such ascomputational biology. An example is developing a simple predictive test for a disease in order to minimize the cost of performing medical tests while maximizing predictive power.

A sensible sparsity constraint is the${\displaystyle L_0}$ norm${\displaystyle \Vert w{\Vert }_0}$, defined as the number of non-zero elements in${\displaystyle w}$. Solving a${\displaystyle L_0}$ regularized learning problem, however, has been demonstrated to beNP-hard.^[7]

The${\displaystyle L_1}$ norm (see alsoNorms) can be used to approximate the optimal${\displaystyle L_0}$ norm via convex relaxation. It can be shown that the${\displaystyle L_1}$ norm induces sparsity. In the case of least squares, this problem is known asLASSO in statistics andbasis pursuit in signal processing.$${\displaystyle \underset{w\in {\mathbb{R}}^p}{min}\frac{1}{n}{\Vert \hat{X}w-\hat{Y}\Vert }^2+\lambda {\Vert w\Vert }_1}$$

${\displaystyle L_1}$ regularization can occasionally produce non-unique solutions. A simple example is provided in the figure when the space of possible solutions lies on a 45 degree line. This can be problematic for certain applications, and is overcome by combining${\displaystyle L_1}$ with${\displaystyle L_2}$ regularization inelastic net regularization, which takes the following form:$${\displaystyle \underset{w\in {\mathbb{R}}^p}{min}\frac{1}{n}{\Vert \hat{X}w-\hat{Y}\Vert }^2+\lambda \left(\alpha {\Vert w\Vert }_1+(1-\alpha ){\Vert w\Vert }_2^2\right),\alpha \in [0,1]}$$

Elastic net regularization tends to have a grouping effect, where correlated input features are assigned equal weights.

Elastic net regularization is commonly used in practice and is implemented in many machine learning libraries.

While the${\displaystyle L_1}$ norm does not result in an NP-hard problem, the${\displaystyle L_1}$ norm is convex but is not strictly differentiable due to the kink at x = 0.Subgradient methods which rely on thesubderivative can be used to solve${\displaystyle L_1}$ regularized learning problems. However, faster convergence can be achieved through proximal methods.

For a problem${\displaystyle \underset{w\in H}{min}F(w)+R(w)}$ such that${\displaystyle F}$ is convex, continuous, differentiable, with Lipschitz continuous gradient (such as the least squares loss function), and${\displaystyle R}$ is convex, continuous, and proper, then the proximal method to solve the problem is as follows. First define theproximal operator$${\displaystyle {\mathrm{prox}}_R(v)=\underset{w\in {\mathbb{R}}^D}{\mathrm{argmin}}\left\{R(w)+\frac{1}{2}{\Vert w-v\Vert }^2\right\},}$$and then iterate$${\displaystyle w_{k+1}=\underset{\gamma ,R}{\mathrm{prox}}\left(w_k-\gamma \mathrm{\nabla }F(w_k)\right)}$$

The proximal method iteratively performs gradient descent and then projects the result back into the space permitted by${\displaystyle R}$.

When${\displaystyle R}$ is theL₁ regularizer, the proximal operator is equivalent to the soft-thresholding operator,

This allows for efficient computation.

Groups of features can be regularized by a sparsity constraint, which can be useful for expressing certain prior knowledge into an optimization problem.

In the case of a linear model with non-overlapping known groups, a regularizer can be defined:$${\displaystyle R(w)=\sum _{g=1}^G{\Vert w_g\Vert }_2,}$$ where$${\displaystyle \Vert w_g{\Vert }_2=\sqrt{\sum _{j=1}^{|G_g|}{\left(w_g^j\right)}^2}}$$

This can be viewed as inducing a regularizer over the${\displaystyle L_2}$ norm over members of each group followed by an${\displaystyle L_1}$ norm over groups.

This can be solved by the proximal method, where the proximal operator is a block-wise soft-thresholding function:

The algorithm described for group sparsity without overlaps can be applied to the case where groups do overlap, in certain situations. This will likely result in some groups with all zero elements, and other groups with some non-zero and some zero elements.

If it is desired to preserve the group structure, a new regularizer can be defined:$${\displaystyle R(w)=inf\left\{\sum _{g=1}^G\Vert w_g{\Vert }_2:w=\sum _{g=1}^G{\overline{w}}_g\right\}}$$

For each${\displaystyle w_g}$,${\displaystyle {\overline{w}}_g}$ is defined as the vector such that the restriction of${\displaystyle {\overline{w}}_g}$ to the group${\displaystyle g}$ equals${\displaystyle w_g}$ and all other entries of${\displaystyle {\overline{w}}_g}$ are zero. The regularizer finds the optimal disintegration of${\displaystyle w}$ into parts. It can be viewed as duplicating all elements that exist in multiple groups. Learning problems with this regularizer can also be solved with the proximal method with a complication. The proximal operator cannot be computed in closed form, but can be effectively solved iteratively, inducing an inner iteration within the proximal method iteration.

When labels are more expensive to gather than input examples, semi-supervised learning can be useful. Regularizers have been designed to guide learning algorithms to learn models that respect the structure of unsupervised training samples. If a symmetric weight matrix${\displaystyle W}$ is given, a regularizer can be defined:$${\displaystyle R(f)=\sum _{i,j}{w}_{ij}{\left(f(x_i)-f(x_j)\right)}^2}$$

If${\displaystyle W_{ij}}$ encodes the result of some distance metric for points${\displaystyle x_i}$ and${\displaystyle x_j}$, it is desirable that${\displaystyle f(x_i)\approx f(x_j)}$. This regularizer captures this intuition, and is equivalent to:$${\displaystyle R(f)={\overline{f}}^{\mathsf{T}}L\overline{f}}$$ where${\displaystyle L=D-W}$ is theLaplacian matrix of the graph induced by${\displaystyle W}$.

The optimization problem${\displaystyle \underset{f\in {\mathbb{R}}^m}{min}R(f),m=u+l}$ can be solved analytically if the constraint${\displaystyle f(x_i)=y_i}$ is applied for all supervised samples. The labeled part of the vector${\displaystyle f}$ is therefore obvious. The unlabeled part of${\displaystyle f}$ is solved for by:$${\displaystyle \underset{f_u\in {\mathbb{R}}^u}{min}{f}^{\mathsf{T}}Lf=\underset{f_u\in {\mathbb{R}}^u}{min}\left\{f_u^{\mathsf{T}}{L}_{uu}{f}_u+f_l^{\mathsf{T}}{L}_{lu}{f}_u+f_u^{\mathsf{T}}{L}_{ul}{f}_l\right\}}$$$${\displaystyle {\mathrm{\nabla }}_{f_u}=2L_{uu}{f}_u+2L_{ul}Y}$$$${\displaystyle f_u=L_{uu}^{†}\left(L_{ul}Y\right)}$$The pseudo-inverse can be taken because${\displaystyle L_{ul}}$ has the same range as${\displaystyle L_{uu}}$.

In the case of multitask learning,${\displaystyle T}$ problems are considered simultaneously, each related in some way. The goal is to learn${\displaystyle T}$ functions, ideally borrowing strength from the relatedness of tasks, that have predictive power. This is equivalent to learning the matrix${\displaystyle W:T\times D}$ .

$${\displaystyle R(w)=\sum _{i=1}^D{\Vert W\Vert }_{2,1}}$$

This regularizer defines an L2 norm on each column and an L1 norm over all columns. It can be solved by proximal methods.

$${\displaystyle R(w)={\Vert \sigma (W)\Vert }_1}$$ where${\displaystyle \sigma (W)}$ is theeigenvalues in thesingular value decomposition of${\displaystyle W}$.

$${\displaystyle R(f_1\cdots f_T)=\sum _{t=1}^T{\Vert f_t-\frac{1}{T}\sum _{s=1}^T{f}_s\Vert }_{H_k}^2}$$

This regularizer constrains the functions learned for each task to be similar to the overall average of the functions across all tasks. This is useful for expressing prior information that each task is expected to share with each other task. An example is predicting blood iron levels measured at different times of the day, where each task represents an individual.

$${\displaystyle R(f_1\cdots f_T)=\sum _{r=1}^C\sum _{t\in I(r)}{\Vert f_t-\frac{1}{I(r)}\sum _{s\in I(r)}{f}_s\Vert }_{H_k}^2}$$ where${\displaystyle I(r)}$ is a cluster of tasks.

This regularizer is similar to the mean-constrained regularizer, but instead enforces similarity between tasks within the same cluster. This can capture more complex prior information. This technique has been used to predictNetflix recommendations. A cluster would correspond to a group of people who share similar preferences.

More generally than above, similarity between tasks can be defined by a function. The regularizer encourages the model to learn similar functions for similar tasks.$${\displaystyle R(f_1\cdots f_T)=\sum _{t,s=1,t\ne s}^{\mathsf{T}}{\Vert f_t-f_s\Vert }^2{M}_{ts}}$$ for a given symmetricsimilarity matrix${\displaystyle M}$.

Bayesian learning methods make use of aprior probability that (usually) gives lower probability to more complex models. Well-known model selection techniques include theAkaike information criterion (AIC),minimum description length (MDL), and theBayesian information criterion (BIC). Alternative methods of controlling overfitting not involving regularization includecross-validation.

Examples of applications of different methods of regularization to thelinear model are:

Model	Fit measure	Entropy measure[5][8]
AIC/BIC	${\displaystyle {\Vert Y-X\beta \Vert }_2}$	${\displaystyle {\Vert \beta \Vert }_0}$
Lasso[9]	${\displaystyle {\Vert Y-X\beta \Vert }_2}$	${\displaystyle {\Vert \beta \Vert }_1}$
Ridge regression[10]	${\displaystyle {\Vert Y-X\beta \Vert }_2}$	${\displaystyle {\Vert \beta \Vert }_2}$
Basis pursuit denoising	${\displaystyle {\Vert Y-X\beta \Vert }_2}$	${\displaystyle \lambda {\Vert \beta \Vert }_1}$
Rudin–Osher–Fatemi model (TV)	${\displaystyle {\Vert Y-X\beta \Vert }_2}$	${\displaystyle \lambda {\Vert \mathrm{\nabla }\beta \Vert }_1}$
Potts model	${\displaystyle {\Vert Y-X\beta \Vert }_2}$	${\displaystyle \lambda {\Vert \mathrm{\nabla }\beta \Vert }_0}$
RLAD[11]	${\displaystyle {\Vert Y-X\beta \Vert }_1}$	${\displaystyle {\Vert \beta \Vert }_1}$
Dantzig Selector[12]	${\displaystyle {\Vert X^{\mathsf{T}}(Y-X\beta )\Vert }_{\mathrm{\infty }}}$	${\displaystyle {\Vert \beta \Vert }_1}$
SLOPE[13]	${\displaystyle {\Vert Y-X\beta \Vert }_2}$	${\displaystyle \sum _{i=1}^p{\lambda }_i{\left|\beta \right|}_{(i)}}$

• Bayesian interpretation of regularization
• Bias–variance tradeoff
• Matrix regularization
• Regularization by spectral filtering
• Regularized least squares
• Lagrange multiplier
• Variance reduction
• L-curve

• Neumaier, A. (1998)."Solving ill-conditioned and singular linear systems: A tutorial on regularization"(PDF).SIAM Review.40 (3):636–666.Bibcode:1998SIAMR..40..636N.doi:10.1137/S0036144597321909. Archived fromthe original(PDF) on 2007-06-30.
• Kukačka, Jan; Golkov, Vladimir; Cremers, Daniel (2017). "Regularization for Deep Learning: A Taxonomy".arXiv:1710.10686 [cs.LG].

• Mathematical analysis
• Inverse problems

• Articles with short description
• Short description is different from Wikidata
• Wikipedia articles needing page number citations from May 2017
• Wikipedia articles needing clarification from October 2018