Inmathematical optimization anddecision theory, aloss function orcost function (sometimes also called an error function)[1] is a function that maps anevent or values of one or more variables onto areal number intuitively representing some "cost" associated with the event. Anoptimization problem seeks to minimize a loss function. Anobjective function is either a loss function or its opposite (in specific domains, variously called areward function, aprofit function, autility function, afitness function, etc.), in which case it is to be maximized. The loss function could include terms from several levels of the hierarchy.
In statistics, typically a loss function is used forparameter estimation, and the event in question is some function of the difference between estimated and true values for an instance of data. The concept, as old asLaplace, was reintroduced in statistics byAbraham Wald in the middle of the 20th century.[2] In the context ofeconomics, for example, this is usuallyeconomic cost orregret. Inclassification, it is the penalty for an incorrect classification of an example. Inactuarial science, it is used in an insurance context to model benefits paid over premiums, particularly since the works ofHarald Cramér in the 1920s.[3] Inoptimal control, the loss is the penalty for failing to achieve a desired value. Infinancial risk management, the function is mapped to a monetary loss.
Leonard J. Savage argued that using non-Bayesian methods such asminimax, the loss function should be based on the idea ofregret, i.e., the loss associated with a decision should be the difference between the consequences of the best decision that could have been made under circumstances will be known and the decision that was in fact taken before they were known.
The use of aquadratic loss function is common, for example when usingleast squares techniques. It is often more mathematically tractable than other loss functions because of the properties ofvariances, as well as being symmetric: an error above the target causes the same loss as the same magnitude of error below the target. If the target ist, then a quadratic loss function is
for some constantC; the value of the constant makes no difference to a decision, and can be ignored by setting it equal to 1. This is also known as thesquared error loss (SEL).[1]
Many commonstatistics, includingt-tests,regression models,design of experiments, and much else, useleast squares methods applied usinglinear regression theory, which is based on the quadratic loss function.
The quadratic loss function is also used inlinear-quadratic optimal control problems. In these problems, even in the absence of uncertainty, it may not be possible to achieve the desired values of all target variables. Often loss is expressed as aquadratic form in the deviations of the variables of interest from their desired values; this approach istractable because it results in linearfirst-order conditions. In the context ofstochastic control, the expected value of the quadratic form is used. The quadratic loss assigns more importance to outliers than to the true data due to its square nature, so alternatives like theHuber, Log-Cosh and SMAE losses are used when the data has many large outliers.
Instatistics anddecision theory, a frequently used loss function is the0-1 loss function
usingIverson bracket notation, i.e. it evaluates to 1 when, and 0 otherwise.
In many applications, objective functions, including loss functions as a particular case, are determined by the problem formulation. In other situations, the decision maker’s preference must be elicited and represented by a scalar-valued function (called alsoutility function) in a form suitable for optimization — the problem thatRagnar Frisch has highlighted in hisNobel Prize lecture.[4]The existing methods for constructing objective functions are collected in the proceedings of two dedicated conferences.[5][6]In particular,Andranik Tangian showed that the most usable objective functions — quadratic and additive — are determined by a fewindifference points. He used this property in the models for constructing these objective functions from eitherordinal orcardinal data that were elicited through computer-assisted interviews with decision makers.[7][8] Among other things, he constructed objective functions to optimally distribute budgets for 16 Westfalian universities[9]and the European subsidies for equalizing unemployment rates among 271 German regions.[10]
In some contexts, the value of the loss function itself is a random quantity because it depends on the outcome of a random variableX.
Bothfrequentist andBayesian statistical theory involve making a decision based on theexpected value of the loss function; however, this quantity is defined differently under the two paradigms.
We first define the expected loss in the frequentist context. It is obtained by taking the expected value with respect to theprobability distribution,Pθ, of the observed data,X. This is also referred to as therisk function[11][12][13][14] of the decision ruleδ and the parameterθ. Here the decision rule depends on the outcome ofX. The risk function is given by:
Here,θ is a fixed but possibly unknown state of nature,X is a vector of observations stochastically drawn from apopulation, is the expectation over all population values ofX,dPθ is aprobability measure over the event space ofX (parametrized by θ) and the integral is evaluated over the entiresupport of X.
In a Bayesian approach, the expectation is calculated using theprior distributionπ* of the parameter θ:
where m(x) is known as thepredictive likelihood wherein θ has been "integrated out,"π* (θ | x) is the posterior distribution, and the order of integration has been changed. One then should choose the actiona* which minimises this expected loss, which is referred to asBayes Risk. In the latter equation, the integrand inside dx is known as thePosterior Risk, and minimising it with respect to decisiona also minimizes the overall Bayes Risk. This optimal decision,a* is known as theBayes (decision) Rule - it minimises the average loss over all possible states of nature θ, over all possible (probability-weighted) data outcomes. One advantage of the Bayesian approach is to that one need only choose the optimal action under the actual observed data to obtain a uniformly optimal one, whereas choosing the actual frequentist optimal decision rule as a function of all possible observations, is a much more difficult problem. Of equal importance though, the Bayes Rule reflects consideration of loss outcomes under different states of nature, θ.
In economics, decision-making under uncertainty is often modelled using thevon Neumann–Morgenstern utility function of the uncertain variable of interest, such as end-of-period wealth. Since the value of this variable is uncertain, so is the value of the utility function; it is the expected value of utility that is maximized.
Adecision rule makes a choice using an optimality criterion. Some commonly used criteria are:
Sound statistical practice requires selecting an estimator consistent with the actual acceptable variation experienced in the context of a particular applied problem. Thus, in the applied use of loss functions, selecting which statistical method to use to model an applied problem depends on knowing the losses that will be experienced from being wrong under the problem's particular circumstances.[15]
A common example involves estimating "location". Under typical statistical assumptions, themean or average is the statistic for estimating location that minimizes the expected loss experienced under thesquared-error loss function, while themedian is the estimator that minimizes expected loss experienced under the absolute-difference loss function. Still different estimators would be optimal under other, less common circumstances.
In economics, when an agent isrisk neutral, the objective function is simply expressed as the expected value of a monetary quantity, such as profit, income, or end-of-period wealth. Forrisk-averse orrisk-loving agents, loss is measured as the negative of autility function, and the objective function to be optimized is the expected value of utility.
Other measures of cost are possible, for examplemortality ormorbidity in the field ofpublic health orsafety engineering.
For mostoptimization algorithms, it is desirable to have a loss function that is globallycontinuous anddifferentiable.
Two very commonly used loss functions are thesquared loss,, and theabsolute loss,. However the absolute loss has the disadvantage that it is not differentiable at. The squared loss has the disadvantage that it has the tendency to be dominated byoutliers—when summing over a set of's (as in), the final sum tends to be the result of a few particularly largea-values, rather than an expression of the averagea-value.
The choice of a loss function is not arbitrary. It is very restrictive and sometimes the loss function may be characterized by its desirable properties.[16] Among the choice principles are, for example, the requirement of completeness of the class of symmetric statistics in the case ofi.i.d. observations, the principle of complete information, and some others.
W. Edwards Deming andNassim Nicholas Taleb argue that empirical reality, not nice mathematical properties, should be the sole basis for selecting loss functions, and real losses often are not mathematically nice and are not differentiable, continuous, symmetric, etc. For example, a person who arrives before a plane gate closure can still make the plane, but a person who arrives after can not, a discontinuity and asymmetry which makes arriving slightly late much more costly than arriving slightly early. In drug dosing, the cost of too little drug may be lack of efficacy, while the cost of too much may be tolerable toxicity, another example of asymmetry. Traffic, pipes, beams, ecologies, climates, etc. may tolerate increased load or stress with little noticeable change up to a point, then become backed up or break catastrophically. These situations, Deming and Taleb argue, are common in real-life problems, perhaps more common than classical smooth, continuous, symmetric, differentials cases.[17]
Differentiable computing | |
|---|---|
| General | |
| Hardware | |
| Software libraries | |
