Inprobability theory, theexpected value (also calledexpectation,expectancy,expectation operator,mathematical expectation,mean,expectation value, orfirstmoment) is a generalization of theweighted average.
The expected value of a random variable with a finite number of outcomes is a weighted average of all possible outcomes. In the case of a continuum of possible outcomes, the expectation is defined byintegration. In the axiomatic foundation for probability provided bymeasure theory, the expectation is given byLebesgue integration.
The expected value of a random variableX is often denoted by,, or, withE also often stylized as orE.[1][2][3]
The idea of the expected value originated in the middle of the 17th century from the study of the so-calledproblem of points, which seeks to divide the stakesin a fair way between two players, who have to end their game before it is properly finished.[4] This problem had been debated for centuries. Many conflicting proposals and solutions had been suggested over the years when it was posed toBlaise Pascal by French writer and amateur mathematicianChevalier de Méré in 1654. Méré claimed that this problem could not be solved and that it showed just how flawed mathematics was when it came to its application to the real world. Pascal, being a mathematician, decided to work on a solution to the problem.
He began to discuss the problem in the famous series of letters toPierre de Fermat. Soon enough, they both independently came up with a solution. They solved the problem in different computational ways, but their results were identical because their computations were based on the same fundamental principle. The principle is that the value of a future gain should be directly proportional to the chance of getting it. This principle seemed to have come naturally to both of them. They were very pleased by the fact that they had found essentially the same solution, and this in turn made them absolutely convinced that they had solved the problem conclusively; however, they did not publish their findings. They only informed a small circle of mutual scientific friends in Paris about it.[5]
In Dutch mathematicianChristiaan Huygens' book, he considered the problem of points, and presented a solution based on the same principle as the solutions of Pascal and Fermat. Huygens published his treatise in 1657, (seeHuygens (1657)) "De ratiociniis in ludo aleæ" on probability theory just after visiting Paris. The book extended the concept of expectation by adding rules for how to calculate expectations in more complicated situations than the original problem (e.g., for three or more players), and can be seen as the first successful attempt at laying down the foundations of thetheory of probability.
In the foreword to his treatise, Huygens wrote:
It should be said, also, that for some time some of the best mathematicians of France have occupied themselves with this kind of calculus so that no one should attribute to me the honour of the first invention. This does not belong to me. But these savants, although they put each other to the test by proposing to each other many questions difficult to solve, have hidden their methods. I have had therefore to examine and go deeply for myself into this matter by beginning with the elements, and it is impossible for me for this reason to affirm that I have even started from the same principle. But finally I have found that my answers in many cases do not differ from theirs.
— Edwards (2002)
In the mid-nineteenth century,Pafnuty Chebyshev became the first person to think systematically in terms of the expectations ofrandom variables.[6]
Neither Pascal nor Huygens used the term "expectation" in its modern sense. In particular, Huygens writes:[7]
That any one Chance or Expectation to win any thing is worth just such a Sum, as wou'd procure in the same Chance and Expectation at a fair Lay. ... If I expect a or b, and have an equal chance of gaining them, my Expectation is worth (a+b)/2.
More than a hundred years later, in 1814,Pierre-Simon Laplace published his tract "Théorie analytique des probabilités", where the concept of expected value was defined explicitly:[8]
... this advantage in the theory of chance is the product of the sum hoped for by the probability of obtaining it; it is the partial sum which ought to result when we do not wish to run the risks of the event in supposing that the division is made proportional to the probabilities. This division is the only equitable one when all strange circumstances are eliminated; because an equal degree of probability gives an equal right for the sum hoped for. We will call this advantagemathematical hope.
The use of the letterE to denote "expected value" goes back toW. A. Whitworth in 1901.[9] The symbol has since become popular for English writers. In German,E stands forErwartungswert, in Spanish foresperanza matemática, and in French forespérance mathématique.[10]
When "E" is used to denote "expected value", authors use a variety of stylizations: the expectation operator can be stylized asE (upright),E (italic), or (inblackboard bold), while a variety of bracket notations (such asE(X),E[X], andEX) are all used.
Another popular notation isμX.⟨X⟩,⟨X⟩av, and are commonly used in physics.[11]M(X) is used in Russian-language literature.
As discussed above, there are several context-dependent ways of defining the expected value. The simplest and original definition deals with the case of finitely many possible outcomes, such as in the flip of a coin. With the theory of infinite series, this can be extended to the case of countably many possible outcomes. It is also very common to consider the distinct case of random variables dictated by (piecewise-)continuousprobability density functions, as these arise in many natural contexts. All of these specific definitions may be viewed as special cases of the general definition based upon the mathematical tools ofmeasure theory andLebesgue integration, which provide these different contexts with an axiomatic foundation and common language.
Any definition of expected value may be extended to define an expected value of a multidimensional random variable, i.e. arandom vectorX. It is defined component by component, asE[X]i = E[Xi]. Similarly, one may define the expected value of arandom matrixX with componentsXij byE[X]ij = E[Xij].
Consider a random variableX with afinite listx1, ...,xk of possible outcomes, each of which (respectively) has probabilityp1, ...,pk of occurring. The expectation ofX is defined as[12]
Since the probabilities must satisfyp1 + ⋅⋅⋅ +pk = 1, it is natural to interpretE[X] as aweighted average of thexi values, with weights given by their probabilitiespi.
In the special case that all possible outcomes areequiprobable (that is,p1 = ⋅⋅⋅ =pk), the weighted average is given by the standardaverage. In the general case, the expected value takes into account the fact that some outcomes are more likely than others.
An illustration of the convergence of sequence averages of rolls of a dice to the expected value of 3.5 as the number of rolls (trials) grows
Let represent the outcome of a roll of a fair six-sided die. More specifically, will be the number ofpips showing on the top face of the die after the toss. The possible values for are 1, 2, 3, 4, 5, and 6, all of which are equally likely with a probability of1/6. The expectation of is If one rolls the die times and computes the average (arithmetic mean) of the results, then as grows, the average willalmost surelyconverge to the expected value, a fact known as thestrong law of large numbers.
Theroulette game consists of a small ball and a wheel with 38 numbered pockets around the edge. As the wheel is spun, the ball bounces around randomly until it settles down in one of the pockets. Suppose random variable represents the (monetary) outcome of a $1 bet on a single number ("straight up" bet). If the bet wins (which happens with probability1/38 in American roulette), the payoff is $35; otherwise the player loses the bet. The expected profit from such a bet will be That is, the expected value to be won from a $1 bet is −$1/19. Thus, in 190 bets, the net loss will probably be about $10.
Random variables with countably infinitely many outcomes
Informally, the expectation of a random variable with acountably infinite set of possible outcomes is defined analogously as the weighted average of all possible outcomes, where the weights are given by the probabilities of realizing each given value. This is to say thatwherex1,x2, ... are the possible outcomes of the random variableX andp1,p2, ... are their corresponding probabilities. In many non-mathematical textbooks, this is presented as the full definition of expected values in this context.[13]
However, there are some subtleties with infinite summation, so the above formula is not suitable as a mathematical definition. In particular, theRiemann series theorem ofmathematical analysis illustrates that the value of certain infinite sums involving positive and negative summands depends on the order in which the summands are given. Since the outcomes of a random variable have no naturally given order, this creates a difficulty in defining expected value precisely.
For this reason, many mathematical textbooks only consider the case that the infinite sum given aboveconverges absolutely, which implies that the infinite sum is a finite number independent of the ordering of summands.[14] In the alternative case that the infinite sum does not converge absolutely, one says the random variabledoes not have finite expectation.[14]
Now consider a random variableX which has aprobability density function given by a functionf on thereal number line. This means that the probability ofX taking on a value in any givenopen interval is given by theintegral off over that interval. The expectation ofX is then given by the integral[15]A general and mathematically precise formulation of this definition usesmeasure theory andLebesgue integration, and the corresponding theory ofabsolutely continuous random variables is described in the next section. The density functions of many common distributions arepiecewise continuous, and as such the theory is often developed in this restricted setting.[16] For such functions, it is sufficient to only consider the standardRiemann integration. Sometimescontinuous random variables are defined as those corresponding to this special class of densities, although the term is used differently by various authors.
Analogously to the countably-infinite case above, there are subtleties with this expression due to the infinite region of integration. Such subtleties can be seen concretely if the distribution ofX is given by theCauchy distributionCauchy(0, π), so thatf(x) = (x2 + π2)−1. It is straightforward to compute in this case thatThe limit of this expression asa → −∞ andb → ∞ does not exist: if the limits are taken so thata = −b, then the limit is zero, while if the constraint2a = −b is taken, then the limit isln(2).
To avoid such ambiguities, in mathematical textbooks it is common to require that the given integralconverges absolutely, withE[X] left undefined otherwise.[17] However, measure-theoretic notions as given below can be used to give a systematic definition ofE[X] for more general random variablesX.
All definitions of the expected value may be expressed in the language ofmeasure theory. In general, ifX is a real-valuedrandom variable defined on aprobability space(Ω, Σ, P), then the expected value ofX, denoted byE[X], is defined as theLebesgue integral[18]Despite the newly abstract situation, this definition is extremely similar in nature to the very simplest definition of expected values, given above, as certain weighted averages. This is because, in measure theory, the value of the Lebesgue integral ofX is defined via weighted averages ofapproximations ofX which take on finitely many values.[19] Moreover, if given a random variable with finitely or countably many possible values, the Lebesgue theory of expectation is identical to the summation formulas given above. However, the Lebesgue theory clarifies the scope of the theory of probability density functions. A random variableX is said to beabsolutely continuous if any of the following conditions are satisfied:
there is a nonnegativemeasurable functionf on the real line such that for anyBorel setA, in which the integral is Lebesgue.
for any Borel setA of real numbers withLebesgue measure equal to zero, the probability ofX being valued inA is also equal to zero
for any positive numberε there is a positive numberδ such that: ifA is a Borel set with Lebesgue measure less thanδ, then the probability ofX being valued inA is less thanε.
These conditions are all equivalent, although this is nontrivial to establish.[20] In this definition,f is called theprobability density function ofX (relative to Lebesgue measure). According to the change-of-variables formula for Lebesgue integration,[21] combined with thelaw of the unconscious statistician,[22] it follows thatfor any absolutely continuous random variableX. The above discussion of continuous random variables is thus a special case of the general Lebesgue theory, due to the fact that every piecewise-continuous function is measurable.
Expected valueμ and median𝑚
The expected value of any real-valued random variable can also be defined on the graph of itscumulative distribution function by a nearby equality of areas. In fact, with a real number if and only if the two surfaces in the--plane, described byrespectively, have the same finite area, i.e. ifand bothimproper Riemann integrals converge. Finally, this is equivalent to the representationalso with convergent integrals.[23]
Expected values as defined above are automatically finite numbers. However, in many cases it is fundamental to be able to consider expected values of±∞. This is intuitive, for example, in the case of theSt. Petersburg paradox, in which one considers a random variable with possible outcomesxi = 2i, with associated probabilitiespi = 2−i, fori ranging over all positive integers. According to the summation formula in the case of random variables with countably many outcomes, one hasIt is natural to say that the expected value equals+∞.
There is a rigorous mathematical theory underlying such ideas, which is often taken as part of the definition of the Lebesgue integral.[19] The first fundamental observation is that, whichever of the above definitions are followed, anynonnegative random variable whatsoever can be given an unambiguous expected value; whenever absolute convergence fails, then the expected value can be defined as+∞. The second fundamental observation is that any random variable can be written as the difference of two nonnegative random variables. Given a random variableX, one defines thepositive and negative parts byX + = max(X, 0) andX − = −min(X, 0). These are nonnegative random variables, and it can be directly checked thatX =X + −X −. SinceE[X +] andE[X −] are both then defined as either nonnegative numbers or+∞, it is then natural to define:
According to this definition,E[X] exists and is finite if and only ifE[X +] andE[X −] are both finite. Due to the formula|X| =X + +X −, this is the case if and only ifE|X| is finite, and this is equivalent to the absolute convergence conditions in the definitions above. As such, the present considerations do not define finite expected values in any cases not previously considered; they are only useful for infinite expectations.
In the case of the St. Petersburg paradox, one hasX − = 0 and soE[X] = +∞ as desired.
Suppose the random variableX takes values1, −2,3, −4, ... with respective probabilities6π−2, 6(2π)−2, 6(3π)−2, 6(4π)−2, .... Then it follows thatX + takes value2k−1 with probability6((2k−1)π)−2 for each positive integerk, and takes value0 with remaining probability. Similarly,X − takes value2k with probability6(2kπ)−2 for each positive integerk and takes value0 with remaining probability. Using the definition for non-negative random variables, one can show that bothE[X +] = ∞ andE[X −] = ∞ (seeHarmonic series). Hence, in this case the expectation ofX is undefined.
Similarly, the Cauchy distribution, as discussed above, has undefined expectation.
In the case of a non-negative integer-valued random variable X, the expected value can also be expressed in terms of itstail probabilities (sometimes called thetail-sum formula):
A more general version holds for any non-negative random variable (discrete or continuous):
where the integrand is the survival function of X.
The following table gives the expected values of some commonly occurringprobability distributions. The third column gives the expected values both in the form immediately given by the definition, as well as in the simplified form obtained by computation therefrom. The details of these computations, which are not always straightforward, can be found in the indicated references.
The basic properties below (and their names in bold) replicate or follow immediately from those ofLebesgue integral. Note that the letters "a.s." stand for "almost surely"—a central property of the Lebesgue integral. Basically, one says that an inequality like is true almost surely, when the probability measure attributes zero-mass to the complementary event
Non-negativity: If (a.s.), then
Linearity of expectation:[34] The expected value operator (orexpectation operator) islinear in the sense that, for any random variables and and a constant whenever the right-hand side is well-defined. Byinduction, this means that the expected value of the sum of any finite number of random variables is the sum of the expected values of the individual random variables, and the expected value scales linearly with a multiplicative constant. Symbolically, for random variables and constants we have If we think of the set of random variables with finite expected value as forming avector space, then the linearity of expectation implies that the expected value is alinear form on this vector space.
Monotonicity: If(a.s.), and both and exist, then Proof follows from the linearity and the non-negativity property for since (a.s.).
Non-degeneracy: If then (a.s.).
If(a.s.), then In other words, if X and Y are random variables that take different values with probability zero, then the expectation of X will equal the expectation of Y.
If(a.s.) for some real numberc, then In particular, for a random variable with well-defined expectation, A well defined expectation implies that there is one number, or rather, one constant that defines the expected value. Thus follows that the expectation of this constant is just the original expected value.
As a consequence of the formula|X| =X+ +X− as discussed above, together with thetriangle inequality, it follows that for any random variable with well-defined expectation, one has
Let1A denote theindicator function of aneventA, thenE[1A] is given by the probability ofA. This is nothing but a different way of stating the expectation of aBernoulli random variable, as calculated in the table above.
Formulas in terms of CDF: If is thecumulative distribution function of a random variableX, then where the values on both sides are well defined or not well defined simultaneously, and the integral is taken in the sense ofLebesgue-Stieltjes. As a consequence ofintegration by parts as applied to this representation ofE[X], it can be proved that with the integrals taken in the sense of Lebesgue.[35] As a special case, for any random variableX valued in the nonnegative integers{0, 1, 2, 3, ...}, one has whereP denotes the underlying probability measure.
Non-multiplicativity: In general, the expected value is not multiplicative, i.e. is not necessarily equal to If and areindependent, then one can show that If the random variables aredependent, then generally although in special cases of dependency the equality may hold.
Law of the unconscious statistician: The expected value of a measurable function of given that has a probability density function is given by theinner product of and:[34] This formula also holds in multidimensional case, when is a function of several random variables, and is theirjoint density.[34][36]
Concentration inequalities control the likelihood of a random variable taking on large values.Markov's inequality is among the best-known and simplest to prove: for anonnegative random variableX and any positive numbera, it states that[37]
IfX is any random variable with finite expectation, then Markov's inequality may be applied to the random variable|X−E[X]|2 to obtainChebyshev's inequalitywhereVar is thevariance.[37] These inequalities are significant for their nearly complete lack of conditional assumptions. For example, for any random variable with finite expectation, the Chebyshev inequality implies that there is at least a 75% probability of an outcome being within twostandard deviations of the expected value. However, in special cases the Markov and Chebyshev inequalities often give much weaker information than is otherwise available. For example, in the case of an unweighted dice, Chebyshev's inequality says that odds of rolling between 1 and 6 is at least 53%; in reality, the odds are of course 100%.[38] TheKolmogorov inequality extends the Chebyshev inequality to the context of sums of random variables.[39]
The following three inequalities are of fundamental importance in the field ofmathematical analysis and its applications to probability theory.
Jensen's inequality: Letf:R →R be aconvex function andX a random variable with finite expectation. Then[40] Part of the assertion is that thenegative part off(X) has finite expectation, so that the right-hand side is well-defined (possibly infinite). Convexity off can be phrased as saying that the output of the weighted average oftwo inputs under-estimates the same weighted average of the two outputs; Jensen's inequality extends this to the setting of completely general weighted averages, as represented by the expectation. In the special case thatf(x) = |x|t/s for positive numberss <t, one obtains the Lyapunov inequality[41] This can also be proved by the Hölder inequality.[40] In measure theory, this is particularly notable for proving the inclusionLs ⊂ Lt ofLp spaces, in the special case ofprobability spaces.
Hölder's inequality: ifp > 1 andq > 1 are numbers satisfyingp −1 +q −1 = 1, then for any random variablesX andY.[40] The special case ofp =q = 2 is called theCauchy–Schwarz inequality, and is particularly well-known.[40]
Minkowski inequality: given any numberp ≥ 1, for any random variablesX andY withE|X|p andE|Y|p both finite, it follows thatE|X +Y|p is also finite and[42]
The Hölder and Minkowski inequalities can be extended to generalmeasure spaces, and are often given in that context. By contrast, the Jensen inequality is special to the case of probability spaces.
Expectations under convergence of random variables
In general, it is not the case that even if pointwise. Thus, one cannot interchange limits and expectation, without additional conditions on the random variables. To see this, let be a random variable distributed uniformly on For define a sequence of random variableswith being the indicator function of the event Then, it follows that pointwise. But, for each Hence,
Analogously, for general sequence of random variables the expected value operator is not-additive, i.e.
An example is easily obtained by setting and for where is as in the previous example.
A number of convergence results specify exact conditions which allow one to interchange limits and expectations, as specified below.
Monotone convergence theorem: Let be a sequence of random variables, with (a.s) for each Furthermore, let pointwise. Then, the monotone convergence theorem states that Using the monotone convergence theorem, one can show that expectation indeed satisfies countable additivity for non-negative random variables. In particular, let be non-negative random variables. It follows from themonotone convergence theorem that
Fatou's lemma: Let be a sequence of non-negative random variables. Fatou's lemma states thatCorollary. Let with for all If (a.s), thenProof is by observing that (a.s.) and applying Fatou's lemma.
Dominated convergence theorem: Let be a sequence of random variables. Ifpointwise (a.s.), (a.s.), and Then, according to the dominated convergence theorem,
;
Uniform integrability: In some cases, the equality holds when the sequence isuniformly integrable.
The expectation of a random variable plays an important role in a variety of contexts.
Instatistics, where one seeksestimates for unknownparameters based on available data gained fromsamples, thesample mean serves as an estimate for the expectation, and is itself a random variable. In such settings, the sample mean is considered to meet the desirable criterion for a "good" estimator in beingunbiased; that is, the expected value of the estimate is equal to thetrue value of the underlying parameter.
For a different example, indecision theory, an agent making an optimal choice in the context of incomplete information is often assumed to maximize the expected value of theirutility function.
It is possible to construct an expected value equal to the probability of an event by taking the expectation of anindicator function that is one if the event has occurred and zero otherwise. This relationship can be used to translate properties of expected values into properties of probabilities, e.g. using thelaw of large numbers to justify estimating probabilities byfrequencies.
The expected values of the powers ofX are called themoments ofX; themoments about the mean ofX are expected values of powers ofX − E[X]. The moments of some random variables can be used to specify their distributions, via theirmoment generating functions.
To empirically estimate the expected value of a random variable, one repeatedly measures observations of the variable and computes thearithmetic mean of the results. If the expected value exists, this procedure estimates the true expected value in an unbiased manner and has the property of minimizing the sum of the squares of theresiduals (the sum of the squared differences between the observations and the estimate). The law of large numbers demonstrates (under fairly mild conditions) that, as thesize of the sample gets larger, thevariance of this estimate gets smaller.
This property is often exploited in a wide variety of applications, including general problems ofstatistical estimation andmachine learning, to estimate (probabilistic) quantities of interest viaMonte Carlo methods, since most quantities of interest can be written in terms of expectation, e.g. where is the indicator function of the set
The mass of probability distribution is balanced at the expected value, here a Beta(α,β) distribution with expected value α/(α+β).
Inclassical mechanics, thecenter of mass is an analogous concept to expectation. For example, supposeX is a discrete random variable with valuesxi and corresponding probabilitiespi. Now consider a weightless rod on which are placed weights, at locationsxi along the rod and having massespi (whose sum is one). The point at which the rod balances is E[X].
Expected values can also be used to compute the variance, by means of the computational formula for the variance
^Mackey, George (July 1980). "HARMONIC ANALYSIS AS THE EXPLOITATION OF SYMMETRY - A HISTORICAL SURVEY".Bulletin of the American Mathematical Society. New Series.3 (1): 549.
^Whitworth, W.A. (1901)Choice and Chance with One Thousand Exercises. Fifth edition. Deighton Bell, Cambridge. [Reprinted by Hafner Publishing Co., New York, 1959.]
Billingsley, Patrick (1995).Probability and measure. Wiley Series in Probability and Mathematical Statistics (Third edition of 1979 original ed.). New York: John Wiley & Sons, Inc.ISBN0-471-00710-2.MR1324786.
Feller, William (1968).An introduction to probability theory and its applications. Volume I (Third edition of 1950 original ed.). New York–London–Sydney: John Wiley & Sons, Inc.MR0228020.
Feller, William (1971).An introduction to probability theory and its applications. Volume II (Second edition of 1966 original ed.). New York–London–Sydney: John Wiley & Sons, Inc.MR0270403.
Johnson, Norman L.;Kotz, Samuel; Balakrishnan, N. (1994).Continuous univariate distributions. Volume 1. Wiley Series in Probability and Mathematical Statistics (Second edition of 1970 original ed.). New York: John Wiley & Sons, Inc.ISBN0-471-58495-9.MR1299979.
