Movatterモバイル変換

[0]ホーム
URL:

Jump to content
WikipediaThe Free Encyclopedia
Search

Law of large numbers

From Wikipedia, the free encyclopedia
Not to be confused withLaw of truly large numbers.
Averages of repeated trials converge to the expected value
Anillustration of the law of large numbers using a particular run of rolls of a singledie. As the number of rolls in this run increases, the average of the values of all the results approaches 3.5. Although each run would show a distinctive shape over a small number of throws (at the left), over a large number of rolls (to the right) the shapes would be extremely similar.
Part of a series onstatistics
Probability theory

Inprobability theory, thelaw of large numbers is amathematical law that states that theaverage of the results obtained from a large number of independent random samples converges to the true value, if it exists.[1] More formally, the law of large numbers states that given a sample of independent and identically distributed values, the sample mean converges to the truemean.

The law of large numbers is important because it guarantees stable long-term results for the averages of somerandomevents.[1][2] For example, while acasino may losemoney in a single spin of theroulette wheel, its earnings will tend towards a predictable percentage over a large number of spins. Any winning streak by a player will eventually be overcome by the parameters of the game. Importantly, the law applies (as the name indicates) only when alarge number of observations are considered. There is no principle that a small number of observations will coincide with the expected value or that a streak of one value will immediately be "balanced" by the others (see thegambler's fallacy).

Throughout its history, many mathematicians have refined this law. Today, the law of large numbers is used in many fields including statistics, probability theory, economics, and insurance.[3]

Examples

[edit]

For example, a single roll of a six-sideddice produces one of the numbers 1, 2, 3, 4, 5, or 6, each with equalprobability. Therefore, theexpected value of the roll is:

1+2+3+4+5+66=3.5{\displaystyle {\frac {1+2+3+4+5+6}{6}}=3.5}

According to the law of large numbers, if a large number of six-sided dice are rolled, the average of their values (sometimes called thesample mean) will approach 3.5, with the precision increasing as more dice are rolled.

It follows from the law of large numbers that theempirical probability of success in a series ofBernoulli trials will converge to the theoretical probability. For aBernoulli random variable, the expected value is the theoretical probability of success, and the average ofn such variables (assuming they areindependent and identically distributed (i.i.d.)) is precisely the relative frequency.

This image illustrates the convergence of relative frequencies to their theoretical probabilities. The probability of picking a red ball from a sack is 0.4 and black ball is 0.6. The left plot shows the relative frequency of picking a black ball, and the right plot shows the relative frequency of picking a red ball, both over 10,000 trials. As the number of trials increases, the relative frequencies approach their respective theoretical probabilities, demonstrating the law of large numbers.

For example, afair coin toss is a Bernoulli trial. When a fair coin is flipped once, the theoretical probability that the outcome will be heads is equal to1⁄2. Therefore, according to the law of large numbers, the proportion of heads in a "large" number of coin flips "should be" roughly1⁄2. In particular, the proportion of heads aftern flips willalmost surelyconverge to1⁄2 asn approaches infinity.

Although the proportion of heads (and tails) approaches1⁄2, almost surely theabsolute difference in the number of heads and tails will become large as the number of flips becomes large. That is, the probability that the absolute difference is a small number approaches zero as the number of flips becomes large. Also, almost surely the ratio of the absolute difference to the number of flips will approach zero. Intuitively, the expected difference grows, but at a slower rate than the number of flips.

Another good example of the law of large numbers is theMonte Carlo method. These methods are a broad class ofcomputationalalgorithms that rely on repeatedrandom sampling to obtain numerical results. The larger the number of repetitions, the better the approximation tends to be. The reason that this method is important is mainly that, sometimes, it is difficult or impossible to use other approaches.[4]

Limitation

[edit]

The average of the results obtained from a large number of trials may fail to converge in some cases. For instance, the average ofn results taken from theCauchy distribution or somePareto distributions (α<1) will not converge asn becomes larger; the reason isheavy tails.[5] The Cauchy distribution and the Pareto distribution represent two cases: the Cauchy distribution does not have an expectation,[6] whereas the expectation of the Pareto distribution (α<1) is infinite.[7] One way to generate the Cauchy-distributed example is where the random numbers equal thetangent of an angle uniformly distributed between −90° and +90°.[8] Themedian is zero, but the expected value does not exist, and indeed the average ofn such variables have the same distribution as one such variable. It does not converge in probability toward zero (or any other value) asn goes to infinity.

If the trials embed aselection bias, typical in human economic/rational behaviour, the law of large numbers does not help in solving the bias, even if the number of trials is increased the selection bias remains.

History

[edit]
Diffusion is an example of the law of large numbers. Initially, there aresolute molecules on the left side of a barrier (magenta line) and none on the right. The barrier is removed, and the solute diffuses to fill the whole container.
  • Top: With a single molecule, the motion appears to be quite random.
  • Middle: With more molecules, there is clearly a trend where the solute fills the container more and more uniformly, but there are also random fluctuations.
  • Bottom: With an enormous number of solute molecules (too many to see), the randomness is essentially gone: The solute appears to move smoothly and systematically from high-concentration areas to low-concentration areas. In realistic situations, chemists can describe diffusion as a deterministic macroscopic phenomenon (seeFick's laws), despite its underlying random nature.

The Italian mathematicianGerolamo Cardano (1501–1576) stated without proof that the accuracies of empirical statistics tend to improve with the number of trials.[9][3] This was then formalized as a law of large numbers. A special form of the law of large numbers (for a binary random variable) was first proved byJacob Bernoulli.[10][3] It took him over 20 years to develop a sufficiently rigorous mathematical proof which was published in hisArs Conjectandi (The Art of Conjecturing) in 1713. He named this his "golden theorem" but it became generally known as "Bernoulli's theorem". This should not be confused withBernoulli's principle, named after Jacob Bernoulli's nephewDaniel Bernoulli. In 1837,S. D. Poisson further described it under the name"la loi des grands nombres" ("the law of large numbers").[11][12][3] Thereafter, it was known under both names, but the "law of large numbers" is most frequently used.

After Bernoulli and Poisson published their efforts, other mathematicians also contributed to refinement of the law, includingChebyshev,[13]Markov,Borel,Cantelli,Kolmogorov andKhinchin.[3] Markov showed that the law can apply to a random variable that does not have a finite variance under some other weaker assumption, and Khinchin showed in 1929 that if the series consists of independent identically distributed random variables, it suffices that theexpected value exists for the weak law of large numbers to be true.[14][15] These further studies have given rise to two prominent forms of the law of large numbers. One is called the "weak" law and the other the "strong" law, in reference to two different modes ofconvergence of the cumulative sample means to the expected value; in particular, as explained below, the strong form implies the weak.[14]

Forms

[edit]

There are two different versions of the law of large numbers that are described below. They are called thestrong law of large numbers and theweak law of large numbers.[16][1] Stated for the case whereX1,X2, ... is an infinite sequence ofindependent and identically distributed (i.i.d.)Lebesgue integrable random variables with expected value E(X1) = E(X2) = ... =μ, both versions of the law state that the sample average

X¯n=1n(X1+⋯+Xn){\displaystyle {\overline {X}}_{n}={\frac {1}{n}}(X_{1}+\cdots +X_{n})}

converges to the expected value:

X¯n→μas n→∞.{\displaystyle {\overline {X}}_{n}\to \mu \quad {\textrm {as}}\ n\to \infty .}1

(Lebesgue integrability ofXj means that the expected value E(Xj) exists according to Lebesgue integration and is finite. It doesnot mean that the associated probability measure isabsolutely continuous with respect toLebesgue measure.)

Introductory probability texts often additionally assume identical finitevarianceVar⁡(Xi)=σ2{\displaystyle \operatorname {Var} (X_{i})=\sigma ^{2}} (for alli{\displaystyle i}) and no correlation between random variables. In that case, the variance of the average ofn random variables is

Var⁡(X¯n)=Var⁡(1n(X1+⋯+Xn))=1n2Var⁡(X1+⋯+Xn)=nσ2n2=σ2n.{\displaystyle \operatorname {Var} ({\overline {X}}_{n})=\operatorname {Var} ({\tfrac {1}{n}}(X_{1}+\cdots +X_{n}))={\frac {1}{n^{2}}}\operatorname {Var} (X_{1}+\cdots +X_{n})={\frac {n\sigma ^{2}}{n^{2}}}={\frac {\sigma ^{2}}{n}}.}

which can be used to shorten and simplify the proofs. This assumption of finitevariance isnot necessary. Large or infinite variance will make the convergence slower, but the law of large numbers holds anyway.[17]

Mutual independence of the random variables can be replaced bypairwise independence[18] orexchangeability[19] in both versions of the law.

The difference between the strong and the weak version is concerned with the mode of convergence being asserted. For interpretation of these modes, seeConvergence of random variables.

Weak law

[edit]
Simulation illustrating the law of large numbers. Each frame, a coin that is red on one side and blue on the other is flipped, and a dot is added in the corresponding column. A pie chart shows the proportion of red and blue so far. Notice that while the proportion varies significantly at first, it approaches 50% as the number of trials increases.

Theweak law of large numbers (also calledKhinchin's law) states that given a collection ofindependent and identically distributed (iid) samples from a random variable with finite mean, the sample meanconverges in probability to the expected value[20]

X¯n →P μwhen n→∞.{\displaystyle {\overline {X}}_{n}\ {\overset {P}{\rightarrow }}\ \mu \qquad {\textrm {when}}\ n\to \infty .}2

That is, for any positive numberε,

limn→∞Pr(|X¯n−μ|<ε)=1.{\displaystyle \lim _{n\to \infty }\Pr \!\left(\,|{\overline {X}}_{n}-\mu |<\varepsilon \,\right)=1.}

Interpreting this result, the weak law states that for any nonzero margin specified (ε), no matter how small, with a sufficiently large sample there will be a very high probability that the average of the observations will be close to the expected value; that is, within the margin.

As mentioned earlier, the weak law applies in the case of i.i.d. random variables, but it also applies in some other cases. For example, the variance may be different for each random variable in the series, keeping the expected value constant. If the variances are bounded, then the law applies, as shown byChebyshev as early as 1867. (If the expected values change during the series, then we can simply apply the law to the average deviation from the respective expected values. The law then states that this converges in probability to zero.) In fact, Chebyshev's proof works so long as the variance of the average of the firstn values goes to zero asn goes to infinity.[15] As an example, assume that each random variable in the series follows aGaussian distribution (normal distribution) with mean zero, but with variance equal to2n/log⁡(n+1){\displaystyle 2n/\log(n+1)}, which is not bounded. At each stage, the average will be normally distributed (as the average of a set of normally distributed variables). The variance of the sum is equal to the sum of the variances, which isasymptotic ton2/log⁡n{\displaystyle n^{2}/\log n}. The variance of the average is therefore asymptotic to1/log⁡n{\displaystyle 1/\log n} and goes to zero.

There are also examples of the weak law applying even though the expected value does not exist.

Strong law

[edit]

Thestrong law of large numbers (also calledKolmogorov's law) states that the sample averageconverges almost surely to the expected value[21]

X¯n ⟶a.s. μwhen n→∞.{\displaystyle {\overline {X}}_{n}\ {\overset {\text{a.s.}}{\longrightarrow }}\ \mu \qquad {\textrm {when}}\ n\to \infty .}3

That is,

Pr(limn→∞X¯n=μ)=1.{\displaystyle \Pr \!\left(\lim _{n\to \infty }{\overline {X}}_{n}=\mu \right)=1.}

What this means is that, as the number of trialsn goes to infinity, the probability that the average of the observations converges to the expected value, is equal to one. The modern proof of the strong law is more complex than that of the weak law, and relies on passing to an appropriate sub-sequence.[17]

The strong law of large numbers can itself be seen as a special case of thepointwise ergodic theorem. This view justifies the intuitive interpretation of the expected value (for Lebesgue integration only) of a random variable when sampled repeatedly as the "long-term average".

Law 3 is called the strong law because random variables which converge strongly (almost surely) are guaranteed to converge weakly (in probability). However the weak law is known to hold in certain conditions where the strong law does not hold and then the convergence is only weak (in probability). SeeDifferences between the weak law and the strong law.

The strong law applies to independent identically distributed random variables having an expected value (like the weak law). This was proved by Kolmogorov in 1930. It can also apply in other cases. Kolmogorov also showed, in 1933, that if the variables are independent and identically distributed, then for the average to converge almost surely onsomething (this can be considered another statement of the strong law), it is necessary that they have an expected value (and then of course the average will converge almost surely on that).[22]

If the summands are independent but not identically distributed, then

X¯n−E⁡[X¯n] ⟶a.s. 0,{\displaystyle {\overline {X}}_{n}-\operatorname {E} {\big [}{\overline {X}}_{n}{\big ]}\ {\overset {\text{a.s.}}{\longrightarrow }}\ 0,}2

provided that eachXk has a finite second moment and

∑k=1∞1k2Var⁡[Xk]<∞.{\displaystyle \sum _{k=1}^{\infty }{\frac {1}{k^{2}}}\operatorname {Var} [X_{k}]<\infty .}

This statement is known asKolmogorov's strong law, see e.g.Sen & Singer (1993, Theorem 2.3.10).

Differences between the weak law and the strong law

[edit]

Theweak law states that for a specified largen, the averageX¯n{\displaystyle {\overline {X}}_{n}} is likely to be nearμ.[23] Thus, it leaves open the possibility that|X¯n−μ|>ε{\displaystyle |{\overline {X}}_{n}-\mu |>\varepsilon } happens an infinite number of times, although at infrequent intervals. (Not necessarily|X¯n−μ|≠0{\displaystyle |{\overline {X}}_{n}-\mu |\neq 0} for alln).

Thestrong law shows that thisalmost surely will not occur. I.e., with probability 1 for anyε > 0 the inequality|X¯n−μ|<ε{\displaystyle |{\overline {X}}_{n}-\mu |<\varepsilon } holds for all large enoughn.[24]

The strong law does not hold in the following cases, but the weak law does.[25][26]

  1. Let X be anexponentially distributed random variable with parameter 1. The random variablesin⁡(X)eXX−1{\displaystyle \sin(X)e^{X}X^{-1}} has no expected value according to Lebesgue integration, but using conditional convergence and interpreting the integral as aDirichlet integral, which is an improperRiemann integral, we can say:E(sin⁡(X)eXX)= ∫x=0∞sin⁡(x)exxe−xdx=π2{\displaystyle E\left({\frac {\sin(X)e^{X}}{X}}\right)=\ \int _{x=0}^{\infty }{\frac {\sin(x)e^{x}}{x}}e^{-x}dx={\frac {\pi }{2}}}
  2. Let X be ageometrically distributed random variable with probability 0.5. The random variable2X(−1)XX−1{\displaystyle 2^{X}(-1)^{X}X^{-1}} does not have an expected value in the conventional sense because the infiniteseries is not absolutely convergent, but using conditional convergence, we can say:E(2X(−1)XX)= ∑x=1∞2x(−1)xx2−x=−ln⁡(2){\displaystyle E\left({\frac {2^{X}(-1)^{X}}{X}}\right)=\ \sum _{x=1}^{\infty }{\frac {2^{x}(-1)^{x}}{x}}2^{-x}=-\ln(2)}
  3. If thecumulative distribution function of a random variable is{1−F(x)=e2xln⁡(x),x≥eF(x)=e−2xln⁡(−x),x≤−e{\displaystyle {\begin{cases}1-F(x)&={\frac {e}{2x\ln(x)}},&x\geq e\\F(x)&={\frac {e}{-2x\ln(-x)}},&x\leq -e\end{cases}}}then it has no expected value, but the weak law is true.[27][28]
  4. LetXk be plus or minusk/log⁡log⁡log⁡k{\textstyle {\sqrt {k/\log \log \log k}}} (starting at sufficiently largek so that the denominator is positive) with probability1⁄2 for each.[22] The variance ofXk is thenk/log⁡log⁡log⁡k.{\textstyle k/\log \log \log k.} Kolmogorov's strong law does not apply because the partial sum in his criterion up tok = n is asymptotic tolog⁡n/log⁡log⁡log⁡n{\displaystyle \log n/\log \log \log n} and this is unbounded. If we replace the random variables with Gaussian variables having the same variances, namelyk/log⁡log⁡log⁡k{\textstyle {\sqrt {k/\log \log \log k}}}, then the average at any point will also be normally distributed. The width of the distribution of the average will tend toward zero (standard deviation asymptotic to1/2log⁡log⁡log⁡n{\textstyle 1/{\sqrt {2\log \log \log n}}}), but for a givenε, there is probability which does not go to zero withn, while the average sometime after thenth trial will come back up toε. Since the width of the distribution of the average is not zero, it must have a positive lower boundp(ε), which means there is a probability of at leastp(ε) that the average will attain ε aftern trials. It will happen with probabilityp(ε)/2 before somem which depends onn. But even afterm, there is still a probability of at leastp(ε) that it will happen. (This seems to indicate thatp(ε)=1 and the average will attain ε an infinite number of times.)

Uniform laws of large numbers

[edit]

There are extensions of the law of large numbers to collections of estimators, where the convergence is uniform over the collection; thus the nameuniform law of large numbers.

Supposef(x,θ) is somefunction defined forθ ∈ Θ, and continuous inθ. Then for any fixedθ, the sequence {f(X1,θ),f(X2,θ), ...} will be a sequence of independent and identically distributed random variables, such that the sample mean of this sequence converges in probability to E[f(X,θ)]. This is thepointwise (inθ) convergence.

A particular example of auniform law of large numbers states the conditions under which the convergence happensuniformly inθ. If[29][30]

  1. Θ is compact,
  2. f(x,θ) is continuous at eachθ ∈ Θ foralmost allxs, andmeasurable function ofx at eachθ.
  3. there exists adominating functiond(x) such that E[d(X)] < ∞, and‖f(x,θ)‖≤d(x)for all θ∈Θ.{\displaystyle \left\|f(x,\theta )\right\|\leq d(x)\quad {\text{for all}}\ \theta \in \Theta .}

Then E[f(X,θ)] is continuous inθ, and

supθ∈Θ‖1n∑i=1nf(Xi,θ)−E⁡[f(X,θ)]‖→P 0.{\displaystyle \sup _{\theta \in \Theta }\left\|{\frac {1}{n}}\sum _{i=1}^{n}f(X_{i},\theta )-\operatorname {E} [f(X,\theta )]\right\|{\overset {\mathrm {P} }{\rightarrow }}\ 0.}

This result is useful to derive consistency of a large class of estimators (seeExtremum estimator).

Borel's law of large numbers

[edit]

Borel's law of large numbers, named afterÉmile Borel, states that if an experiment is repeated a large number of times, independently under identical conditions, then the proportion of times that any specified event is expected to occur approximately equals the probability of the event's occurrence on any particular trial; the larger the number of repetitions, the better the approximation tends to be. More precisely, ifE denotes the event in question,p its probability of occurrence, andNn(E) the number of timesE occurs in the firstn trials, then with probability one,[31]Nn(E)n→p as n→∞.{\displaystyle {\frac {N_{n}(E)}{n}}\to p{\text{ as }}n\to \infty .}

This theorem makes rigorous the intuitive notion of probability as the expected long-run relative frequency of an event's occurrence. It is a special case of any of several more general laws of large numbers in probability theory.


Proof of the weak law

[edit]

GivenX1,X2, ... an infinite sequence ofi.i.d. random variables with finite expected valueE(X1)=E(X2)=⋯=μ<∞{\displaystyle E(X_{1})=E(X_{2})=\cdots =\mu <\infty }, we are interested in the convergence of the sample average

X¯n=1n(X1+⋯+Xn).{\displaystyle {\overline {X}}_{n}={\tfrac {1}{n}}(X_{1}+\cdots +X_{n}).}

The weak law of large numbers states:

X¯n →P μwhen n→∞.{\displaystyle {\overline {X}}_{n}\ {\overset {P}{\rightarrow }}\ \mu \qquad {\textrm {when}}\ n\to \infty .}2

Proof using Chebyshev's inequality assuming finite variance

[edit]

This proof uses the assumption of finitevarianceVar⁡(Xi)=σ2{\displaystyle \operatorname {Var} (X_{i})=\sigma ^{2}} (for alli{\displaystyle i}). The independence of the random variables implies no correlation between them, and we have that

Var⁡(X¯n)=Var⁡(1n(X1+⋯+Xn))=1n2Var⁡(X1+⋯+Xn)=nσ2n2=σ2n.{\displaystyle \operatorname {Var} ({\overline {X}}_{n})=\operatorname {Var} ({\tfrac {1}{n}}(X_{1}+\cdots +X_{n}))={\frac {1}{n^{2}}}\operatorname {Var} (X_{1}+\cdots +X_{n})={\frac {n\sigma ^{2}}{n^{2}}}={\frac {\sigma ^{2}}{n}}.}

The common mean μ of the sequence is the mean of the sample average:

E(X¯n)=μ.{\displaystyle E({\overline {X}}_{n})=\mu .}

UsingChebyshev's inequality onX¯n{\displaystyle {\overline {X}}_{n}} results in

P⁡(|X¯n−μ|≥ε)≤σ2nε2.{\displaystyle \operatorname {P} (\left|{\overline {X}}_{n}-\mu \right|\geq \varepsilon )\leq {\frac {\sigma ^{2}}{n\varepsilon ^{2}}}.}

This may be used to obtain the following:

P⁡(|X¯n−μ|<ε)=1−P⁡(|X¯n−μ|≥ε)≥1−σ2nε2.{\displaystyle \operatorname {P} (\left|{\overline {X}}_{n}-\mu \right|<\varepsilon )=1-\operatorname {P} (\left|{\overline {X}}_{n}-\mu \right|\geq \varepsilon )\geq 1-{\frac {\sigma ^{2}}{n\varepsilon ^{2}}}.}

Asn approaches infinity, the expression approaches 1. And by definition ofconvergence in probability, we have obtained

X¯n →P μwhen n→∞.{\displaystyle {\overline {X}}_{n}\ {\overset {P}{\rightarrow }}\ \mu \qquad {\textrm {when}}\ n\to \infty .}2

Proof using convergence of characteristic functions

[edit]

ByTaylor's theorem forcomplex functions, thecharacteristic function of any random variable,X, with finite mean μ, can be written as

φX(t)=1+itμ+o(t),t→0.{\displaystyle \varphi _{X}(t)=1+it\mu +o(t),\quad t\rightarrow 0.}

AllX1,X2, ... have the same characteristic function, so we will simply denote thisφX.

Among the basic properties of characteristic functions there are

φ1nX(t)=φX(tn)andφX+Y(t)=φX(t)φY(t){\displaystyle \varphi _{{\frac {1}{n}}X}(t)=\varphi _{X}({\tfrac {t}{n}})\quad {\text{and}}\quad \varphi _{X+Y}(t)=\varphi _{X}(t)\varphi _{Y}(t)\quad } ifX andY are independent.

These rules can be used to calculate the characteristic function ofX¯n{\displaystyle {\overline {X}}_{n}} in terms ofφX:

φX¯n(t)=[φX(tn)]n=[1+iμtn+o(tn)]n→eitμ,asn→∞.{\displaystyle \varphi _{{\overline {X}}_{n}}(t)=\left[\varphi _{X}\left({t \over n}\right)\right]^{n}=\left[1+i\mu {t \over n}+o\left({t \over n}\right)\right]^{n}\,\rightarrow \,e^{it\mu },\quad {\text{as}}\quad n\to \infty .}

The limiteitμ is the characteristic function of the constant random variable μ, and hence by theLévy continuity theorem,X¯n{\displaystyle {\overline {X}}_{n}}converges in distribution to μ:

X¯n→Dμforn→∞.{\displaystyle {\overline {X}}_{n}\,{\overset {\mathcal {D}}{\rightarrow }}\,\mu \qquad {\text{for}}\qquad n\to \infty .}

μ is a constant, which implies that convergence in distribution to μ and convergence in probability to μ are equivalent (seeConvergence of random variables.) Therefore,

X¯n →P μwhen n→∞.{\displaystyle {\overline {X}}_{n}\ {\overset {P}{\rightarrow }}\ \mu \qquad {\textrm {when}}\ n\to \infty .}2

This shows that the sample mean converges in probability to the derivative of the characteristic function at the origin, as long as the latter exists.

Proof of the strong law

[edit]

We give a relatively simple proof of the strong law under the assumptions that theXi{\displaystyle X_{i}} areiid,E[Xi]=:μ<∞{\displaystyle {\mathbb {E} }[X_{i}]=:\mu <\infty },Var⁡(Xi)=σ2<∞{\displaystyle \operatorname {Var} (X_{i})=\sigma ^{2}<\infty }, andE[Xi4]=:τ<∞{\displaystyle {\mathbb {E} }[X_{i}^{4}]=:\tau <\infty }.

Let us first note thatwithout loss of generality we can assume thatμ=0{\displaystyle \mu =0} by centering. In this case, the strong law says that

Pr(limn→∞X¯n=0)=1,{\displaystyle \Pr \!\left(\lim _{n\to \infty }{\overline {X}}_{n}=0\right)=1,}orPr(ω:limn→∞Sn(ω)n=0)=1.{\displaystyle \Pr \left(\omega :\lim _{n\to \infty }{\frac {S_{n}(\omega )}{n}}=0\right)=1.}It is equivalent to show thatPr(ω:limn→∞Sn(ω)n≠0)=0,{\displaystyle \Pr \left(\omega :\lim _{n\to \infty }{\frac {S_{n}(\omega )}{n}}\neq 0\right)=0,}Note thatlimn→∞Sn(ω)n≠0⟺∃ϵ>0,|Sn(ω)n|≥ϵ infinitely often,{\displaystyle \lim _{n\to \infty }{\frac {S_{n}(\omega )}{n}}\neq 0\iff \exists \epsilon >0,\left|{\frac {S_{n}(\omega )}{n}}\right|\geq \epsilon \ {\mbox{infinitely often}},}and thus to prove the strong law we need to show that for everyϵ>0{\displaystyle \epsilon >0}, we havePr(ω:|Sn(ω)|≥nϵ infinitely often)=0.{\displaystyle \Pr \left(\omega :|S_{n}(\omega )|\geq n\epsilon {\mbox{ infinitely often}}\right)=0.}Define the eventsAn={ω:|Sn|≥nϵ}{\displaystyle A_{n}=\{\omega :|S_{n}|\geq n\epsilon \}}, and if we can show that∑n=1∞Pr(An)<∞,{\displaystyle \sum _{n=1}^{\infty }\Pr(A_{n})<\infty ,}then the Borel-Cantelli Lemma implies the result. So let us estimatePr(An){\displaystyle \Pr(A_{n})}.

We computeE[Sn4]=E[(∑i=1nXi)4]=E[∑1≤i,j,k,l≤nXiXjXkXl].{\displaystyle {\mathbb {E} }[S_{n}^{4}]={\mathbb {E} }\left[\left(\sum _{i=1}^{n}X_{i}\right)^{4}\right]={\mathbb {E} }\left[\sum _{1\leq i,j,k,l\leq n}X_{i}X_{j}X_{k}X_{l}\right].}We first claim that every term of the formXi3Xj,Xi2XjXk,XiXjXkXl{\displaystyle X_{i}^{3}X_{j},X_{i}^{2}X_{j}X_{k},X_{i}X_{j}X_{k}X_{l}} where all subscripts are distinct, must have zero expectation. This is becauseE[Xi3Xj]=E[Xi3]E[Xj]{\displaystyle {\mathbb {E} }[X_{i}^{3}X_{j}]={\mathbb {E} }[X_{i}^{3}]{\mathbb {E} }[X_{j}]} by independence, and the last term is zero—and similarly for the other terms. Therefore the only terms in the sum with nonzero expectation areE[Xi4]{\displaystyle {\mathbb {E} }[X_{i}^{4}]} andE[Xi2Xj2]{\displaystyle {\mathbb {E} }[X_{i}^{2}X_{j}^{2}]}. Since theXi{\displaystyle X_{i}} are identically distributed, all of these are the same, and moreoverE[Xi2Xj2]=(E[Xi2])2{\displaystyle {\mathbb {E} }[X_{i}^{2}X_{j}^{2}]=({\mathbb {E} }[X_{i}^{2}])^{2}}.

There aren{\displaystyle n} terms of the formE[Xi4]{\displaystyle {\mathbb {E} }[X_{i}^{4}]} and3n(n−1){\displaystyle 3n(n-1)} terms of the form(E[Xi2])2{\displaystyle ({\mathbb {E} }[X_{i}^{2}])^{2}}, and soE[Sn4]=nτ+3n(n−1)σ4.{\displaystyle {\mathbb {E} }[S_{n}^{4}]=n\tau +3n(n-1)\sigma ^{4}.}Note that the right-hand side is a quadratic polynomial inn{\displaystyle n}, and as such there exists aC>0{\displaystyle C>0} such thatE[Sn4]≤Cn2{\displaystyle {\mathbb {E} }[S_{n}^{4}]\leq Cn^{2}} forn{\displaystyle n} sufficiently large. By Markov,Pr(|Sn|≥nϵ)≤1(nϵ)4E[Sn4]≤Cϵ4n2,{\displaystyle \Pr(|S_{n}|\geq n\epsilon )\leq {\frac {1}{(n\epsilon )^{4}}}{\mathbb {E} }[S_{n}^{4}]\leq {\frac {C}{\epsilon ^{4}n^{2}}},}forn{\displaystyle n} sufficiently large, and therefore this series is summable. Since this holds for anyϵ>0{\displaystyle \epsilon >0}, we have established the strong law of large numbers.[32] The proof can be strengthened immensely by dropping all finiteness assumptions on the second and fourth moments. It can also be extended for example to discuss partial sums of distributions without any finite moments. Such proofs use more intricate arguments to prove the same Borel-Cantelli predicate, a strategy attributed to Kolmogorov to conceptually bring the limit inside the probability parentheses.[33]

Consequences

[edit]

The law of large numbers provides an expectation of an unknown distribution from a realization of the sequence, but also any feature of theprobability distribution.[1] By applyingBorel's law of large numbers, one could easily obtain theprobability mass function. For each event in the objective probability mass function, one could approximate the probability of the event's occurrence with the proportion of times that any specified event occurs. The larger the number of repetitions, the better the approximation. As for the continuous case:C=(a−h,a+h]{\displaystyle C=(a-h,a+h]}, for small positive h. Thus, for large n:

Nn(C)n≈p=P(X∈C)=∫a−ha+hf(x)dx≈2hf(a){\displaystyle {\frac {N_{n}(C)}{n}}\thickapprox p=P(X\in C)=\int _{a-h}^{a+h}f(x)\,dx\thickapprox 2hf(a)}

With this method, one can cover the whole x-axis with a grid (with grid size 2h) and obtain a bar graph which is called ahistogram.

Applications

[edit]

One application of the law of large numbers is an important method of approximation known as theMonte Carlo method,[3] which uses a random sampling of numbers to approximate numerical results. The algorithm to compute an integral of f(x) on an interval [a, b] is as follows:[3]

  1. Simulate uniform random variables X1, X2, …, Xn which can be done using a software, and use a random number table that gives U1, U2, …, Un independent and identically distributed (i.i.d.) random variables on [0, 1]. Then let Xi = a + (b - a) Ui for i= 1, 2, …, n. Then X1, X2, …, Xn are independent and identically distributed uniform random variables on [a, b].
  2. Evaluate f(X1), f(X2), …, f(Xn).
  3. Take the average of f(X1), f(X2), …, f(Xn) by computing(b−a)f(X1)+f(X2)+⋯+f(Xn)n{\displaystyle (b-a){\tfrac {f(X_{1})+f(X_{2})+\dots +f(X_{n})}{n}}}, and then by the strong law of large numbers this converges to(b−a)E⁡(f(X1))=(b−a)∫abf(x)1b−adx=∫abf(x)dx{\displaystyle (b-a)\operatorname {E} (f(X_{1}))=(b-a)\int _{a}^{b}f(x){\tfrac {1}{b-a}}\,dx=\int _{a}^{b}f(x){dx}}.

We can find the integral off(x)=cos2⁡(x)x3+1{\displaystyle f(x)=\cos ^{2}(x){\sqrt {x^{3}+1}}} on [-1, 2]. Using traditional methods to compute this integral is very difficult, so the Monte Carlo method can be used here.[3] Using the above algorithm, we get

∫−12f(x)dx=0.905{\displaystyle \int _{-1}^{2}f(x)\,dx=0.905} when n = 25

and

∫−12f(x)dx=1.028{\displaystyle \int _{-1}^{2}f(x)\,dx=1.028} when n = 250.

We observe that as n increases, the numerical value also increases. When we get the actual results for the integral we get

∫−12f(x)dx=1.000194{\displaystyle \int _{-1}^{2}f(x)\,dx=1.000194}.

When the LLN was used, the approximation of the integral was closer to its true value, and thus more accurate.[3]

Another example is the integration off(x)=ex−1e−1{\displaystyle f(x)={\frac {e^{x}-1}{e-1}}} over [0, 1].[34] Using the Monte Carlo method and the LLN, we can see that as the number of samples increases, the numerical value gets ever closer to 0.4180233.[34]

See also

[edit]

Notes

[edit]
  1. ^abcdDekking, Michel (2005).A Modern Introduction to Probability and Statistics. Springer. pp. 181–190.ISBN 9781852338961.
  2. ^Yao, Kai; Gao, Jinwu (2016). "Law of Large Numbers for Uncertain Random Variables".IEEE Transactions on Fuzzy Systems.24 (3):615–621.Bibcode:2016ITFS...24..615Y.doi:10.1109/TFUZZ.2015.2466080.ISSN 1063-6706.S2CID 2238905.
  3. ^abcdefghiSedor, Kelly."The Law of Large Numbers and its Applications"(PDF).
  4. ^Kroese, Dirk P.; Brereton, Tim; Taimre, Thomas; Botev, Zdravko I. (2014). "Why the Monte Carlo method is so important today".Wiley Interdisciplinary Reviews: Computational Statistics.6 (6):386–392.doi:10.1002/wics.1314.hdl:1959.4/unsworks_43203.S2CID 18521840.
  5. ^Dekking, Michel, ed. (2005).A modern introduction to probability and statistics: understanding why and how. Springer texts in statistics. London [Heidelberg]: Springer. p. 187.ISBN 978-1-85233-896-1.
  6. ^Dekking, Michel (2005).A Modern Introduction to Probability and Statistics. Springer. pp. 92.ISBN 9781852338961.{{cite book}}: CS1 maint: url-status (link)
  7. ^Dekking, Michel (2005).A Modern Introduction to Probability and Statistics. Springer. pp. 63.ISBN 9781852338961.{{cite book}}: CS1 maint: url-status (link)
  8. ^Pitman, E. J. G.; Williams, E. J. (1967)."Cauchy-Distributed Functions of Cauchy Variates".The Annals of Mathematical Statistics.38 (3):916–918.doi:10.1214/aoms/1177698885.ISSN 0003-4851.JSTOR 2239008.
  9. ^Mlodinow, L. (2008).The Drunkard's Walk. New York: Random House. p. 50.
  10. ^Bernoulli, Jakob (1713). "4".Ars Conjectandi: Usum & Applicationem Praecedentis Doctrinae in Civilibus, Moralibus & Oeconomicis (in Latin). Translated by Sheynin, Oscar.
  11. ^Poisson names the "law of large numbers" (la loi des grands nombres) in:Poisson, S. D. (1837).Probabilité des jugements en matière criminelle et en matière civile, précédées des règles générales du calcul des probabilitiés (in French). Paris, France: Bachelier. p. 7. He attempts a two-part proof of the law on pp. 139–143 and pp. 277 ff.
  12. ^Hacking, Ian (1983). "19th-century Cracks in the Concept of Determinism".Journal of the History of Ideas.44 (3):455–475.doi:10.2307/2709176.JSTOR 2709176.
  13. ^Tchebichef, P. (1846)."Démonstration élémentaire d'une proposition générale de la théorie des probabilités".Journal für die reine und angewandte Mathematik (in French).1846 (33):259–267.doi:10.1515/crll.1846.33.259.S2CID 120850863.
  14. ^abSeneta 2013.
  15. ^abYuri Prohorov."Law of large numbers".Encyclopedia of Mathematics. EMS Press.
  16. ^Bhattacharya, Rabi; Lin, Lizhen; Patrangenaru, Victor (2016).A Course in Mathematical Statistics and Large Sample Theory. Springer Texts in Statistics. New York, NY: Springer New York.doi:10.1007/978-1-4939-4032-5.ISBN 978-1-4939-4030-1.
  17. ^ab"The strong law of large numbers – What's new". Terrytao.wordpress.com. 19 June 2008. Retrieved2012-06-09.
  18. ^Etemadi, N. Z. (1981)."An elementary proof of the strong law of large numbers".Wahrscheinlichkeitstheorie Verw Gebiete.55 (1):119–122.doi:10.1007/BF01013465.S2CID 122166046.
  19. ^Kingman, J. F. C. (April 1978)."Uses of Exchangeability".The Annals of Probability.6 (2).doi:10.1214/aop/1176995566.ISSN 0091-1798.
  20. ^Loève 1977, Chapter 1.4, p. 14
  21. ^Loève 1977, Chapter 17.3, p. 251
  22. ^abYuri Prokhorov."Strong law of large numbers".Encyclopedia of Mathematics.
  23. ^"What Is the Law of Large Numbers? (Definition) | Built In".builtin.com. Retrieved2023-10-20.
  24. ^Ross (2009)
  25. ^Lehmann, Erich L.; Romano, Joseph P. (2006-03-30).Weak law converges to constant. Springer.ISBN 9780387276052.
  26. ^Dguvl Hun Hong; Sung Ho Lee (1998)."A Note on the Weak Law of Large Numbers for Exchangeable Random Variables"(PDF).Communications of the Korean Mathematical Society.13 (2):385–391. Archived fromthe original(PDF) on 2016-07-01. Retrieved2014-06-28.
  27. ^Mukherjee, Sayan."Law of large numbers"(PDF). Archived fromthe original(PDF) on 2013-03-09. Retrieved2014-06-28.
  28. ^J. Geyer, Charles."Law of large numbers"(PDF).
  29. ^Newey & McFadden 1994, Lemma 2.4
  30. ^Jennrich, Robert I. (1969)."Asymptotic Properties of Non-Linear Least Squares Estimators".The Annals of Mathematical Statistics.40 (2):633–643.doi:10.1214/aoms/1177697731.
  31. ^Wen, Liu (1991)."An Analytic Technique to Prove Borel's Strong Law of Large Numbers".The American Mathematical Monthly.98 (2):146–148.doi:10.2307/2323947.JSTOR 2323947.
  32. ^Another proof was given byEtemadi, Nasrollah (1981)."An elementary proof of the strong law of large numbers".Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete.55. Springer:119–122.doi:10.1007/BF01013465.S2CID 122166046.
  33. ^For a proof without the added assumption of a finite fourth moment, see Section 22 ofBillingsley, Patrick (1979).Probability and Measure.
  34. ^abReiter, Detlev (2008), Fehske, H.; Schneider, R.; Weiße, A. (eds.),"The Monte Carlo Method, an Introduction",Computational Many-Particle Physics, Lecture Notes in Physics, vol. 739, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 63–78,doi:10.1007/978-3-540-74686-7_3,ISBN 978-3-540-74685-0, retrieved2023-12-08{{citation}}: CS1 maint: work parameter with ISBN (link)

References

[edit]
  • Grimmett, G. R.; Stirzaker, D. R. (1992).Probability and Random Processes (2nd ed.). Oxford: Clarendon Press.ISBN 0-19-853665-8.
  • Durrett, Richard (1995).Probability: Theory and Examples (2nd ed.). Duxbury Press.
  • Martin Jacobsen (1992).Videregående Sandsynlighedsregning [Advanced Probability Theory] (in Danish) (3rd ed.). Copenhagen: HCØ-tryk.ISBN 87-91180-71-6.
  • Loève, Michel (1977).Probability theory 1 (4th ed.). Springer.
  • Newey, Whitney K.;McFadden, Daniel (1994). "36".Large sample estimation and hypothesis testing. Handbook of econometrics. Vol. IV. Elsevier Science. pp. 2111–2245.
  • Ross, Sheldon (2009).A first course in probability (8th ed.). Prentice Hall.ISBN 978-0-13-603313-4.
  • Sen, P. K; Singer, J. M. (1993).Large sample methods in statistics. Chapman & Hall.
  • Seneta, Eugene (2013). "A Tricentenary history of the Law of Large Numbers".Bernoulli.19 (4):1088–1121.arXiv:1309.6488.doi:10.3150/12-BEJSP12.S2CID 88520834.

External links

[edit]
Discrete time
Continuous time
Both
Fields and other
Time series models
Financial models
Actuarial models
Queueing models
Properties
Limit theorems
Inequalities
Tools
Disciplines
International
National
Other
Retrieved from "https://en.wikipedia.org/w/index.php?title=Law_of_large_numbers&oldid=1337375214"
Categories:
Hidden categories:
[8]ページ先頭
©2009-2026 Movatter.jp