Movatterモバイル変換

Bernoulli distribution

From Wikipedia, the free encyclopedia

Probability distribution modeling a coin toss which need not be fair

Bernoulli distribution
Probability mass function Three examples of Bernoulli distribution: $P(x=0)=0{.}2$ and $P(x=1)=0{.}8$ $P(x=0)=0{.}8$ and $P(x=1)=0{.}2$ $P(x=0)=0{.}5$ and $P(x=1)=0{.}5$
Parameters	$0\leq p\leq 1$ $q=1-p$
Support	$k\in \{0,1\}$
PMF	${\begin{cases}q=1-p&{\text{if }}k=0\\p&{\text{if }}k=1\end{cases}}$
CDF	${\begin{cases}0&{\text{if }}k<0\\1-p&{\text{if }}0\leq k<1\\1&{\text{if }}k\geq 1\end{cases}}$
Mean	$p {\displaystyle p}$
Median	${\begin{cases}0&{\text{if }}p<1/2\\\left[0,1\right]&{\text{if }}p=1/2\\1&{\text{if }}p>1/2\end{cases}}$
Mode	${\begin{cases}0&{\text{if }}p<1/2\\0,1&{\text{if }}p=1/2\\1&{\text{if }}p>1/2\end{cases}}$
Variance	$p(1-p)=pq$
MAD	$2p(1-p)=2pq$
Skewness	${\frac {q-p}{\sqrt {pq}}}$
Excess kurtosis	${\frac {1-6pq}{pq}}$
Entropy	$-q\ln q-p\ln p$
MGF	$q+pe^{t}$
CF	$q+pe^{it}$
PGF	$q+pz$
Fisher information	${\frac {1}{pq}}$

Probability theory
Part of a series onstatistics

Probability Axioms Determinism System Indeterminism Randomness
Probability space Sample space Event Collectively exhaustive events Elementary event Mutual exclusivity Outcome Singleton Experiment Bernoulli trial Probability distribution Bernoulli distribution Binomial distribution Exponential distribution Normal distribution Pareto distribution Poisson distribution Probability measure Random variable Bernoulli process Continuous or discrete Expected value Variance Markov chain Observed value Random walk Stochastic process
Complementary event Joint probability Marginal probability Conditional probability
Independence Conditional independence Law of total probability Law of large numbers Bayes' theorem Boole's inequality
Venn diagram Tree diagram
v t e

Inprobability theory andstatistics, theBernoulli distribution, named after Swiss mathematicianJacob Bernoulli,^[1] is thediscrete probability distribution of arandom variable which takes the value 1 with probability $p {\displaystyle p}$ and the value 0 with probability $q=1-p$ . Less formally, it can be thought of as a model for the set of possible outcomes of any singleexperiment that asks ayes–no question. Such questions lead tooutcomes that areBoolean-valued: a singlebit whose value is success/yes/true/one withprobabilityp and failure/no/false/zero with probabilityq. It can be used to represent a (possibly biased)coin toss where 1 and 0 would represent "heads" and "tails", respectively, andp would be the probability of the coin landing on heads (or vice versa where 1 would represent tails andp would be the probability of tails). In particular, unfair coins would have $p\neq 1/2.$

The Bernoulli distribution is a special case of thebinomial distribution where a single trial is conducted (son would be 1 for such a binomial distribution). It is also a special case of thetwo-point distribution, for which the possible outcomes need not be 0 and 1.^[2]

Properties

[edit]

If $X {\displaystyle X}$ is a random variable with a Bernoulli distribution, then:

${\begin{aligned}\Pr(X{=}1)&=p,\\\Pr(X{=}0)&=q=1-p.\end{aligned}}$

Theprobability mass function $f {\displaystyle f}$ of this distribution, over possible outcomesk, is^[3]

$f(k;p)={\begin{cases}p&{\text{if }}k=1,\\q=1-p&{\text{if }}k=0.\end{cases}}$

This can also be expressed as

$f(k;p)=p^{k}(1-p)^{1-k}\quad {\text{for }}k\in \{0,1\}$

or as

$f(k;p)=pk+(1-p)(1-k)\quad {\text{for }}k\in \{0,1\}.$

The Bernoulli distribution is a special case of thebinomial distribution with $n=1.$ ^[4]

Thekurtosis goes to infinity for high and low values of $p, {\displaystyle p,}$ but for $p=1/2$ the two-point distributions including the Bernoulli distribution have a lowerexcess kurtosis, namely −2, than any other probability distribution.

The Bernoulli distributions for $0\leq p\leq 1$ form anexponential family.

Themaximum likelihood estimator of $p {\displaystyle p}$ based on a random sample is thesample mean.

The probability mass distribution function of a Bernoulli experiment along with its corresponding cumulative distribution function

Mean

[edit]

Theexpected value of a Bernoulli random variable $X {\displaystyle X}$ is

$\operatorname {E} [X]=p$

This is because for a Bernoulli distributed random variable $X {\displaystyle X}$ with $\Pr(X{=}1)=p$ and ${\textstyle \Pr(X{=}0)=q}$ we find^[3]

${\begin{aligned}\operatorname {E} [X]&=\Pr(X{=}1)\cdot 1+\Pr(X{=}0)\cdot 0\\[1ex]&=p\cdot 1+q\cdot 0\\[1ex]&=p.\end{aligned}}$

Variance

[edit]

Thevariance of a Bernoulli distributed $X {\displaystyle X}$ is

$\operatorname {Var} [X]=pq=p(1-p)$

We first find

${\begin{aligned}\operatorname {E} [X^{2}]&=\Pr(X{=}1)\cdot 1^{2}+\Pr(X{=}0)\cdot 0^{2}\\&=p\cdot 1^{2}+q\cdot 0^{2}\\&=p=\operatorname {E} [X]\end{aligned}}$

From this follows^[3]

${\begin{aligned}\operatorname {Var} [X]&=\operatorname {E} [X^{2}]-\operatorname {E} [X]^{2}=\operatorname {E} [X]-\operatorname {E} [X]^{2}\\[1ex]&=p-p^{2}=p(1-p)=pq\end{aligned}}$

With this result it is easy to prove that, for any Bernoulli distribution, its variance will have a value inside $[0,1/4]$ .

Skewness

[edit]

Theskewness is ${\frac {q-p}{\sqrt {pq}}}={\frac {1-2p}{\sqrt {pq}}}$ . When we take the standardized Bernoulli distributed random variable ${\frac {X-\operatorname {E} [X]}{\sqrt {\operatorname {Var} [X]}}}$ we find that this random variable attains ${\frac {q}{\sqrt {pq}}}$ with probability $p {\displaystyle p}$ and attains $-{\frac {p}{\sqrt {pq}}}$ with probability $q {\displaystyle q}$ . Thus we get

${\begin{aligned}\gamma _{1}&=\operatorname {E} \left[\left({\frac {X-\operatorname {E} [X]}{\sqrt {\operatorname {Var} [X]}}}\right)^{3}\right]\\&=p\cdot \left({\frac {q}{\sqrt {pq}}}\right)^{3}+q\cdot \left(-{\frac {p}{\sqrt {pq}}}\right)^{3}\\&={\frac {1}{{\sqrt {pq}}^{3}}}\left(pq^{3}-qp^{3}\right)\\&={\frac {pq}{{\sqrt {pq}}^{3}}}(q^{2}-p^{2})\\&={\frac {(1-p)^{2}-p^{2}}{\sqrt {pq}}}\\&={\frac {1-2p}{\sqrt {pq}}}={\frac {q-p}{\sqrt {pq}}}.\end{aligned}}$

Higher moments and cumulants

[edit]

The raw moments are all equal because $1^{k}=1$ and $0^{k}=0$ .

$\operatorname {E} [X^{k}]=\Pr(X{=}1)\cdot 1^{k}+\Pr(X{=}0)\cdot 0^{k}=p\cdot 1+q\cdot 0=p=\operatorname {E} [X].$

The central moment of order $k {\displaystyle k}$ is given by $\mu _{k}=(1-p)(-p)^{k}+p(1-p)^{k}.$ The first six central moments are ${\begin{aligned}\mu _{1}&=0,\\\mu _{2}&=p(1-p),\\\mu _{3}&=p(1-p)(1-2p),\\\mu _{4}&=p(1-p)(1-3p(1-p)),\\\mu _{5}&=p(1-p)(1-2p)(1-2p(1-p)),\\\mu _{6}&=p(1-p)(1-5p(1-p)(1-p(1-p))).\end{aligned}}$ The higher central moments can be expressed more compactly in terms of $\mu _{2}$ and $\mu _{3}$ ${\begin{aligned}\mu _{4}&=\mu _{2}(1-3\mu _{2}),\\\mu _{5}&=\mu _{3}(1-2\mu _{2}),\\\mu _{6}&=\mu _{2}(1-5\mu _{2}(1-\mu _{2})).\end{aligned}}$ The first six cumulants are ${\begin{aligned}\kappa _{1}&=p,\\\kappa _{2}&=\mu _{2},\\\kappa _{3}&=\mu _{3},\\\kappa _{4}&=\mu _{2}(1-6\mu _{2}),\\\kappa _{5}&=\mu _{3}(1-12\mu _{2}),\\\kappa _{6}&=\mu _{2}(1-30\mu _{2}(1-4\mu _{2})).\end{aligned}}$

Entropy and Fisher's Information

[edit]

Entropy

[edit]

Entropy is a measure of uncertainty or randomness in a probability distribution. For a Bernoulli random variable $X {\displaystyle X}$ with success probability $p {\displaystyle p}$ and failure probability $q=1-p$ , the entropy $H(X)$ is defined as:

${\begin{aligned}H(X)&=\mathbb {E} _{p}\ln {\frac {1}{\Pr(X)}}\\[1ex]&=-\Pr(X{=}0)\ln \Pr(X{=}0)-\Pr(X{=}1)\ln \Pr(X{=}1)\\[1ex]&=-(q\ln q+p\ln p).\end{aligned}}$

The entropy is maximized when $p=0.5$ , indicating the highest level of uncertainty when both outcomes are equally likely. The entropy is zero when $p=0$ or $p=1$ , where one outcome is certain.

Fisher's Information

[edit]

Fisher information measures the amount of information that an observable random variable $X {\displaystyle X}$ carries about an unknown parameter $p {\displaystyle p}$ upon which the probability of $X {\displaystyle X}$ depends. For the Bernoulli distribution, the Fisher information with respect to the parameter $p {\displaystyle p}$ is given by:

$I(p)={\frac {1}{pq}}$

Proof:

TheLikelihood Function for a Bernoulli random variable $X {\displaystyle X}$ is: $L(p;X)=p^{X}(1-p)^{1-X}$ This represents the probability of observing $X {\displaystyle X}$ given the parameter $p {\displaystyle p}$ .
TheLog-Likelihood Function is: $\ln L(p;X)=X\ln p+(1-X)\ln(1-p)$
The Score Function (the first derivative of the log-likelihood with respect to $p {\displaystyle p}$ is: ${\frac {\partial }{\partial p}}\ln L(p;X)={\frac {X}{p}}-{\frac {1-X}{1-p}}$
The second derivative of the log-likelihood function is: ${\frac {\partial ^{2}}{\partial p^{2}}}\ln L(p;X)=-{\frac {X}{p^{2}}}-{\frac {1-X}{(1-p)^{2}}}$
Fisher information is calculated as the negative expected value of the second derivative of the log-likelihood: ${\begin{aligned}I(p)=-E\left[{\frac {\partial ^{2}}{\partial p^{2}}}\ln L(p;X)\right]=-\left(-{\frac {p}{p^{2}}}-{\frac {1-p}{(1-p)^{2}}}\right)={\frac {1}{p(1-p)}}={\frac {1}{pq}}\end{aligned}}$

It is maximized when $p=0.5$ , reflecting maximum uncertainty and thus maximum information about the parameter $p {\displaystyle p}$ .

Related distributions

[edit]

If $X_{1},\dots ,X_{n}$ are independent, identically distributed (i.i.d.) random variables, allBernoulli trials with success probability p, then theirsum is distributed according to abinomial distribution with parametersn andp:
$\sum _{k=1}^{n}X_{k}\sim \operatorname {B} (n,p)$ (binomial distribution).^[3]

The Bernoulli distribution is simply

\operatorname {B} (1,p)

, also written as

{\textstyle \mathrm {Bernoulli} (p).}

Thecategorical distribution is the generalization of the Bernoulli distribution for variables with any constant number of discrete values.
TheBeta distribution is theconjugate prior of the Bernoulli distribution.^[5]
Thegeometric distribution models the number of independent and identical Bernoulli trials needed to get one success.
If ${\textstyle Y\sim \mathrm {Bernoulli} \left({\frac {1}{2}}\right)}$ , then ${\textstyle 2Y-1}$ has aRademacher distribution.

References

[edit]

^Uspensky, James Victor (1937).Introduction to Mathematical Probability. New York:McGraw-Hill. p. 45.OCLC 996937.
^Dekking, Frederik; Kraaikamp, Cornelis; Lopuhaä, Hendrik; Meester, Ludolf (9 October 2010).A Modern Introduction to Probability and Statistics (1 ed.).Springer London. pp. 43–48.ISBN 9781849969529.
^^a ^b ^c ^dBertsekas, Dimitri P. (2002).Introduction to Probability.Tsitsiklis, John N., Τσιτσικλής, Γιάννης Ν. Belmont, Mass.: Athena Scientific.ISBN 188652940X.OCLC 51441829.
^McCullagh, Peter;Nelder, John (1989).Generalized Linear Models, Second Edition. Boca Raton: Chapman and Hall/CRC. Section 4.2.2.ISBN 0-412-31760-5.
^Orloff, Jeremy; Bloom, Jonathan."Conjugate priors: Beta and normal"(PDF).math.mit.edu. RetrievedOctober 20, 2023.

Author's mention

[edit]

Abhirath, dwivedi;Kotz, Samuel;Kemp, Adrienne W. (1993).Univariate Discrete Distributions (2nd ed.). Wiley.ISBN 0-471-54897-9.
Peatman, John G. (1963).Introduction to Applied Statistics. New York: Harper & Row. pp. 162–171.

External links

[edit]

Wikimedia Commons has media related toBernoulli distribution.

"Binomial distribution",Encyclopedia of Mathematics,EMS Press, 2001 [1994].
Weisstein, Eric W."Bernoulli Distribution".MathWorld.
Interactive graphic:Univariate Distribution Relationships.

Probability distributions (list)

Discrete
univariate

with finite support	Benford Bernoulli Beta-binomial Binomial Categorical Hypergeometric Negative Poisson binomial Rademacher Soliton Discrete uniform Zipf Zipf–Mandelbrot
with infinite support	Beta negative binomial Borel Conway–Maxwell–Poisson Discrete phase-type Delaporte Extended negative binomial Flory–Schulz Gauss–Kuzmin Geometric Logarithmic Mixed Poisson Negative binomial Panjer Parabolic fractal Poisson Skellam Yule–Simon Zeta

Continuous
univariate

supported on a bounded interval	Arcsine ARGUS Balding–Nichols Bates Beta Generalized Beta rectangular Continuous Bernoulli Irwin–Hall Kumaraswamy Logit-normal Noncentral beta PERT Power function Raised cosine Reciprocal Triangular U-quadratic Uniform Wigner semicircle
supported on a semi-infinite interval	Benini Benktander 1st kind Benktander 2nd kind Beta prime Burr Chi Chi-squared Noncentral Inverse Scaled Dagum Davis Erlang Hyper Exponential Hyperexponential Hypoexponential Logarithmic F Noncentral Folded normal Fréchet Gamma Generalized Inverse gamma/Gompertz Gompertz Shifted Half-logistic Half-normal Hotelling'sT-squared Hartman–Watson Inverse Gaussian Generalized Kolmogorov Lévy Log-Cauchy Log-Laplace Log-logistic Log-normal Log-t Lomax Matrix-exponential Maxwell–Boltzmann Maxwell–Jüttner Mittag-Leffler Nakagami Pareto Phase-type Poly-Weibull Rayleigh Relativistic Breit–Wigner Rice Truncated normal type-2 Gumbel Weibull Discrete Wilks's lambda
supported on the whole real line	Cauchy Exponential power Fisher'sz Kaniadakis κ-Gaussian Gaussianq Generalized hyperbolic Generalized logistic (logistic-beta) Generalized normal Geometric stable Gumbel Holtsmark Hyperbolic secant Johnson'sS_U Landau Laplace Asymmetric Logistic Noncentralt Normal (Gaussian) Normal-inverse Gaussian Skew normal Slash Stable Student'st Tracy–Widom Variance-gamma Voigt
with support whose type varies	Generalized chi-squared Generalized extreme value Generalized Pareto Marchenko–Pastur Kaniadakisκ-exponential Kaniadakisκ-Gamma Kaniadakisκ-Weibull Kaniadakisκ-Logistic Kaniadakisκ-Erlang q-exponential q-Gaussian q-Weibull Shifted log-logistic Tukey lambda