| Part of a series on |
| Bayesian statistics |
|---|
 |
| Posterior =Likelihood ×Prior ÷Evidence |
| Background |
| Model building |
| Posterior approximation |
| Estimators |
| Evidence approximation |
| Model evaluation |
Bayesian probability (/ˈbeɪziən/BAY-zee-ən or/ˈbeɪʒən/BAY-zhən)[1] is aninterpretation of the concept of probability, in which, instead offrequency orpropensity of some phenomenon, probability is interpreted as reasonable expectation[2] representing a state of knowledge[3] or as quantification of a personal belief.[4]
The Bayesian interpretation of probability can be seen as an extension ofpropositional logic that enables reasoning withhypotheses;[5][6] that is, with propositions whosetruth or falsity is unknown. In the Bayesian view, a probability is assigned to a hypothesis, whereas underfrequentist inference, a hypothesis is typically tested without being assigned a probability.
Bayesian probability belongs to the category of evidential probabilities; to evaluate the probability of a hypothesis, the Bayesian probabilist specifies aprior probability. This, in turn, is then updated to aposterior probability in the light of new, relevantdata (evidence).[7] The Bayesian interpretation provides a standard set of procedures and formulae to perform this calculation.
The termBayesian derives from the 18th-century mathematician and theologianThomas Bayes, who provided the first mathematical treatment of a non-trivial problem of statisticaldata analysis using what is now known asBayesian inference.[8]: 131 MathematicianPierre-Simon Laplace pioneered and popularized what is now called Bayesian probability.[8]: 97–98
Bayesian methods are characterized by concepts and procedures as follows:
Broadly speaking, there are two interpretations of Bayesian probability. For objectivists, who interpret probability as an extension oflogic,probability quantifies the reasonable expectation that everyone (even a "robot") who shares the same knowledge should share in accordance with the rules of Bayesian statistics, which can be justified byCox's theorem.[3][10] For subjectivists,probability corresponds to a personal belief.[4] Rationality and coherence allow for substantial variation within the constraints they pose; the constraints are justified by theDutch book argument or bydecision theory andde Finetti's theorem.[4] The objective and subjective variants of Bayesian probability differ mainly in their interpretation and construction of the prior probability.
The termBayesian derives fromThomas Bayes (1702–1761), who proved a special case of what is now calledBayes' theorem in a paper titled "An Essay Towards Solving a Problem in the Doctrine of Chances".[11] In that special case, the prior and posterior distributions werebeta distributions and the data came fromBernoulli trials. It wasPierre-Simon Laplace (1749–1827) who introduced a general version of the theorem and used it to approach problems incelestial mechanics, medical statistics,reliability, andjurisprudence.[12] Early Bayesian inference, which used uniform priors following Laplace'sprinciple of insufficient reason, was called "inverse probability" (because itinfers backwards from observations to parameters, or from effects to causes).[13] After the 1920s, "inverse probability" was largely supplanted by a collection of methods that came to be calledfrequentist statistics.[13]
In the 20th century, the ideas of Laplace developed in two directions, giving rise toobjective andsubjective currents in Bayesian practice.Harold Jeffreys'Theory of Probability (first published in 1939) played an important role in the revival of the Bayesian view of probability, followed by works byAbraham Wald (1950) andLeonard J. Savage (1954). The adjectiveBayesian itself dates to the 1950s; the derivedBayesianism,neo-Bayesianism is of 1960s coinage.[14][15][16] In the objectivist stream, the statistical analysis depends on only the model assumed and the data analysed.[17] No subjective decisions need to be involved. In contrast, "subjectivist" statisticians deny the possibility of fully objective analysis for the general case.
In the 1980s, there was a dramatic growth in research and applications of Bayesian methods, mostly attributed to the discovery ofMarkov chain Monte Carlo methods and the consequent removal of many of the computational problems, and to an increasing interest in nonstandard, complex applications.[18] While frequentist statistics remains strong (as demonstrated by the fact that much of undergraduate teaching is based on it[19]), Bayesian methods are widely accepted and used, e.g., in the field ofmachine learning.[20]
The use of Bayesian probabilities as the basis ofBayesian inference has been supported by several arguments, such asCox axioms, theDutch book argument, arguments based ondecision theory andde Finetti's theorem.
Richard T. Cox showed that Bayesian updating follows from several axioms, including twofunctional equations and a hypothesis of differentiability.[10][21] The assumption of differentiability or even continuity is controversial; Halpern found a counterexample based on his observation that the Boolean algebra of statements may be finite.[22] Other axiomatizations have been suggested by various authors with the purpose of making the theory more rigorous.[9]
Bruno de Finetti proposed the Dutch book argument based on betting. A cleverbookmaker makes aDutch book by setting theodds and bets to ensure that the bookmaker profits—at the expense of the gamblers—regardless of the outcome of the event (a horse race, for example) on which the gamblers bet. It is associated withprobabilities implied by the odds not beingcoherent.
However,Ian Hacking noted that traditional Dutch book arguments did not specify Bayesian updating: they left open the possibility that non-Bayesian updating rules could avoid Dutch books. For example,Hacking writes[23][24] "And neither the Dutch book argument, nor any other in the personalist arsenal of proofs of the probability axioms, entails the dynamic assumption. Not one entails Bayesianism. So the personalist requires the dynamic assumption to be Bayesian. It is true that in consistency a personalist could abandon the Bayesian model of learning from experience. Salt could lose its savour."
In fact, there are non-Bayesian updating rules that also avoid Dutch books (as discussed in the literature on "probability kinematics"[25] following the publication ofRichard C. Jeffrey's rule, which is itself regarded as Bayesian[26]). The additional hypotheses sufficient to (uniquely) specify Bayesian updating are substantial[27] and not universally seen as satisfactory.[28]
Adecision-theoretic justification of the use of Bayesian inference (and hence of Bayesian probabilities) was given byAbraham Wald, who proved that everyadmissible statistical procedure is either a Bayesian procedure or a limit of Bayesian procedures.[29] Conversely, every Bayesian procedure isadmissible.[30]
Following the work onexpected utilitytheory ofRamsey andvon Neumann, decision-theorists have accounted forrational behavior using a probability distribution for theagent.Johann Pfanzagl completed theTheory of Games and Economic Behavior by providing an axiomatization of subjective probability and utility, a task left uncompleted by von Neumann andOskar Morgenstern: their original theory supposed that all the agents had the same probability distribution, as a convenience.[31] Pfanzagl's axiomatization was endorsed by Oskar Morgenstern: "Von Neumann and I have anticipated ... [the question whether probabilities] might, perhaps more typically, be subjective and have stated specifically that in the latter case axioms could be found from which could derive the desired numerical utility together with a number for the probabilities (cf. p. 19 of The Theory of Games and Economic Behavior). We did not carry this out; it was demonstrated by Pfanzagl ... with all the necessary rigor".[32]
Ramsey andSavage noted that the individual agent's probability distribution could be objectively studied in experiments. Procedures fortesting hypotheses about probabilities (using finite samples) are due toRamsey (1931) andde Finetti (1931, 1937, 1964, 1970). BothBruno de Finetti[33][34] andFrank P. Ramsey[34][35] acknowledge their debts topragmatic philosophy, particularly (for Ramsey) toCharles S. Peirce.[34][35]
The "Ramsey test" for evaluating probability distributions is implementable in theory, and has kept experimental psychologists occupied for a half century.[36]This work demonstrates that Bayesian-probability propositions can befalsified, and so meet an empirical criterion ofCharles S. Peirce, whose work inspired Ramsey. (Thisfalsifiability-criterion was popularized byKarl Popper.[37][38])
Modern work on the experimental evaluation of personal probabilities uses the randomization,blinding, and Boolean-decision procedures of the Peirce-Jastrow experiment.[39] Since individuals act according to different probability judgments, these agents' probabilities are "personal" (but amenable to objective study).
Personal probabilities are problematic for science and for some applications where decision-makers lack the knowledge or time to specify an informed probability-distribution (on which they are prepared to act). To meet the needs of science and of human limitations, Bayesian statisticians have developed "objective" methods for specifying prior probabilities.
Indeed, some Bayesians have argued the prior state of knowledge definesthe (unique) prior probability-distribution for "regular" statistical problems; cf.well-posed problems. Finding the right method for constructing such "objective" priors (for appropriate classes of regular problems) has been the quest of statistical theorists from Laplace toJohn Maynard Keynes,Harold Jeffreys, andEdwin Thompson Jaynes. These theorists and their successors have suggested several methods for constructing "objective" priors (Unfortunately, it is not always clear how to assess the relative "objectivity" of the priors proposed under these methods):
Each of these methods contributes useful priors for "regular" one-parameter problems, and each prior can handle some challengingstatistical models (with "irregularity" or several parameters). Each of these methods has been useful in Bayesian practice. Indeed, methods for constructing "objective" (alternatively, "default" or "ignorance") priors have been developed by avowed subjective (or "personal") Bayesians likeJames Berger (Duke University) andJosé-Miguel Bernardo (Universitat de València), simply because such priors are needed for Bayesian practice, particularly in science.[40] The quest for "the universal method for constructing priors" continues to attract statistical theorists.[40]
Thus, the Bayesian statistician needs either to use informed priors (using relevant expertise or previous data) or to choose among the competing methods for constructing "objective" priors.
The works ofWald,Statistical Decision Functions (1950) andSavage,The Foundation of Statistics (1954) are commonly regarded starting points for current Bayesian approaches
This revolution, which may or may not succeed, is neo-Bayesianism. Jeffreys tried to introduce this approach, but did not succeed at the time in giving it general appeal.
It is curious that even in its activities unrelated to ethics, humanity searches for a religion. At the present time, the religion being 'pushed' the hardest is Bayesianism.
(updated printing, 2007, of Holden-Day, 1976)
Updated classic textbook. Bayesian theory clearly presented
