Part of a series on
Bayesian statistics
Posterior =Likelihood ×Prior ÷Evidence
Background
Bayesian inference Bayesian probability Bayes' theorem Bernstein–von Mises theorem Coherence Cox's theorem Cromwell's rule Likelihood principle Principle of indifference Principle of maximum entropy
Model building
Conjugate prior Linear regression Empirical Bayes Hierarchical model
Posterior approximation
Markov chain Monte Carlo Laplace's approximation Integrated nested Laplace approximations Variational inference Approximate Bayesian computation
Estimators
Bayesian estimator Credible interval Maximum a posteriori estimation
Evidence approximation
Evidence lower bound Nested sampling
Model evaluation
Bayes factor (Schwarz criterion) Model averaging Posterior predictive
Mathematics portal
v t e

Instatistics, theBayesian information criterion (BIC) orSchwarz information criterion (alsoSIC,SBC,SBIC) is a criterion formodel selection among a finite set of models; models with lower BIC are generally preferred. It is based, in part, on thelikelihood function and it is closely related to theAkaike information criterion (AIC).

When fitting models, it is possible to increase the maximum likelihood by adding parameters, but doing so may result inoverfitting. Both BIC and AIC attempt to resolve this problem by introducing a penalty term for the number of parameters in the model; the penalty term is larger in BIC than in AIC for sample sizes greater than 7.^[1]

The BIC was developed by Gideon E. Schwarz and published in a 1978 paper,^[2] as a large-sample approximation to theBayes factor.

The BIC is formally defined as^[3][a]

${\displaystyle \mathrm{B}\mathrm{I}\mathrm{C}=k\ln (n)-2\ln (\hat{L}).}$

where

• ${\displaystyle \hat{L}}$ = the maximized value of thelikelihood function of the model${\displaystyle M}$, i.e.${\displaystyle \hat{L}=p(x\mid \hat{\theta },M)}$, where${\displaystyle \{\hat{\theta }\}}$ are the parameter values that maximize the likelihood function and${\displaystyle x}$ is the observed data;
• ${\displaystyle n}$ = the number of data points in${\displaystyle x}$, the number ofobservations, or equivalently, the sample size;
• ${\displaystyle k}$ = the number ofparameters estimated by the model. For example, inmultiple linear regression, the estimated parameters are the intercept, the${\displaystyle q}$ slope parameters, and the constant variance of the errors; thus,${\displaystyle k=q+2}$.

The BIC can be derived by integrating out the parameters of the model usingLaplace's method, starting with the followingmodel evidence:^{[5][6]: 217}

${\displaystyle p(x\mid M)=\int p(x\mid \theta ,M)\pi (\theta \mid M)d\theta }$

where${\displaystyle \pi (\theta \mid M)}$ is the prior for${\displaystyle \theta }$ under model${\displaystyle M}$.

The log-likelihood,${\displaystyle \ln (p(x\mid \theta ,M))}$, is then expanded to a second orderTaylor series about theMLE,${\displaystyle \hat{\theta }}$, assuming it is twice differentiable as follows:

${\displaystyle \ln (p(x\mid \theta ,M))=\ln (\hat{L})-\frac{n}{2}(\theta -\hat{\theta }{)}^{\mathrm{T}}\mathcal{I}(\hat{\theta })(\theta -\hat{\theta })+R(x,\theta ),}$

where${\displaystyle \mathcal{I}(\theta )}$ is the averageobserved information per observation, and${\displaystyle R(x,\theta )}$ denotes the residual term. To the extent that${\displaystyle R(x,\theta )}$ is negligible and${\displaystyle \pi (\theta \mid M)}$ is relatively linear near${\displaystyle \hat{\theta }}$, we can integrate out${\displaystyle \theta }$ to get the following:

${\displaystyle p(x\mid M)\approx \hat{L}{\left(\frac{2\pi }{n}\right)}^{\frac{k}{2}}|\mathcal{I}(\hat{\theta }){|}^{-\frac{1}{2}}\pi (\hat{\theta })}$

As${\displaystyle n}$ increases, we can ignore${\displaystyle |\mathcal{I}(\hat{\theta })|}$ and${\displaystyle \pi (\hat{\theta })}$ as they are${\displaystyle O(1)}$. Thus,

${\displaystyle p(x\mid M)=\exp \left(\ln \hat{L}-\frac{k}{2}\ln (n)+O(1)\right)=\exp \left(-\frac{\mathrm{B}\mathrm{I}\mathrm{C}}{2}+O(1)\right),}$

where BIC is defined as above, and${\displaystyle \hat{L}}$ either (a) is the Bayesian posterior mode or (b) uses the MLE and the prior${\displaystyle \pi (\theta \mid M)}$ has nonzero slope at the MLE. Then the posterior

${\displaystyle p(M\mid x)\propto p(x\mid M)p(M)\approx \exp \left(-\frac{\mathrm{B}\mathrm{I}\mathrm{C}}{2}\right)p(M)}$

When picking from several models, ones with lower BIC values are generally preferred. The BIC is an increasingfunction of the error variance${\displaystyle {\sigma }_e^2}$ and an increasing function ofk. That is, unexplained variation in thedependent variable and the number of explanatory variables increase the value of BIC. However, a lower BIC does not necessarily indicate one model is better than another. Because it involves approximations, the BIC is merely a heuristic. In particular, differences in BIC should never be treated like transformed Bayes factors.

It is important to keep in mind that the BIC can be used to compare estimated models only when the numerical values of the dependent variable^[b] are identical for all models being compared. The models being compared need not benested, unlike the case when models are being compared using anF-test or alikelihood ratio test.^{[citation needed]}

This sectionneeds additional citations forverification. Please helpimprove this article byadding citations to reliable sources in this section. Unsourced material may be challenged and removed.(November 2011) (Learn how and when to remove this message)

• The BIC generally penalizes free parameters more strongly than theAkaike information criterion, though it depends on the size ofn and relative magnitude ofn and k.
• It is independent of the prior.
• It can measure the efficiency of the parameterized model in terms of predicting the data.
• It penalizes the complexity of the model where complexity refers to the number of parameters in the model.
• It is approximately equal to theminimum description length criterion but with negative sign.
• It can be used to choose the number of clusters according to the intrinsic complexity present in a particular dataset.
• It is closely related to other penalized likelihood criteria such asDeviance information criterion and theAkaike information criterion.

The BIC suffers from two main limitations^[7]

1. the above approximation is only valid for sample size${\displaystyle n}$ much larger than the number${\displaystyle k}$ of parameters in the model.
2. the BIC cannot handle complex collections of models as in the variable selection (orfeature selection) problem in high-dimension.^[7]

Under the assumption that the model errors or disturbances are independent and identically distributed according to anormal distribution and the boundary condition that the derivative of thelog likelihood with respect to the true variance is zero, this becomes (up to an additive constant, which depends only onn and not on the model):^[8]

${\displaystyle \mathrm{B}\mathrm{I}\mathrm{C}=n\ln (\widehat{{\sigma }_e^2})+k\ln (n)}$

where${\displaystyle \widehat{{\sigma }_e^2}}$ is the error variance. The error variance in this case is defined as

${\displaystyle \widehat{{\sigma }_e^2}=\frac{1}{n}\sum _{i=1}^n(x_i-{\hat{x}}_i{)}^2.}$

whichis a biased estimator for the true variance.

In terms of theresidual sum of squares (RSS) the BIC is

${\displaystyle \mathrm{B}\mathrm{I}\mathrm{C}=n\ln (\text{RSS}/n)+k\ln (n)}$

When testing multiple linear models against a saturated model, the BIC can be rewritten in terms of thedeviance${\displaystyle {\chi }^2}$ as:^[9]

${\displaystyle \mathrm{B}\mathrm{I}\mathrm{C}={\chi }^2+k\ln (n)}$

where${\displaystyle k}$ is the number of model parameters in the test.

• Bhat, H. S.; Kumar, N (2010)."On the derivation of the Bayesian Information Criterion"(PDF). Archived fromthe original(PDF) on 28 March 2012.
• Findley, D. F. (1991). "Counterexamples to parsimony and BIC".Annals of the Institute of Statistical Mathematics.43 (3):505–514.doi:10.1007/BF00053369.S2CID 58910242.
• Kass, R. E.; Wasserman, L. (1995). "A reference Bayesian test for nested hypotheses and its relationship to the Schwarz criterion".Journal of the American Statistical Association.90 (431):928–934.doi:10.2307/2291327.JSTOR 2291327.
• Liddle, A. R. (2007)."Information criteria for astrophysical model selection".Monthly Notices of the Royal Astronomical Society.377 (1):L74 –L78.arXiv:astro-ph/0701113.Bibcode:2007MNRAS.377L..74L.doi:10.1111/j.1745-3933.2007.00306.x.S2CID 2884450.
• McQuarrie, A. D. R.; Tsai, C.-L. (1998).Regression and Time Series Model Selection.World Scientific.

• Sparse Vector Autoregressive Modeling

• Model selection
• Bayesian inference
• Regression variable selection

• Articles with short description
• Short description is different from Wikidata
• All articles with unsourced statements
• Articles with unsourced statements from February 2019
• Articles needing additional references from November 2011
• All articles needing additional references