Dirichlet-Multinomial
Notation	${\displaystyle \mathrm{D}\mathrm{i}\mathrm{r}\mathrm{M}\mathrm{u}\mathrm{l}\mathrm{t}(n,\mathit{\alpha })}$
Parameters	${\displaystyle n\in \{0,1,2,\dots \}}$ number of trials ${\displaystyle {\alpha }_1,\dots ,{\alpha }_K>0,{\alpha }_0=\sum {\alpha }_k}$
Support	${\displaystyle x_i\in \{0,\dots ,n\}}$ ${\displaystyle \mathrm{\Sigma }{x}_i=n,1\le i\le K}$
PMF	${\displaystyle \frac{\mathrm{\Gamma }\left({\alpha }_0\right)\mathrm{\Gamma }\left(n+1\right)}{\mathrm{\Gamma }\left(n+{\alpha }_0\right)}\prod _{k=1}^K\frac{\mathrm{\Gamma }(x_k+{\alpha }_k)}{\mathrm{\Gamma }({\alpha }_k)\mathrm{\Gamma }\left(x_k+1\right)}}$[1]
Mean	${\displaystyle \mathrm{E}(X_i)=n\frac{{\alpha }_i}{{\alpha }_0}}$
Variance	${\displaystyle \mathrm{Var}(X_i)=n\frac{{\alpha }_i}{{\alpha }_0}\left(1-\frac{{\alpha }_i}{{\alpha }_0}\right)\left(\frac{n+{\alpha }_0}{1+{\alpha }_0}\right)}$ ${\displaystyle \mathrm{C}\mathrm{o}\mathrm{v}(X_i,X_k)=-n\frac{{\alpha }_i}{{\alpha }_0}\frac{{\alpha }_k}{{\alpha }_0}\left(\frac{n+{\alpha }_0}{1+{\alpha }_0}\right)(i\ne k)}$
MGF	${\displaystyle \mathrm{E}(\prod _{k=1}^K{e}^{t_k\cdot x_k})=\frac{\mathrm{\Gamma }({\alpha }_0)\mathrm{\Gamma }(n+1)}{\mathrm{\Gamma }(n+{\alpha }_0)}\cdot D_n(\mathit{\alpha },(e^{t_1},...,e^{t_K}))}$ with ${\displaystyle D_n=\frac{1}{n}\sum _{u=1}^n\left[\left(\sum _{k=1}^K{\alpha }_k\cdot e^{t_k\cdot u}\right)D_{n-u}\right],D_0=1}$[1]
CF	${\displaystyle \mathrm{E}(\prod _{k=1}^K{e}^{i{t}_k\cdot x_k})=\frac{\mathrm{\Gamma }({\alpha }_0)\mathrm{\Gamma }(n+1)}{\mathrm{\Gamma }(n+{\alpha }_0)}\cdot D_n(\mathit{\alpha },(e^{i{t}_1},...,e^{i{t}_K}))}$ with ${\displaystyle D_n=\frac{1}{n}\sum _{u=1}^n\left[\left(\sum _{k=1}^K{\alpha }_k\cdot e^{i{t}_k\cdot u}\right)D_{n-u}\right],D_0=1}$[1]
PGF	${\displaystyle \mathrm{E}(\prod _{k=1}^K{z_k}^{x_k})=\frac{\mathrm{\Gamma }({\alpha }_0)\mathrm{\Gamma }(n+1)}{\mathrm{\Gamma }(n+{\alpha }_0)}\cdot D_n(\mathit{\alpha },\mathbf{z})}$ with ${\displaystyle D_n=\frac{1}{n}\sum _{u=1}^n\left[\left(\sum _{k=1}^K{\alpha }_k\cdot {z_k}^u\right)D_{n-u}\right],D_0=1}$[1]

Inprobability theory andstatistics, theDirichlet-multinomial distribution is a family of discrete multivariateprobability distributions on a finite support of non-negative integers. It is also called theDirichlet compound multinomial distribution (DCM) ormultivariate Pólya distribution (afterGeorge Pólya). It is acompound probability distribution, where a probability vectorp is drawn from aDirichlet distribution with parameter vector${\displaystyle \mathit{\alpha }}$, and an observation drawn from amultinomial distribution with probability vectorp and number of trialsn. The Dirichlet parameter vector captures the prior belief about the situation and can be seen as a pseudocount: observations of each outcome that occur before the actual data is collected. The compounding corresponds to aPólya urn scheme. It is frequently encountered inBayesian statistics,machine learning,empirical Bayes methods andclassical statistics as anoverdispersedmultinomial distribution.

It reduces to thecategorical distribution as a special case whenn = 1. It also approximates themultinomial distribution arbitrarily well for largeα. The Dirichlet-multinomial is a multivariate extension of thebeta-binomial distribution, as the multinomial and Dirichlet distributions are multivariate versions of thebinomial distribution andbeta distributions, respectively.

The Dirichlet distribution is aconjugate distribution to the multinomial distribution. This fact leads to an analytically tractablecompound distribution.For a random vector of category counts${\displaystyle \mathbf{x}=(x_1,\dots ,x_K)}$, distributed according to amultinomial distribution, themarginal distribution is obtained by integrating on the distribution forp which can be thought of as arandom vector following a Dirichlet distribution:

${\displaystyle Pr(\mathbf{x}\mid n,\mathit{\alpha })={\int }_{\mathbf{p}}\mathrm{M}\mathrm{u}\mathrm{l}\mathrm{t}(\mathbf{x}\mid n,\mathbf{p})\mathrm{D}\mathrm{i}\mathrm{r}(\mathbf{p}\mid \mathit{\alpha })\text{d}\mathbf{p}}$

which results in the following explicit formula:

${\displaystyle Pr(\mathbf{x}\mid n,\mathit{\alpha })=\frac{\mathrm{\Gamma }\left({\alpha }_0\right)\mathrm{\Gamma }\left(n+1\right)}{\mathrm{\Gamma }\left(n+{\alpha }_0\right)}\prod _{k=1}^K\frac{\mathrm{\Gamma }(x_k+{\alpha }_k)}{\mathrm{\Gamma }({\alpha }_k)\mathrm{\Gamma }\left(x_k+1\right)}}$

where${\displaystyle {\alpha }_0}$ is defined as the sum${\displaystyle {\alpha }_0=\sum {\alpha }_k}$. Another form for this same compound distribution, written more compactly in terms of thebeta function,B, is as follows:

${\displaystyle Pr(\mathbf{x}\mid n,\mathit{\alpha })=\frac{nB\left({\alpha }_0,n\right)}{\prod _{k:x_k>0}{x}_{k}B\left({\alpha }_k,x_k\right)}.}$

The latter form emphasizes the fact that zero count categories can be ignored in the calculation - a useful fact when the number of categories is very large andsparse (e.g. word counts in documents).

Observe that the pdf is the Beta-binomial distribution when${\displaystyle K=2}$. It can also be shown that it approaches the multinomial distribution as${\displaystyle {\alpha }_0}$ approaches infinity. The parameter${\displaystyle {\alpha }_0}$ governs the degree of overdispersion orburstiness relative to the multinomial. Alternative choices to denote${\displaystyle {\alpha }_0}$ found in the literature are S and A.

The Dirichlet-multinomial distribution can also be motivated via anurn model for positiveinteger values of the vector${\displaystyle \mathit{\alpha }}$, known as thePolya urn model. Specifically, imagine an urn containing balls of${\displaystyle K}$ colors numbering${\displaystyle {\alpha }_i}$ for the ith color, where random draws are made. When a ball is randomly drawn and observed, then two balls of the same color are returned to the urn. If this is performed${\displaystyle n}$ times, then the probability of observing the random vector${\displaystyle x}$ of color counts is a Dirichlet-multinomial with parameters${\displaystyle n}$ and${\displaystyle \mathit{\alpha }}$.If the random draws are with simple replacement (no balls over and above the observed ball are added to the urn), then the distribution follows a multinomial distribution and if the random draws are made without replacement, the distribution follows amultivariate hypergeometric distribution.

Once again, let${\displaystyle {\alpha }_0=\sum {\alpha }_k}$ and let${\displaystyle p_i=\frac{{\alpha }_i}{\sum {\alpha }_k}=\frac{{\alpha }_i}{{\alpha }_0}}$, then theexpected number of times the outcomei was observed overn trials is

${\displaystyle \mathrm{E}(X_i)=n{p}_i=n\frac{{\alpha }_i}{{\alpha }_0}.}$

Thecovariance matrix is as follows. Each diagonal entry is thevariance of a beta-binomially distributed random variable, and is therefore

${\displaystyle \mathrm{var}(X_i)=n{p}_i(1-p_i)\left(\frac{n+\sum {\alpha }_k}{1+\sum {\alpha }_k}\right)=n\frac{{\alpha }_i}{{\alpha }_0}\left(1-\frac{{\alpha }_i}{{\alpha }_0}\right)\left(\frac{n+{\alpha }_0}{1+{\alpha }_0}\right).}$

The off-diagonal entries are thecovariances:

${\displaystyle \mathrm{cov}(X_i,X_j)=-n{p}_i{p}_j\left(\frac{n+\sum {\alpha }_k}{1+\sum {\alpha }_k}\right)=-n\frac{{\alpha }_i{\alpha }_j}{{\alpha }_0^2}\left(\frac{n+{\alpha }_0}{1+{\alpha }_0}\right)}$

fori,j distinct.

All covariances are negative because for fixedn, an increase in one component of a Dirichlet-multinomial vector requires a decrease in another component.

This is aK ×Kpositive-semidefinite matrix ofrankK − 1.

The entries of the correspondingcorrelation matrix are

${\displaystyle \rho (X_i,X_i)=1.}$

${\displaystyle \rho (X_i,X_j)=\frac{\mathrm{cov}(X_i,X_j)}{\sqrt{\mathrm{var}(X_i)\mathrm{var}(X_j)}}=\frac{-p_i{p}_j(\frac{n+{\alpha }_0}{1+{\alpha }_0})}{\sqrt{p_i(1-p_i)(\frac{n+{\alpha }_0}{1+{\alpha }_0})p_j(1-p_j)(\frac{n+{\alpha }_0}{1+{\alpha }_0})}}=-\sqrt{\frac{{\alpha }_i{\alpha }_j}{({\alpha }_0-{\alpha }_i)({\alpha }_0-{\alpha }_j)}}.}$

The sample size drops out of this expression.

Each of thek components separately has a beta-binomial distribution.

Thesupport of the Dirichlet-multinomial distribution is the set

${\displaystyle \{(n_1,\dots ,n_k)\in {\mathbb{N}}^k|n_1+\cdots +n_k=n\}.}$

Its number of elements is

${\displaystyle (\genfrac{}{}{0ex}{}{n+k-1}{k-1}).}$

In matrix notation,

${\displaystyle \mathrm{E}(\mathbf{X})=n\mathbf{p},}$

and

${\displaystyle \mathrm{var}(\mathbf{X})=n\{\mathrm{diag}(\mathbf{p})-\mathbf{p}{\mathbf{p}}^{\mathrm{T}}\}\left(\frac{n+{\alpha }_0}{1+{\alpha }_0}\right),}$

withp^T = the row vector transpose of the column vectorp. Letting

${\displaystyle {\alpha }_0=\frac{1-{\rho }^2}{{\rho }^2}}$, we can write alternatively

${\displaystyle \mathrm{var}(\mathbf{X})=n\{\mathrm{diag}(\mathbf{p})-\mathbf{p}{\mathbf{p}}^{\mathrm{T}}\}(1+{\rho }^2(n-1)),}$

The parameter${\displaystyle \rho }$ is known as the "intra class" or "intra cluster" correlation. It is this positive correlation which gives rise to overdispersion relative to the multinomial distribution.

If

${\displaystyle X=(X_1,\dots ,X_K)\sim \mathrm{DM}({\alpha }_1,\cdots ,{\alpha }_K)}$

then, if the random variables with subscriptsi andj are dropped from the vector and replaced by their sum^{[citation needed]},

${\displaystyle X^{\prime }=(X_1,\dots ,X_i+X_j,\dots ,X_K)\sim \mathrm{DM}\left({\alpha }_1,\cdots ,{\alpha }_i+{\alpha }_j,\cdots ,{\alpha }_K\right).}$

This aggregation property may be used to derive the marginal distribution of${\displaystyle X_i}$.

Conceptually, we are makingN independent draws from a categorical distribution withK categories. Let us represent the independent draws as random categorical variables${\displaystyle z_n}$ for${\displaystyle n=1\dots N}$. Let us denote the number of times a particular category${\displaystyle k}$ has been seen (for${\displaystyle k=1\dots K}$) among all the categorical variables as${\displaystyle n_k}$, and${\displaystyle \sum _k{n}_k=N}$. Then, we have two separate views onto this problem:

1. A set of${\displaystyle N}$ categorical variables${\displaystyle z_1,\dots ,z_N}$.
2. A single vector-valued variable${\displaystyle \mathbf{x}=(n_1,\dots ,n_K)}$, distributed according to amultinomial distribution.

The former case is a set of random variables specifying eachindividual outcome, while the latter is a variable specifying thenumber of outcomes of each of theK categories. The distinction is important, as the two cases have correspondingly different probability distributions.

The parameter of the categorical distribution is${\displaystyle \mathbf{p}=(p_1,p_2,\dots ,p_K),}$ where${\displaystyle p_k}$ is the probability to draw value${\displaystyle k}$;${\displaystyle \mathbf{p}}$ is likewise the parameter of the multinomial distribution${\displaystyle P(\mathbf{x}|\mathbf{p})}$. Rather than specifying${\displaystyle \mathbf{p}}$ directly, we give it aconjugate prior distribution, and hence it is drawn from a Dirichlet distribution with parameter vector${\displaystyle \mathit{\alpha }=({\alpha }_1,{\alpha }_2,\dots ,{\alpha }_K)}$.

By integrating out${\displaystyle \mathbf{p}}$, we obtain a compound distribution. However, the form of the distribution is different depending on which view we take.

For categorical variables${\displaystyle \mathbb{Z}=z_1,\dots ,z_N}$, themarginaljoint distribution is obtained by integrating out${\displaystyle \mathbf{p}}$:

${\displaystyle Pr(\mathbb{Z}\mid \mathit{\alpha })={\int }_{\mathbf{p}}Pr(\mathbb{Z}\mid \mathbf{p})Pr(\mathbf{p}\mid \mathit{\alpha })\text{d}\mathbf{p}}$

which results in the following explicit formula:

${\displaystyle Pr(\mathbb{Z}\mid \mathit{\alpha })=\frac{\mathrm{\Gamma }\left({\alpha }_0\right)}{\mathrm{\Gamma }\left(N+{\alpha }_0\right)}\prod _{k=1}^K\frac{\mathrm{\Gamma }(n_k+{\alpha }_k)}{\mathrm{\Gamma }({\alpha }_k)}}$

where${\displaystyle \mathrm{\Gamma }}$ is thegamma function, with

${\displaystyle {\alpha }_0=\sum _k{\alpha }_k\text{ and }N=\sum _k{n}_k\text{, and where }{n}_k=\text{number of }{z}_n\text{'s with the value }k.}$

Note the absence of the multinomial coefficient due to the formula being about the probability of a sequence of categorical variables instead of a probability on the counts within each category.

Although the variables${\displaystyle z_1,\dots ,z_N}$ do not appear explicitly in the above formula, they enter in through the${\displaystyle n_k}$ values.^{[clarification needed]}

Another useful formula, particularly in the context ofGibbs sampling, asks what the conditional density of a given variable${\displaystyle z_n}$ is, conditioned on all the other variables (which we will denote${\displaystyle {\mathbb{Z}}^{(-n)}}$). It turns out to have an extremely simple form:

${\displaystyle Pr(z_n=k\mid {\mathbb{Z}}^{(-n)},\mathit{\alpha })\propto n_k^{(-n)}+{\alpha }_k}$

where${\displaystyle n_k^{(-n)}}$ specifies the number of counts of category${\displaystyle k}$ seen in all variables other than${\displaystyle z_n}$.

It may be useful to show how to derive this formula. In general,conditional distributions are proportional to the correspondingjoint distributions, so we simply start with the above formula for the joint distribution of all the${\displaystyle z_1,\dots ,z_N}$ values and then eliminate any factors not dependent on the particular${\displaystyle z_n}$ in question. To do this, we make use of the notation${\displaystyle n_k^{(-n)}}$ defined above, and

We also use the fact that

${\displaystyle \mathrm{\Gamma }(n+1)=n\mathrm{\Gamma }(n)}$

Then:

In general, it is not necessary to worry about thenormalizing constant at the time of deriving the equations for conditional distributions. The normalizing constant will be determined as part of the algorithm for sampling from the distribution (seeCategorical distribution#Sampling). However, when the conditional distribution is written in the simple form above, it turns out that the normalizing constant assumes a simple form:

${\displaystyle \sum _k\left(n_k^{(-n)}+{\alpha }_k\right)={\alpha }_0+\sum _k{n}_k^{(-n)}={\alpha }_0+N-1}$

Hence

${\displaystyle Pr(z_n=k\mid {\mathbb{Z}}^{(-n)},\mathit{\alpha })=\frac{n_k^{(-n)}+{\alpha }_k}{{\alpha }_0+N-1}}$

This formula is closely related to theChinese restaurant process, which results from taking the limit as${\displaystyle K\to \mathrm{\infty }}$.

In a largerBayesian network in which categorical (or so-called "multinomial") distributions occur withDirichlet distribution priors as part of a larger network, all Dirichlet priors can be collapsed provided that the only nodes depending on them are categorical distributions. The collapsing happens for each Dirichlet-distribution node separately from the others, and occurs regardless of any other nodes that may depend on the categorical distributions. It also occurs regardless of whether the categorical distributions depend on nodes additional to the Dirichlet priors (although in such a case, those other nodes must remain as additional conditioning factors). Essentially, all of the categorical distributions depending on a given Dirichlet-distribution node become connected into a single Dirichlet-multinomial joint distribution defined by the above formula. The joint distribution as defined this way will depend on the parent(s) of the integrated-out Dirichet prior nodes, as well as any parent(s) of the categorical nodes other than the Dirichlet prior nodes themselves.

In the following sections, we discuss different configurations commonly found in Bayesian networks. We repeat the probability density from above, and define it using the symbol${\displaystyle \mathrm{DirMult}(\mathbb{Z}\mid \mathit{\alpha })}$:

${\displaystyle Pr(\mathbb{Z}\mid \mathit{\alpha })=\mathrm{DirMult}(\mathbb{Z}\mid \mathit{\alpha })=\frac{\mathrm{\Gamma }\left(\sum _k{\alpha }_k\right)}{\mathrm{\Gamma }\left(\sum _k{n}_k+{\alpha }_k\right)}\prod _{k=1}^K\frac{\mathrm{\Gamma }(n_k+{\alpha }_k)}{\mathrm{\Gamma }({\alpha }_k)}}$

Imagine we have a hierarchical model as follows:

In cases like this, we have multiple Dirichet priors, each of which generates some number of categorical observations (possibly a different number for each prior). The fact that they are all dependent on the same hyperprior, even if this is a random variable as above, makes no difference. The effect of integrating out a Dirichlet prior links the categorical variables attached to that prior, whose joint distribution simply inherits any conditioning factors of the Dirichlet prior. The fact that multiple priors may share a hyperprior makes no difference:

${\displaystyle Pr(\mathbb{Z}\mid \mathit{\alpha })=\prod _d\mathrm{DirMult}({\mathbb{Z}}_d\mid \mathit{\alpha })}$

where${\displaystyle {\mathbb{Z}}_d}$ is simply the collection of categorical variables dependent on priord.

Accordingly, the conditional probability distribution can be written as follows:

${\displaystyle Pr(z_{dn}=k\mid {\mathbb{Z}}^{(-dn)},\mathit{\alpha })\propto n_{k,d}^{(-n)}+{\alpha }_k}$

where${\displaystyle n_{k,d}^{(-n)}}$ specifically means the number of variablesamong the set${\displaystyle {\mathbb{Z}}_d}$, excluding${\displaystyle z_{dn}}$ itself, that have the value${\displaystyle k}$ .

It is necessary to countonly the variables having the valuek that are tied together to the variable in question through having the same prior. We donot want to count any other variables also having the valuek.

Now imagine a slightly more complicated hierarchical model as follows:

This model is the same as above, but in addition, each of the categorical variables has a child variable dependent on it. This is typical of amixture model.

Again, in the joint distribution, only the categorical variables dependent on the same prior are linked into a single Dirichlet-multinomial:

${\displaystyle Pr(\mathbb{Z},\mathbb{W}\mid \mathit{\alpha },\mathit{\varphi })=\prod _d\mathrm{DirMult}({\mathbb{Z}}_d\mid \mathit{\alpha })\prod _{d=1}^M\prod _{n=1}^{N_d}\mathrm{F}(w_{dn}\mid z_{dn},\mathit{\varphi })}$

The conditional distribution of the categorical variables dependent only on their parents and ancestors would have the identical form as above in the simpler case. However, in Gibbs sampling it is necessary to determine the conditional distribution of a given node${\displaystyle z_{dn}}$ dependent not only on${\displaystyle {\mathbb{Z}}^{(-dn)}}$ and ancestors such as${\displaystyle \alpha }$ but onall the other parameters.

The simplified expression for the conditional distribution is derived above simply by rewriting the expression for the joint probability and removing constant factors. Hence, the same simplification would apply in a larger joint probability expression such as the one in this model, composed of Dirichlet-multinomial densities plus factors for many other random variables dependent on the values of the categorical variables.

This yields the following:

${\displaystyle Pr(z_{dn}=k\mid {\mathbb{Z}}^{(-dn)},\mathbb{W},\mathit{\alpha },\mathit{\varphi })\propto (n_{k,d}^{(-n)}+{\alpha }_k)\mathrm{F}(w_{dn}\mid z_{dn},\mathit{\varphi })}$

Here the probability density of${\displaystyle \mathrm{F}}$ appears directly. To dorandom sampling over${\displaystyle z_{dn}}$, we would compute the unnormalized probabilities for allK possibilities for${\displaystyle z_{dn}}$ using the above formula, then normalize them and proceed as normal using the algorithm described in thecategorical distribution article.

Correctly speaking, the additional factor that appears in the conditional distribution is derived not from the model specification but directly from the joint distribution. This distinction is important when considering models where a given node with Dirichlet-prior parent has multiple dependent children, particularly when those children are dependent on each other (e.g. if they share a parent that is collapsed out). This is discussed more below.

Now imagine we have a hierarchical model as follows:

Here we have a tricky situation where we have multiple Dirichlet priors as before and a set of dependent categorical variables, but the relationship between the priors and dependent variables isn't fixed, unlike before. Instead, the choice of which prior to use is dependent on another random categorical variable. This occurs, for example, in topic models, and indeed the names of the variables above are meant to correspond to those inlatent Dirichlet allocation. In this case, the set${\displaystyle \mathbb{W}}$ is a set of words, each of which is drawn from one of${\displaystyle K}$ possible topics, where each topic is a Dirichlet prior over a vocabulary of${\displaystyle V}$ possible words, specifying the frequency of different words in the topic. However, the topic membership of a given word isn't fixed; rather, it's determined from a set oflatent variables${\displaystyle \mathbb{Z}}$. There is one latent variable per word, a${\displaystyle K}$ -dimensionalcategorical variable specifying the topic the word belongs to.

In this case, all variables dependent on a given prior are tied together (i.e.correlated) in a group, as before — specifically, all words belonging to a given topic are linked. In this case, however, the group membership shifts, in that the words are not fixed to a given topic but the topic depends on the value of a latent variable associated with the word. However, the definition of the Dirichlet-multinomial density doesn't actually depend on the number of categorical variables in a group (i.e. the number of words in the document generated from a given topic), but only on the counts of how many variables in the group have a given value (i.e. among all the word tokens generated from a given topic, how many of them are a given word). Hence, we can still write an explicit formula for the joint distribution:

${\displaystyle Pr(\mathbb{W}\mid \mathit{\alpha },\mathbb{Z})=\prod _{k=1}^K\mathrm{DirMult}({\mathbb{W}}_k\mid \mathbb{Z},\mathit{\alpha })=\prod _{k=1}^K\left[\frac{\mathrm{\Gamma }\left(\sum _v{\alpha }_v\right)}{\mathrm{\Gamma }\left(\sum _v{n}_v^k+{\alpha }_v\right)}\prod _{v=1}^V\frac{\mathrm{\Gamma }(n_v^k+{\alpha }_v)}{\mathrm{\Gamma }({\alpha }_v)}\right]}$

Here we use the notation${\displaystyle n_v^k}$ to denote the number of word tokens whose value is word symbolv and which belong to topick.

The conditional distribution still has the same form:

${\displaystyle Pr(w_n=v\mid {\mathbb{W}}^{(-n)},\mathbb{Z},\mathit{\alpha })\propto n_v^{k,(-n)}+{\alpha }_v}$

Here again,only the categorical variables for words belonging to a given topic are linked (even though this linking will depend on the assignments of the latent variables), and hence the word counts need to be overonly the words generated by a given topic. Hence the symbol${\displaystyle n_v^{k,(-n)}}$, which is the count of words tokens having the word symbolv, butonly among those generated by topick, and excluding the word itself whose distribution is being described.

(The reason why excluding the word itself is necessary, and why it even makes sense at all, is that in aGibbs sampling context, we repeatedly resample the values of each random variable, after having run through and sampled all previous variables. Hence the variable will already have a value, and we need to exclude this existing value from the various counts that we make use of.)

We now show how to combine some of the above scenarios to demonstrate how toGibbs sample a real-world model, specifically a smoothedlatent Dirichlet allocation (LDA)topic model.

The model is as follows:

Essentially we combine the previous three scenarios: We have categorical variables dependent on multiple priors sharing a hyperprior; we have categorical variables with dependent children (thelatent variable topic identities); and we have categorical variables with shifting membership in multiple priors sharing a hyperprior. In the standard LDA model, the words are completely observed, and hence we never need to resample them. (However, Gibbs sampling would equally be possible if only some or none of the words were observed. In such a case, we would want to initialize the distribution over the words in some reasonable fashion — e.g. from the output of some process that generates sentences, such as amachine translation model — in order for the resultingposterior latent variable distributions to make any sense.)

Using the above formulas, we can write down the conditional probabilities directly:

Here we have defined the counts more explicitly to clearly separate counts of words and counts of topics:

As in the scenario above with categorical variables with dependent children, the conditional probability of those dependent children appears in the definition of the parent's conditional probability. In this case, each latent variable has only a single dependent child word, so only one such term appears. (If there were multiple dependent children, all would have to appear in the parent's conditional probability, regardless of whether there was overlap between different parents and the same children, i.e. regardless of whether the dependent children of a given parent also have other parents. In a case where a child has multiple parents, the conditional probability for that child appears in the conditional probability definition of each of its parents.)

The definition above specifies only theunnormalized conditional probability of the words, while the topic conditional probability requires theactual (i.e. normalized) probability. Hence we have to normalize by summing over all word symbols:

where

It's also worth making another point in detail, which concerns the second factor above in the conditional probability. Remember that the conditional distribution in general is derived from the joint distribution, and simplified by removing terms not dependent on the domain of the conditional (the part on the left side of the vertical bar). When a node${\displaystyle z}$ has dependent children, there will be one or more factors${\displaystyle \mathrm{F}(\cdots \mid z)}$ in the joint distribution that are dependent on${\displaystyle z}$.Usually there is one factor for each dependent node, and it has the same density function as the distribution appearing the mathematical definition. However, if a dependent node has another parent as well (a co-parent), and that co-parent is collapsed out, then the node will become dependent on all other nodes sharing that co-parent, and in place of multiple terms for each such node, the joint distribution will have only one joint term. We have exactly that situation here. Even though${\displaystyle z_{dn}}$ has only one child${\displaystyle w_{dn}}$, that child has a Dirichlet co-parent that we have collapsed out, which induces a Dirichlet-multinomial over the entire set of nodes${\displaystyle {\mathbb{W}}^k}$.

It happens in this case that this issue does not cause major problems, precisely because of the one-to-one relationship between${\displaystyle z_{dn}}$ and${\displaystyle w_{dn}}$. We can rewrite the joint distribution as follows:

where in the set${\displaystyle {\mathbb{W}}^{k,(-dn)}}$ (i.e. the set of nodes${\displaystyle {\mathbb{W}}^k}$ excluding${\displaystyle w_{dn}}$ ), none of the nodes have${\displaystyle z_{dn}}$ as a parent. Hence it can be eliminated as a conditioning factor (line 2), meaning that the entire factor can be eliminated from the conditional distribution (line 3).

Here is another model, with a different set of issues. This is an implementation of an unsupervisedNaive Bayes model for document clustering. That is, we would like toclassify documents into multiple categories (e.g. "spam" or "non-spam", or "scientific journal article", "newspaper article about finance", "newspaper article about politics", "love letter") based on textual content. However, we don't already know the correct category of any documents; instead, we want tocluster them based on mutual similarities. (For example, a set of scientific articles will tend to be similar to each other in word use but very different from a set of love letters.) This is a type ofunsupervised learning. (The same technique can be used for doingsemi-supervised learning, i.e. where we know the correct category of some fraction of the documents and would like to use this knowledge to help in clustering the remaining documents.)

The model is as follows:

In many ways, this model is very similar to theLDAtopic model described above, but it assumes one topic per document rather than one topic per word, with a document consisting of a mixture of topics. This can be seen clearly in the above model, which is identical to the LDA model except that there is only onelatent variable per document instead of one per word. Once again, we assume that we are collapsing all of the Dirichlet priors.

The conditional probability for a given word is almost identical to the LDA case. Once again, all words generated by the same Dirichlet prior are interdependent. In this case, this means the words of all documents having a given label — again, this can vary depending on the label assignments, but all we care about is the total counts. Hence:

where

However, there is a critical difference in the conditional distribution of the latent variables for the label assignments, which is that a given label variable has multiple children nodes instead of just one — in particular, the nodes for all the words in the label's document. This relates closely to the discussion above about the factor${\displaystyle \mathrm{F}(\cdots \mid z_d)}$ that stems from the joint distribution. In this case, the joint distribution needs to be taken over all words in all documents containing a label assignment equal to the value of${\displaystyle z_d}$, and has the value of a Dirichlet-multinomial distribution. Furthermore, we cannot reduce this joint distribution down to a conditional distribution over a single word. Rather, we can reduce it down only to a smaller joint conditional distribution over the words in the document for the label in question, and hence we cannot simplify it using the trick above that yields a simple sum of expected count and prior. Although it is in fact possible to rewrite it as a product of such individual sums, the number of factors is very large, and is not clearly more efficient than directly computing the Dirichlet-multinomial distribution probability.

The one-dimensional version of the Dirichlet-multinomial distribution is known as theBeta-binomial distribution.

The Dirichlet-multinomial distribution has a relationship with thenegative binomial distribution analogous to the relationship of themultinomial distribution with thePoisson distribution.^[2]

The Dirichlet-multinomial distribution is used in automateddocument classification and clustering,genetics,economy, combat modeling, and quantitative marketing.

This article includes a list ofgeneral references, butit lacks sufficient correspondinginline citations. Please help toimprove this article byintroducing more precise citations.(June 2012) (Learn how and when to remove this message)

• Beta-binomial distribution
• Chinese restaurant process
• Stars and bars (combinatorics)
• Dirichlet process
• Generalized Dirichlet distribution
• Krichevsky–Trofimov estimator
• Dirichlet negative multinomial distribution

• Multivariate discrete distributions
• Discrete distributions
• Compound probability distributions

• Articles with short description
• Short description matches Wikidata
• All articles with unsourced statements
• Articles with unsourced statements from December 2021
• Wikipedia articles needing clarification from June 2021
• Articles lacking in-text citations from June 2012
• All articles lacking in-text citations