Inmachine learning,diffusion models, also known asdiffusion-based generative models orscore-based generative models, are a class oflatent variablegenerative models. A diffusion model consists of two major components: the forward diffusion process, and the reverse sampling process. The goal of diffusion models is to learn adiffusion process for a given dataset, such that the process can generate new elements that are distributed similarly as the original dataset. A diffusion model models data as generated by a diffusion process, whereby a new datum performs arandom walk with drift through the space of all possible data.^[1] A trained diffusion model can be sampled in many ways, with different efficiency and quality.

There are various equivalent formalisms, includingMarkov chains, denoising diffusion probabilistic models, noise conditioned score networks, and stochastic differential equations.^[2] They are typically trained usingvariational inference.^[3] The model responsible for denoising is typically called its "backbone". The backbone may be of any kind, but they are typicallyU-nets ortransformers.

As of 2024^[update], diffusion models are mainly used forcomputer vision tasks, includingimage denoising,inpainting,super-resolution,image generation, and video generation. These typically involve training a neural network to sequentiallydenoise images blurred withGaussian noise.^[1][4] The model is trained to reverse the process of adding noise to an image. After training to convergence, it can be used for image generation by starting with an image composed of random noise, and applying the network iteratively to denoise the image.

Diffusion-based image generators have seen widespread commercial interest, such asStable Diffusion andDALL-E. These models typically combine diffusion models with other models, such as text-encoders and cross-attention modules to allow text-conditioned generation.^[5]

Other than computer vision, diffusion models have also found applications innatural language processing^[6] such astext generation^[7] andsummarization,^[8] sound generation,^[9] andreinforcement learning.^[10][11]

Diffusion models were introduced in 2015 as a method to train a model that can sample from a highly complex probability distribution. They used techniques fromnon-equilibrium thermodynamics, especiallydiffusion.^[12]

Consider, for example, how one might model the distribution of all naturally occurring photos. Each image is a point in the space of all images, and the distribution of naturally occurring photos is a "cloud" in space, which, by repeatedly adding noise to the images, diffuses out to the rest of the image space, until the cloud becomes all but indistinguishable from aGaussian distribution${\displaystyle \mathcal{N}(0,I)}$. A model that can approximately undo the diffusion can then be used to sample from the original distribution. This is studied in "non-equilibrium" thermodynamics, as the starting distribution is not in equilibrium, unlike the final distribution.

The equilibrium distribution is the Gaussian distribution${\displaystyle \mathcal{N}(0,I)}$, with pdf${\displaystyle \rho (x)\propto e^{-\frac{1}{2}\Vert x{\Vert }^2}}$. This is just theMaxwell–Boltzmann distribution of particles in a potential well${\displaystyle V(x)=\frac{1}{2}\Vert x{\Vert }^2}$ at temperature 1. The initial distribution, being very much out of equilibrium, would diffuse towards the equilibrium distribution, making biased random steps that are a sum of pure randomness (like aBrownian walker) and gradient descent down the potential well. The randomness is necessary: if the particles were to undergo only gradient descent, then they will all fall to the origin, collapsing the distribution.

The 2020 paper proposed the Denoising Diffusion Probabilistic Model (DDPM), which improves upon the previous method byvariational inference.^[3][13]

To present the model, some notation is required.

• ${\displaystyle {\beta }_1,...,{\beta }_T\in (0,1)}$ are fixed constants.
• ${\displaystyle {\alpha }_t:=1-{\beta }_t}$
• ${\displaystyle {\overline{\alpha }}_t:={\alpha }_1\cdots {\alpha }_t}$
• ${\displaystyle {\sigma }_t:=\sqrt{1-{\overline{\alpha }}_t}}$
• ${\displaystyle {\tilde{\sigma }}_t:=\frac{{\sigma }_{t-1}}{{\sigma }_t}\sqrt{{\beta }_t}}$
• ${\displaystyle {\tilde{\mu }}_t(x_t,x_0):=\frac{\sqrt{{\alpha }_t}(1-{\overline{\alpha }}_{t-1})x_t+\sqrt{{\overline{\alpha }}_{t-1}}(1-{\alpha }_t)x_0}{{\sigma }_t^2}}$
• ${\displaystyle \mathcal{N}(\mu ,\mathrm{\Sigma })}$ is the normal distribution with mean${\displaystyle \mu }$ and variance${\displaystyle \mathrm{\Sigma }}$, and${\displaystyle \mathcal{N}(x|\mu ,\mathrm{\Sigma })}$ is the probability density at${\displaystyle x}$.
• A vertical bar denotesconditioning.

Aforward diffusion process starts at some starting point${\displaystyle x_0\sim q}$, where${\displaystyle q}$ is the probability distribution to be learned, then repeatedly adds noise to it by$${\displaystyle x_t=\sqrt{1-{\beta }_t}{x}_{t-1}+\sqrt{{\beta }_t}{z}_t}$$where${\displaystyle z_1,...,z_T}$ are IID (Independent and identically distributed random variables) samples from${\displaystyle \mathcal{N}(0,I)}$. The coefficients${\displaystyle \sqrt{1-{\beta }_t}}$ and${\displaystyle \sqrt{{\beta }_t}}$ ensure that${\displaystyle \text{Var}(X_t)=I}$ assuming that${\displaystyle \text{Var}(X_0)=I}$. The values of${\displaystyle {\beta }_t}$ are chosen such that for any starting distribution of${\displaystyle x_0}$, if it has finite second moment, then${\displaystyle \underset{t\to \mathrm{\infty }}{lim}{x}_t|x_0}$ converges^{[clarification needed]} to${\displaystyle \mathcal{N}(0,I)}$.

The entire diffusion process then satisfies$${\displaystyle q(x_{0:T})=q(x_0)q(x_1|x_0)\cdots q(x_T|x_{T-1})=q(x_0)\mathcal{N}(x_1|\sqrt{{\alpha }_1}{x}_0,{\beta }_{1}I)\cdots \mathcal{N}(x_T|\sqrt{{\alpha }_T}{x}_{T-1},{\beta }_{T}I)}$$or$${\displaystyle \ln q(x_{0:T})=\ln q(x_0)-\sum _{t=1}^T\frac{1}{2{\beta }_t}\Vert x_t-\sqrt{1-{\beta }_t}{x}_{t-1}{\Vert }^2+C}$$where${\displaystyle C}$ is a normalization constant and often omitted. In particular, we note that${\displaystyle x_{1:T}|x_0}$ is aGaussian process, which affords us considerable freedom inreparameterization. For example, by standard manipulation with Gaussian process,$${\displaystyle x_t|x_0\sim N\left(\sqrt{{\overline{\alpha }}_t}{x}_0,{\sigma }_t^{2}I\right)}$$$${\displaystyle x_{t-1}|x_t,x_0\sim \mathcal{N}({\tilde{\mu }}_t(x_t,x_0),{\tilde{\sigma }}_t^{2}I)}$$In particular, notice that for large${\displaystyle t}$, the variable${\displaystyle x_t|x_0\sim N\left(\sqrt{{\overline{\alpha }}_t}{x}_0,{\sigma }_t^{2}I\right)}$ converges to${\displaystyle \mathcal{N}(0,I)}$. That is, after a long enough diffusion process, we end up with some${\displaystyle x_T}$ that is very close to${\displaystyle \mathcal{N}(0,I)}$, with all traces of the original${\displaystyle x_0\sim q}$ gone.

For example, since$${\displaystyle x_t|x_0\sim N\left(\sqrt{{\overline{\alpha }}_t}{x}_0,{\sigma }_t^{2}I\right)}$$we can sample${\displaystyle x_t|x_0}$ directly "in one step", instead of going through all the intermediate steps${\displaystyle x_1,x_2,...,x_{t-1}}$.

The key idea of DDPM is to use a neural network parametrized by${\displaystyle \theta }$. The network takes in two arguments${\displaystyle x_t,t}$, and outputs a vector${\displaystyle {\mu }_{\theta }(x_t,t)}$ and a matrix${\displaystyle {\mathrm{\Sigma }}_{\theta }(x_t,t)}$, such that each step in the forward diffusion process can be approximately undone by${\displaystyle x_{t-1}\sim \mathcal{N}({\mu }_{\theta }(x_t,t),{\mathrm{\Sigma }}_{\theta }(x_t,t))}$. This then gives us a backward diffusion process${\displaystyle p_{\theta }}$ defined by$${\displaystyle p_{\theta }(x_T)=\mathcal{N}(x_T|0,I)}$$$${\displaystyle p_{\theta }(x_{t-1}|x_t)=\mathcal{N}(x_{t-1}|{\mu }_{\theta }(x_t,t),{\mathrm{\Sigma }}_{\theta }(x_t,t))}$$The goal now is to learn the parameters${\displaystyle \theta }$ such that${\displaystyle p_{\theta }(x_0)}$ is as close to${\displaystyle q(x_0)}$ as possible. To do that, we usemaximum likelihood estimation with variational inference.

TheELBO inequality states that${\displaystyle \ln p_{\theta }(x_0)\ge E_{x_{1:T}\sim q(\cdot |x_0)}[\ln p_{\theta }(x_{0:T})-\ln q(x_{1:T}|x_0)]}$, and taking one more expectation, we get$${\displaystyle E_{x_0\sim q}[\ln p_{\theta }(x_0)]\ge E_{x_{0:T}\sim q}[\ln p_{\theta }(x_{0:T})-\ln q(x_{1:T}|x_0)]}$$We see that maximizing the quantity on the right would give us a lower bound on the likelihood of observed data. This allows us to perform variational inference.

Define the loss function$${\displaystyle L(\theta ):=-E_{x_{0:T}\sim q}[\ln p_{\theta }(x_{0:T})-\ln q(x_{1:T}|x_0)]}$$and now the goal is to minimize the loss bystochastic gradient descent. The expression may be simplified to^[14]$${\displaystyle L(\theta )=\sum _{t=1}^T{E}_{x_{t-1},x_t\sim q}[-\ln p_{\theta }(x_{t-1}|x_t)]+E_{x_0\sim q}[D_{KL}(q(x_T|x_0)\Vert p_{\theta }(x_T))]+C}$$where${\displaystyle C}$ does not depend on the parameter, and thus can be ignored. Since${\displaystyle p_{\theta }(x_T)=\mathcal{N}(x_T|0,I)}$ also does not depend on the parameter, the term${\displaystyle E_{x_0\sim q}[D_{KL}(q(x_T|x_0)\Vert p_{\theta }(x_T))]}$ can also be ignored. This leaves just${\displaystyle L(\theta )=\sum _{t=1}^T{L}_t}$ with${\displaystyle L_t=E_{x_{t-1},x_t\sim q}[-\ln p_{\theta }(x_{t-1}|x_t)]}$ to be minimized.

Since${\displaystyle x_{t-1}|x_t,x_0\sim \mathcal{N}({\tilde{\mu }}_t(x_t,x_0),{\tilde{\sigma }}_t^{2}I)}$, this suggests that we should use${\displaystyle {\mu }_{\theta }(x_t,t)={\tilde{\mu }}_t(x_t,x_0)}$; however, the network does not have access to${\displaystyle x_0}$, and so it has to estimate it instead. Now, since${\displaystyle x_t|x_0\sim N\left(\sqrt{{\overline{\alpha }}_t}{x}_0,{\sigma }_t^{2}I\right)}$, we may write${\displaystyle x_t=\sqrt{{\overline{\alpha }}_t}{x}_0+{\sigma }_{t}z}$, where${\displaystyle z}$ is some unknown Gaussian noise. Now we see that estimating${\displaystyle x_0}$ is equivalent to estimating${\displaystyle z}$.

Therefore, let the network output a noise vector${\displaystyle {ϵ}_{\theta }(x_t,t)}$, and let it predict$${\displaystyle {\mu }_{\theta }(x_t,t)={\tilde{\mu }}_t\left(x_t,\frac{x_t-{\sigma }_t{ϵ}_{\theta }(x_t,t)}{\sqrt{{\overline{\alpha }}_t}}\right)=\frac{x_t-{ϵ}_{\theta }(x_t,t){\beta }_t/{\sigma }_t}{\sqrt{{\alpha }_t}}}$$It remains to design${\displaystyle {\mathrm{\Sigma }}_{\theta }(x_t,t)}$. The DDPM paper suggested not learning it (since it resulted in "unstable training and poorer sample quality"), but fixing it at some value${\displaystyle {\mathrm{\Sigma }}_{\theta }(x_t,t)={\zeta }_t^{2}I}$, where either${\displaystyle {\zeta }_t^2={\beta }_t\text{ or }{\tilde{\sigma }}_t^2}$ yielded similar performance.

With this, the loss simplifies to$${\displaystyle L_t=\frac{{\beta }_t^2}{2{\alpha }_t{\sigma }_t^2{\zeta }_t^2}{E}_{x_0\sim q;z\sim \mathcal{N}(0,I)}\left[{\Vert {ϵ}_{\theta }(x_t,t)-z\Vert }^2\right]+C}$$which may be minimized by stochastic gradient descent. The paper noted empirically that an even simpler loss function$${\displaystyle L_{simple,t}=E_{x_0\sim q;z\sim \mathcal{N}(0,I)}\left[{\Vert {ϵ}_{\theta }(x_t,t)-z\Vert }^2\right]}$$resulted in better models.

After a noise prediction network is trained, it can be used for generating data points in the original distribution in a loop as follows:

1. Compute the noise estimate${\displaystyle ϵ\leftarrow {ϵ}_{\theta }(x_t,t)}$
2. Compute the original data estimate${\displaystyle {\tilde{x}}_0\leftarrow (x_t-{\sigma }_tϵ)/\sqrt{{\overline{\alpha }}_t}}$
3. Sample the previous data${\displaystyle x_{t-1}\sim \mathcal{N}({\tilde{\mu }}_t(x_t,{\tilde{x}}_0),{\tilde{\sigma }}_t^{2}I)}$
4. Change time${\displaystyle t\leftarrow t-1}$

Score-based generative model is another formulation of diffusion modelling. They are also called noise conditional score network (NCSN) or score-matching with Langevin dynamics (SMLD).^{[15][16][17][18]}

Consider the problem of image generation. Let${\displaystyle x}$ represent an image, and let${\displaystyle q(x)}$ be the probability distribution over all possible images. If we have${\displaystyle q(x)}$ itself, then we can say for certain how likely a certain image is. However, this is intractable in general.

Most often, we are uninterested in knowing the absolute probability of a certain image. Instead, we are usually only interested in knowing how likely a certain image is compared to its immediate neighbors — e.g. how much more likely is an image of cat compared to some small variants of it? Is it more likely if the image contains two whiskers, or three, or with some Gaussian noise added?

Consequently, we are actually quite uninterested in${\displaystyle q(x)}$ itself, but rather,${\displaystyle {\mathrm{\nabla }}_x\ln q(x)}$. This has two major effects:

• One, we no longer need to normalize${\displaystyle q(x)}$, but can use any${\displaystyle \tilde{q}(x)=Cq(x)}$, where${\displaystyle C=\int \tilde{q}(x)dx>0}$ is any unknown constant that is of no concern to us.
• Two, we are comparing${\displaystyle q(x)}$ neighbors${\displaystyle q(x+dx)}$, by${\displaystyle \frac{q(x)}{q(x+dx)}=e^{-⟨{\mathrm{\nabla }}_x\ln q,dx⟩}}$

Let thescore function be${\displaystyle s(x):={\mathrm{\nabla }}_x\ln q(x)}$; then consider what we can do with${\displaystyle s(x)}$.

As it turns out,${\displaystyle s(x)}$ allows us to sample from${\displaystyle q(x)}$ using thermodynamics. Specifically, if we have a potential energy function${\displaystyle U(x)=-\ln q(x)}$, and a lot of particles in the potential well, then the distribution at thermodynamic equilibrium is theBoltzmann distribution${\displaystyle q_U(x)\propto e^{-U(x)/k_{B}T}=q(x{)}^{1/k_{B}T}}$. At temperature${\displaystyle k_{B}T=1}$, the Boltzmann distribution is exactly${\displaystyle q(x)}$.

Therefore, to model${\displaystyle q(x)}$, we may start with a particle sampled at any convenient distribution (such as the standard Gaussian distribution), then simulate the motion of the particle forwards according to theLangevin equation$${\displaystyle d{x}_t=-{\mathrm{\nabla }}_{x_t}U(x_t)dt+d{W}_t}$$and the Boltzmann distribution is,by Fokker-Planck equation, the unique thermodynamic equilibrium. So no matter what distribution${\displaystyle x_0}$ has, the distribution of${\displaystyle x_t}$ converges in distribution to${\displaystyle q}$ as${\displaystyle t\to \mathrm{\infty }}$.

Given a density${\displaystyle q}$, we wish to learn a score function approximation${\displaystyle f_{\theta }\approx \mathrm{\nabla }\ln q}$. This isscore matching.^[19] Typically, score matching is formalized as minimizingFisher divergence function${\displaystyle E_q[\Vert f_{\theta }(x)-\mathrm{\nabla }\ln q(x){\Vert }^2]}$. By expanding the integral, and performing an integration by parts,$${\displaystyle E_q[\Vert f_{\theta }(x)-\mathrm{\nabla }\ln q(x){\Vert }^2]=E_q[\Vert f_{\theta }{\Vert }^2+2\mathrm{\nabla }\cdot f_{\theta }]+C}$$giving us a loss function, also known as theHyvärinen scoring rule, that can be minimized by stochastic gradient descent.

Suppose we need to model the distribution of images, and we want${\displaystyle x_0\sim \mathcal{N}(0,I)}$, a white-noise image. Now, most white-noise images do not look like real images, so${\displaystyle q(x_0)\approx 0}$ for large swaths of${\displaystyle x_0\sim \mathcal{N}(0,I)}$. This presents a problem for learning the score function, because if there are no samples around a certain point, then we can't learn the score function at that point. If we do not know the score function${\displaystyle {\mathrm{\nabla }}_{x_t}\ln q(x_t)}$ at that point, then we cannot impose the time-evolution equation on a particle:$${\displaystyle d{x}_t={\mathrm{\nabla }}_{x_t}\ln q(x_t)dt+d{W}_t}$$To deal with this problem, we performannealing. If${\displaystyle q}$ is too different from a white-noise distribution, then progressively add noise until it is indistinguishable from one. That is, we perform a forward diffusion, then learn the score function, then use the score function to perform a backward diffusion.

Consider again the forward diffusion process, but this time in continuous time:$${\displaystyle x_t=\sqrt{1-{\beta }_t}{x}_{t-1}+\sqrt{{\beta }_t}{z}_t}$$By taking the${\displaystyle {\beta }_t\to \beta (t)dt,\sqrt{dt}{z}_t\to d{W}_t}$ limit, we obtain a continuous diffusion process, in the form of astochastic differential equation:$${\displaystyle d{x}_t=-\frac{1}{2}\beta (t)x_{t}dt+\sqrt{\beta (t)}d{W}_t}$$where${\displaystyle W_t}$ is aWiener process (multidimensional Brownian motion).

Now, the equation is exactly a special case of theoverdamped Langevin equation$${\displaystyle d{x}_t=-\frac{D}{k_{B}T}({\mathrm{\nabla }}_{x}U)dt+\sqrt{2D}d{W}_t}$$where${\displaystyle D}$ is diffusion tensor,${\displaystyle T}$ is temperature, and${\displaystyle U}$ is potential energy field. If we substitute in${\displaystyle D=\frac{1}{2}\beta (t)I,k_{B}T=1,U=\frac{1}{2}\Vert x{\Vert }^2}$, we recover the above equation. This explains why the phrase "Langevin dynamics" is sometimes used in diffusion models.

Now the above equation is for the stochastic motion of a single particle. Suppose we have a cloud of particles distributed according to${\displaystyle q}$ at time${\displaystyle t=0}$, then after a long time, the cloud of particles would settle into the stable distribution of${\displaystyle \mathcal{N}(0,I)}$. Let${\displaystyle {\rho }_t}$ be the density of the cloud of particles at time${\displaystyle t}$, then we have$${\displaystyle {\rho }_0=q;{\rho }_T\approx \mathcal{N}(0,I)}$$and the goal is to somehow reverse the process, so that we can start at the end and diffuse back to the beginning.

ByFokker-Planck equation, the density of the cloud evolves according to$${\displaystyle {\mathrm{\partial }}_t\ln {\rho }_t=\frac{1}{2}\beta (t)\left(n+(x+\mathrm{\nabla }\ln {\rho }_t)\cdot \mathrm{\nabla }\ln {\rho }_t+\mathrm{\Delta }\ln {\rho }_t\right)}$$where${\displaystyle n}$ is the dimension of space, and${\displaystyle \mathrm{\Delta }}$ is theLaplace operator. Equivalently,$${\displaystyle {\mathrm{\partial }}_t{\rho }_t=\frac{1}{2}\beta (t)(\mathrm{\nabla }\cdot (x{\rho }_t)+\mathrm{\Delta }{\rho }_t)}$$

If we have solved${\displaystyle {\rho }_t}$ for time${\displaystyle t\in [0,T]}$, then we can exactly reverse the evolution of the cloud. Suppose we start with another cloud of particles with density${\displaystyle {\nu }_0={\rho }_T}$, and let the particles in the cloud evolve according to

$${\displaystyle d{y}_t=\frac{1}{2}\beta (T-t)y_{t}dt+\beta (T-t)\underset{\text{score function }}{\underbrace{{\mathrm{\nabla }}_{y_t}\ln {\rho }_{T-t}\left(y_t\right)}}dt+\sqrt{\beta (T-t)}d{W}_t}$$

then by plugging into the Fokker-Planck equation, we find that${\displaystyle {\mathrm{\partial }}_t{\rho }_{T-t}={\mathrm{\partial }}_t{\nu }_t}$. Thus this cloud of points is the original cloud, evolving backwards.^[20]

At the continuous limit,$${\displaystyle {\overline{\alpha }}_t=(1-{\beta }_1)\cdots (1-{\beta }_t)=e^{\sum _i\ln (1-{\beta }_i)}\to e^{-{\int }_0^t\beta (t)dt}}$$and so$${\displaystyle x_t|x_0\sim N\left(e^{-\frac{1}{2}{\int }_0^t\beta (t)dt}{x}_0,\left(1-e^{-{\int }_0^t\beta (t)dt}\right)I\right)}$$In particular, we see that we can directly sample from any point in the continuous diffusion process without going through the intermediate steps, by first sampling${\displaystyle x_0\sim q,z\sim \mathcal{N}(0,I)}$, then get${\displaystyle x_t=e^{-\frac{1}{2}{\int }_0^t\beta (t)dt}{x}_0+\left(1-e^{-{\int }_0^t\beta (t)dt}\right)z}$. That is, we can quickly sample${\displaystyle x_t\sim {\rho }_t}$ for any${\displaystyle t\ge 0}$.

Now, define a certain probability distribution${\displaystyle \gamma }$ over${\displaystyle [0,\mathrm{\infty })}$, then the score-matching loss function is defined as the expected Fisher divergence:$${\displaystyle L(\theta )=E_{t\sim \gamma ,x_t\sim {\rho }_t}[\Vert f_{\theta }(x_t,t){\Vert }^2+2\mathrm{\nabla }\cdot f_{\theta }(x_t,t)]}$$After training,${\displaystyle f_{\theta }(x_t,t)\approx \mathrm{\nabla }\ln {\rho }_t}$, so we can perform the backwards diffusion process by first sampling${\displaystyle x_T\sim \mathcal{N}(0,I)}$, then integrating the SDE from${\displaystyle t=T}$ to${\displaystyle t=0}$:$${\displaystyle x_{t-dt}=x_t+\frac{1}{2}\beta (t)x_{t}dt+\beta (t)f_{\theta }(x_t,t)dt+\sqrt{\beta (t)}d{W}_t}$$This may be done by any SDE integration method, such asEuler–Maruyama method.

The name "noise conditional score network" is explained thus:

• "network", because${\displaystyle f_{\theta }}$ is implemented as a neural network.
• "score", because the output of the network is interpreted as approximating the score function${\displaystyle \mathrm{\nabla }\ln {\rho }_t}$.
• "noise conditional", because${\displaystyle {\rho }_t}$ is equal to${\displaystyle {\rho }_0}$ blurred by an added Gaussian noise that increases with time, and so the score function depends on the amount of noise added.

DDPM and score-based generative models are equivalent.^[16][1][21] This means that a network trained using DDPM can be used as a NCSN, and vice versa.

We know that${\displaystyle x_t|x_0\sim N\left(\sqrt{{\overline{\alpha }}_t}{x}_0,{\sigma }_t^{2}I\right)}$, so byTweedie's formula, we have$${\displaystyle {\mathrm{\nabla }}_{x_t}\ln q(x_t)=\frac{1}{{\sigma }_t^2}(-x_t+\sqrt{{\overline{\alpha }}_t}{E}_q[x_0|x_t])}$$As described previously, the DDPM loss function is${\displaystyle \sum _t{L}_{simple,t}}$ with$${\displaystyle L_{simple,t}=E_{x_0\sim q;z\sim \mathcal{N}(0,I)}\left[{\Vert {ϵ}_{\theta }(x_t,t)-z\Vert }^2\right]}$$where${\displaystyle x_t=\sqrt{{\overline{\alpha }}_t}{x}_0+{\sigma }_{t}z}$. By a change of variables,$${\displaystyle L_{simple,t}=E_{x_0,x_t\sim q}\left[{\Vert {ϵ}_{\theta }(x_t,t)-\frac{x_t-\sqrt{{\overline{\alpha }}_t}{x}_0}{{\sigma }_t}\Vert }^2\right]=E_{x_t\sim q,x_0\sim q(\cdot |x_t)}\left[{\Vert {ϵ}_{\theta }(x_t,t)-\frac{x_t-\sqrt{{\overline{\alpha }}_t}{x}_0}{{\sigma }_t}\Vert }^2\right]}$$and the term inside becomes a least squares regression, so if the network actually reaches the global minimum of loss, then we have${\displaystyle {ϵ}_{\theta }(x_t,t)=\frac{x_t-\sqrt{{\overline{\alpha }}_t}{E}_q[x_0|x_t]}{{\sigma }_t}=-{\sigma }_t{\mathrm{\nabla }}_{x_t}\ln q(x_t)}$

Thus, given a good score-based network, its predicted score is a good prediction of the noise (after scaling by${\displaystyle {\sigma }_t}$), and thus can be used for denoising.

Conversely, the continuous limit${\displaystyle x_{t-1}=x_{t-dt},{\beta }_t=\beta (t)dt,z_t\sqrt{dt}=d{W}_t}$ of the backward equation$${\displaystyle x_{t-1}=\frac{x_t}{\sqrt{{\alpha }_t}}-\frac{{\beta }_t}{{\sigma }_t\sqrt{{\alpha }_t}}{ϵ}_{\theta }(x_t,t)+\sqrt{{\beta }_t}{z}_t;z_t\sim \mathcal{N}(0,I)}$$gives us precisely the same equation as score-based diffusion:$${\displaystyle x_{t-dt}=x_t(1+\beta (t)dt/2)+\beta (t){\mathrm{\nabla }}_{x_t}\ln q(x_t)dt+\sqrt{\beta (t)}d{W}_t}$$Thus, at infinitesimal steps of DDPM, a denoising network performs score-based diffusion.

In DDPM, the sequence of numbers is called a (discrete time)noise schedule. In general, consider a strictly increasing monotonic function${\displaystyle \sigma }$ of type${\displaystyle \mathbb{R}\to (0,1)}$, such as thesigmoid function. In that case, a noise schedule is a sequence of real numbers. It then defines a sequence of noises${\displaystyle {\sigma }_t:=\sigma ({\lambda }_t)}$, which then derives the other quantities${\displaystyle {\beta }_t=1-\frac{1-{\sigma }_t^2}{1-{\sigma }_{t-1}^2}}$.

In order to use arbitrary noise schedules, instead of training a noise prediction model${\displaystyle {ϵ}_{\theta }(x_t,t)}$, one trains${\displaystyle {ϵ}_{\theta }(x_t,{\sigma }_t)}$.

Similarly, for the noise conditional score network, instead of training${\displaystyle f_{\theta }(x_t,t)}$, one trains${\displaystyle f_{\theta }(x_t,{\sigma }_t)}$.

The original DDPM method for generating images is slow, since the forward diffusion process usually takes${\displaystyle T\sim 1000}$ to make the distribution of${\displaystyle x_T}$ to appear close to Gaussian. However this means the backward diffusion process also take 1000 steps. Unlike the forward diffusion process, which can skip steps as${\displaystyle x_t|x_0}$ is Gaussian for all${\displaystyle t\ge 1}$, the backward diffusion process does not allow skipping steps. For example, to sample${\displaystyle x_{t-2}|x_{t-1}\sim \mathcal{N}({\mu }_{\theta }(x_{t-1},t-1),{\mathrm{\Sigma }}_{\theta }(x_{t-1},t-1))}$ requires the model to first sample${\displaystyle x_{t-1}}$. Attempting to directly sample${\displaystyle x_{t-2}|x_t}$ would require us to marginalize out${\displaystyle x_{t-1}}$, which is generally intractable.

DDIM^[22] is a method to take any model trained on DDPM loss, and use it to sample with some steps skipped, sacrificing an adjustable amount of quality. If we generate the Markovian chain case in DDPM to non-Markovian case, DDIM corresponds to the case that the reverse process has variance equals to 0. In other words, the reverse process (and also the forward process) is deterministic. When using fewer sampling steps, DDIM outperforms DDPM.

In detail, the DDIM sampling method is as follows. Start with the forward diffusion process${\displaystyle x_t=\sqrt{{\overline{\alpha }}_t}{x}_0+{\sigma }_tϵ}$. Then, during the backward denoising process, given${\displaystyle x_t,{ϵ}_{\theta }(x_t,t)}$, the original data is estimated as$${\displaystyle x_0^{\prime }=\frac{x_t-{\sigma }_t{ϵ}_{\theta }(x_t,t)}{\sqrt{{\overline{\alpha }}_t}}}$$then the backward diffusion process can jump to any step, and the next denoised sample is$${\displaystyle x_s=\sqrt{{\overline{\alpha }}_s}{x}_0^{\prime }+\sqrt{{\sigma }_s^2-({\sigma }_s^{\prime }{)}^2}{ϵ}_{\theta }(x_t,t)+{\sigma }_s^{\prime }ϵ}$$where${\displaystyle {\sigma }_s^{\prime }}$ is an arbitrary real number within the range${\displaystyle [0,{\sigma }_s]}$, and${\displaystyle ϵ\sim \mathcal{N}(0,I)}$ is a newly sampled Gaussian noise.^[14] If all${\displaystyle {\sigma }_s^{\prime }=0}$, then the backward process becomes deterministic, and this special case of DDIM is also called "DDIM". The original paper noted that when the process is deterministic, samples generated with only 20 steps are already very similar to ones generated with 1000 steps on the high-level.

The original paper recommended defining a single "eta value"${\displaystyle \eta \in [0,1]}$, such that${\displaystyle {\sigma }_s^{\prime }=\eta {\tilde{\sigma }}_s}$. When${\displaystyle \eta =1}$, this is the original DDPM. When${\displaystyle \eta =0}$, this is the fully deterministic DDIM. For intermediate values, the process interpolates between them.

By the equivalence, the DDIM algorithm also applies for score-based diffusion models.

Since the diffusion model is a general method for modelling probability distributions, if one wants to model a distribution over images, one can first encode the images into a lower-dimensional space by an encoder, then use a diffusion model to model the distribution over encoded images. Then to generate an image, one can sample from the diffusion model, then use a decoder to decode it into an image.^[23]

The encoder-decoder pair is most often avariational autoencoder (VAE).

^[24][who?] proposed various architectural improvements. For example, they proposed log-space interpolation during backward sampling. Instead of sampling from${\displaystyle x_{t-1}\sim \mathcal{N}({\tilde{\mu }}_t(x_t,{\tilde{x}}_0),{\tilde{\sigma }}_t^{2}I)}$, they recommended sampling from${\displaystyle \mathcal{N}({\tilde{\mu }}_t(x_t,{\tilde{x}}_0),({\sigma }_t^v{\tilde{\sigma }}_t^{1-v}{)}^{2}I)}$ for a learned parameter${\displaystyle v}$.

In thev-prediction formalism, the noising formula${\displaystyle x_t=\sqrt{{\overline{\alpha }}_t}{x}_0+\sqrt{1-{\overline{\alpha }}_t}{ϵ}_t}$ is reparameterised by an angle${\displaystyle {\varphi }_t}$ such that${\displaystyle \cos {\varphi }_t=\sqrt{{\overline{\alpha }}_t}}$ and a "velocity" defined by${\displaystyle \cos {\varphi }_t{ϵ}_t-\sin {\varphi }_t{x}_0}$. The network is trained to predict the velocity${\displaystyle {\hat{v}}_{\theta }}$, and denoising is by${\displaystyle x_{{\varphi }_t-\delta }=\cos (\delta )x_{{\varphi }_t}-\sin (\delta ){\hat{v}}_{\theta }(x_{{\varphi }_t})}$.^[25] This parameterization was found to improve performance, as the model can be trained to reach total noise (i.e.${\displaystyle {\varphi }_t={90}^{\circ }}$) and then reverse it, whereas the standard parameterization never reaches total noise since${\displaystyle \sqrt{{\overline{\alpha }}_t}>0}$ is always true.^[26]

Classifier guidance was proposed in 2021 to improve class-conditional generation by using a classifier. The original publication usedCLIP text encoders to improve text-conditional image generation.^[27]

Suppose we wish to sample not from the entire distribution of images, but conditional on the image description. We don't want to sample a generic image, but an image that fits the description "black cat with red eyes". Generally, we want to sample from the distribution${\displaystyle p(x|y)}$, where${\displaystyle x}$ ranges over images, and${\displaystyle y}$ ranges over classes of images (a description "black cat with red eyes" is just a very detailed class, and a class "cat" is just a very vague description).

Taking the perspective of thenoisy channel model, we can understand the process as follows: To generate an image${\displaystyle x}$ conditional on description${\displaystyle y}$, we imagine that the requester really had in mind an image${\displaystyle x}$, but the image is passed through a noisy channel and came out garbled, as${\displaystyle y}$. Image generation is then nothing but inferring which${\displaystyle x}$ the requester had in mind.

In other words, conditional image generation is simply "translating from a textual language into a pictorial language". Then, as in noisy-channel model, we use Bayes theorem to get$${\displaystyle p(x|y)\propto p(y|x)p(x)}$$in other words, if we have a good model of the space of all images, and a good image-to-class translator, we get a class-to-image translator "for free". In the equation for backward diffusion, the score${\displaystyle \mathrm{\nabla }\ln p(x)}$ can be replaced by$${\displaystyle {\mathrm{\nabla }}_x\ln p(x|y)=\underset{\text{score}}{\underbrace{{\mathrm{\nabla }}_x\ln p(x)}}+\underset{\text{classifier guidance}}{\underbrace{{\mathrm{\nabla }}_x\ln p(y|x)}}}$$where${\displaystyle {\mathrm{\nabla }}_x\ln p(x)}$ is the score function, trained as previously described, and${\displaystyle {\mathrm{\nabla }}_x\ln p(y|x)}$ is found by using a differentiable image classifier.

During the diffusion process, we need to condition on the time, giving$${\displaystyle {\mathrm{\nabla }}_{x_t}\ln p(x_t|y,t)={\mathrm{\nabla }}_{x_t}\ln p(y|x_t,t)+{\mathrm{\nabla }}_{x_t}\ln p(x_t|t)}$$Although, usually the classifier model does not depend on time, in which case${\displaystyle p(y|x_t,t)=p(y|x_t)}$.

Classifier guidance is defined for the gradient of score function, thus for score-based diffusion network, but as previously noted, score-based diffusion models are equivalent to denoising models by${\displaystyle {ϵ}_{\theta }(x_t,t)=-{\sigma }_t{\mathrm{\nabla }}_{x_t}\ln p(x_t|t)}$, and similarly,${\displaystyle {ϵ}_{\theta }(x_t,y,t)=-{\sigma }_t{\mathrm{\nabla }}_{x_t}\ln p(x_t|y,t)}$. Therefore, classifier guidance works for denoising diffusion as well, using the modified noise prediction:^[27]$${\displaystyle {ϵ}_{\theta }(x_t,y,t)={ϵ}_{\theta }(x_t,t)-\underset{\text{classifier guidance}}{\underbrace{{\sigma }_t{\mathrm{\nabla }}_{x_t}\ln p(y|x_t,t)}}}$$

The classifier-guided diffusion model samples from${\displaystyle p(x|y)}$, which is concentrated around themaximum a posteriori estimate${\displaystyle \arg \underset{x}{max}p(x|y)}$. If we want to force the model to move towards themaximum likelihood estimate${\displaystyle \arg \underset{x}{max}p(y|x)}$, we can use$${\displaystyle p_{\gamma }(x|y)\propto p(y|x{)}^{\gamma }p(x)}$$where${\displaystyle \gamma >0}$ is interpretable asinverse temperature. In the context of diffusion models, it is usually called theguidance scale. A high${\displaystyle \gamma }$ would force the model to sample from a distribution concentrated around${\displaystyle \arg \underset{x}{max}p(y|x)}$. This sometimes improves quality of generated images.^[27]