ABoltzmann machine (also calledSherrington–Kirkpatrick model with external field orstochastic Ising model), named afterLudwig Boltzmann, is aspin-glass model with an external field, i.e., aSherrington–Kirkpatrick model,^[1] that is a stochasticIsing model. It is astatistical physics technique applied in the context ofcognitive science.^[2] It is also classified as aMarkov random field.^[3]

Boltzmann machines are theoretically intriguing because of the locality andHebbian nature of their training algorithm (being trained by Hebb's rule), and because of theirparallelism and the resemblance of their dynamics to simplephysical processes. Boltzmann machines with unconstrained connectivity have not been proven useful for practical problems inmachine learning orinference, but if the connectivity is properly constrained, the learning can be made efficient enough to be useful for practical problems.^[4]

They are named after theBoltzmann distribution instatistical mechanics, which is used in theirsampling function. They were heavily popularized and promoted byGeoffrey Hinton,Terry Sejnowski andYann LeCun in cognitive sciences communities, particularly inmachine learning,^[2] as part of "energy-based models" (EBM), becauseHamiltonians ofspin glasses as energy are used as a starting point to define the learning task.^[5]

A Boltzmann machine, like aSherrington–Kirkpatrick model, is a network of units with a total "energy" (Hamiltonian) defined for the overall network. Its units producebinary results. Boltzmann machine weights arestochastic. The global energy${\displaystyle E}$ in a Boltzmann machine is identical in form to that ofHopfield networks andIsing models:

Where:

• ${\displaystyle w_{ij}}$ is the connection strength between unit${\displaystyle j}$ and unit${\displaystyle i}$.
• ${\displaystyle s_i}$ is the state,${\displaystyle s_i\in \{0,1\}}$, of unit${\displaystyle i}$.
• ${\displaystyle {\theta }_i}$ is the bias of unit${\displaystyle i}$ in the global energy function. (${\displaystyle -{\theta }_i}$ is the activation threshold for the unit.)

Often the weights${\displaystyle w_{ij}}$ are represented as a symmetric matrix${\displaystyle W=[w_{ij}]}$ with zeros along the diagonal.

The difference in the global energy that results from a single unit${\displaystyle i}$ equaling 0 (off) versus 1 (on), written${\displaystyle \mathrm{\Delta }{E}_i}$, assuming a symmetric matrix of weights, is given by:

This can be expressed as the difference of energies of two states:

${\displaystyle \mathrm{\Delta }{E}_i=E_{\text{i=off}}-E_{\text{i=on}}}$

Substituting the energy of each state with its relative probability according to theBoltzmann factor(the property of aBoltzmann distribution that the energy of a state is proportional to the negative log probability of that state)yields:

${\displaystyle \mathrm{\Delta }{E}_i=-k_{B}T\ln (p_{\text{i=off}})-(-k_{B}T\ln (p_{\text{i=on}})),}$

where${\displaystyle k_B}$ is theBoltzmann constant and is absorbed into the artificial notion of temperature${\displaystyle T}$.Noting that the probabilities of the unit beingon oroff sum to${\displaystyle 1}$ allows for the simplification:

${\displaystyle -\frac{\mathrm{\Delta }{E}_i}{k_{B}T}=-\ln (p_{i=\text{on}})+\ln (p_{i=\text{off}})=\ln (\frac{1-p_{i=\text{on}}}{p_{i=\text{on}}})=\ln (p_{i=\text{on}}^{-1}-1),}$

whence the probability that the${\displaystyle i}$-th unit is given by

${\displaystyle p_{i=\text{on}}=\frac{1}{1+\exp (-\frac{\mathrm{\Delta }{E}_i}{k_{B}T})},}$

where thescalar${\displaystyle T}$ is referred to as thetemperature of the system.This relation is the source of thelogistic function found in probability expressions in variants of the Boltzmann machine.

The network runs by repeatedly choosing a unit and resetting its state. After running for long enough at a certain temperature, the probability of a global state of the network depends only upon that global state's energy, according to aBoltzmann distribution, and not on the initial state from which the process was started. This means that log-probabilities of global states become linear in their energies. This relationship is true when the machine is "atthermal equilibrium", meaning that the probability distribution of global states has converged. Running the network beginning from a high temperature, its temperature gradually decreases until reaching athermal equilibrium at a lower temperature. It then may converge to a distribution where the energy level fluctuates around the global minimum. This process is calledsimulated annealing.

To train the network so that the chance it will converge to a global state according to an external distribution over these states, the weights must be set so that the global states with the highest probabilities get the lowest energies. This is done by training.

The units in the Boltzmann machine are divided into 'visible' units, V, and 'hidden' units, H. The visible units are those that receive information from the 'environment', i.e. thetraining set is a set of binary vectors over the set V. The distribution over the training set is denoted${\displaystyle P^{+}(V)}$.

The distribution over global states converges as the Boltzmann machine reachesthermal equilibrium. We denote this distribution, after wemarginalize it over the hidden units, as${\displaystyle P^{-}(V)}$.

Our goal is to approximate the "real" distribution${\displaystyle P^{+}(V)}$ using the${\displaystyle P^{-}(V)}$ produced by the machine. The similarity of the two distributions is measured by theKullback–Leibler divergence,${\displaystyle G}$:

${\displaystyle G=\sum _v{P}^{+}(v)\ln \left(\frac{P^{+}(v)}{P^{-}(v)}\right)}$

where the sum is over all the possible states of${\displaystyle V}$.${\displaystyle G}$ is a function of the weights, since they determine the energy of a state, and the energy determines${\displaystyle P^{-}(v)}$, as promised by the Boltzmann distribution. Agradient descent algorithm over${\displaystyle G}$ changes a given weight,${\displaystyle w_{ij}}$, by subtracting thepartial derivative of${\displaystyle G}$ with respect to the weight.

Boltzmann machine training involves two alternating phases. One is the "positive" phase where the visible units' states are clamped to a particular binary state vector sampled from the training set (according to${\displaystyle P^{+}}$). The other is the "negative" phase where the network is allowed to run freely, i.e. only the input nodes have their state determined by external data, but the output nodes are allowed to float. The gradient with respect to a given weight,${\displaystyle w_{ij}}$, is given by the equation:^[2]

${\displaystyle \frac{\mathrm{\partial }G}{\mathrm{\partial }{w}_{ij}}=-\frac{1}{R}[p_{ij}^{+}-p_{ij}^{-}]}$

where:

• ${\displaystyle p_{ij}^{+}}$ is the probability that unitsi andj are both on when the machine is at equilibrium on the positive phase.
• ${\displaystyle p_{ij}^{-}}$ is the probability that unitsi andj are both on when the machine is at equilibrium on the negative phase.
• ${\displaystyle R}$ denotes thelearning rate

This result follows from the fact that atthermal equilibrium the probability${\displaystyle P^{-}(s)}$ of any global state${\displaystyle s}$ when the network is free-running is given by the Boltzmann distribution.

This learning rule is biologically plausible because the only information needed to change the weights is provided by "local" information. That is, the connection (synapse, biologically) does not need information about anything other than the two neurons it connects. This is more biologically realistic than the information needed by a connection in many other neural network training algorithms, such asbackpropagation.

The training of a Boltzmann machine does not use theEM algorithm, which is heavily used inmachine learning. By minimizing theKL-divergence, it is equivalent to maximizing the log-likelihood of the data. Therefore, the training procedure performs gradient ascent on the log-likelihood of the observed data. This is in contrast to the EM algorithm, where the posterior distribution of the hidden nodes must be calculated before the maximization of the expected value of the complete data likelihood during the M-step.

Training the biases is similar, but uses only single node activity:

${\displaystyle \frac{\mathrm{\partial }G}{\mathrm{\partial }{\theta }_i}=-\frac{1}{R}[p_i^{+}-p_i^{-}]}$

Theoretically the Boltzmann machine is a rather general computational medium. For instance, if trained on photographs, the machine would theoretically model the distribution of photographs, and could use that model to, for example,complete a partial photograph.

Unfortunately, Boltzmann machines experience a serious practical problem, namely that it seems to stop learning correctly when the machine is scaled up to anything larger than a trivial size.^{[citation needed]} This is due to important effects, specifically:

• the required time order to collect equilibrium statistics grows exponentially with the machine's size, and with the magnitude of the connection strengths^{[citation needed]}
• connection strengths are more plastic when the connected units have activation probabilities intermediate between zero and one, leading to a so-called variance trap. The net effect is that noise causes the connection strengths to follow arandom walk until the activities saturate.

Although learning is impractical in general Boltzmann machines, it can be made quite efficient in a restricted Boltzmann machine (RBM) which does not allow intralayer connections between hidden units and visible units, i.e. there is no connection between visible to visible and hidden to hidden units. After training one RBM, the activities of its hidden units can be treated as data for training a higher-level RBM. This method of stacking RBMs makes it possible to train many layers of hidden units efficiently and is one of the most commondeep learning strategies. As each new layer is added the generative model improves.

An extension to the restricted Boltzmann machine allows using real valued data rather than binary data.^[6]

One example of a practical RBM application is in speech recognition.^[7]

A deep Boltzmann machine (DBM) is a type of binary pairwiseMarkov random field (undirected probabilisticgraphical model) with multiple layers ofhiddenrandom variables. It is a network of symmetrically coupled stochasticbinary units. It comprises a set of visible units${\displaystyle \mathit{\nu }\in \{0,1{\}}^D}$ and layers of hidden units${\displaystyle {\mathit{h}}^{(1)}\in \{0,1{\}}^{F_1},{\mathit{h}}^{(2)}\in \{0,1{\}}^{F_2},\dots ,{\mathit{h}}^{(L)}\in \{0,1{\}}^{F_L}}$. No connection links units of the same layer (likeRBM). For theDBM, the probability assigned to vectorν is

${\displaystyle p(\mathit{\nu })=\frac{1}{Z}\sum _h{e}^{\sum _{ij}{W}_{ij}^{(1)}{\nu }_i{h}_j^{(1)}+\sum _{jl}{W}_{jl}^{(2)}{h}_j^{(1)}{h}_l^{(2)}+\sum _{lm}{W}_{lm}^{(3)}{h}_l^{(2)}{h}_m^{(3)}},}$

where${\displaystyle \mathit{h}=\{{\mathit{h}}^{(1)},{\mathit{h}}^{(2)},{\mathit{h}}^{(3)}\}}$ are the set of hidden units, and${\displaystyle \theta =\{{\mathit{W}}^{(1)},{\mathit{W}}^{(2)},{\mathit{W}}^{(3)}\}}$ are the model parameters, representing visible-hidden and hidden-hidden interactions.^[8] In aDBN only the top two layers form a restricted Boltzmann machine (which is an undirectedgraphical model), while lower layers form a directed generative model. In a DBM all layers are symmetric and undirected.

LikeDBNs, DBMs can learn complex and abstract internal representations of the input in tasks such asobject orspeech recognition, using limited, labeled data to fine-tune the representations built using a large set of unlabeled sensory input data. However, unlike DBNs and deepconvolutional neural networks, they pursue the inference and training procedure in both directions, bottom-up and top-down, which allow the DBM to better unveil the representations of the input structures.^[9][10][11]

However, the slow speed of DBMs limits their performance and functionality. Because exact maximum likelihood learning is intractable for DBMs, only approximate maximum likelihood learning is possible. Another option is to use mean-field inference to estimate data-dependent expectations and approximate the expected sufficient statistics by usingMarkov chain Monte Carlo (MCMC).^[8] This approximate inference, which must be done for each test input, is about 25 to 50 times slower than a single bottom-up pass in DBMs. This makes joint optimization impractical for large data sets, and restricts the use of DBMs for tasks such as feature representation.

The need for deep learning withreal-valued inputs, as inGaussian RBMs, led to the spike-and-slabRBM (ssRBM), which models continuous-valued inputs withbinarylatent variables.^[12] Similar to basicRBMs and its variants, a spike-and-slab RBM is abipartite graph, while like GRBMs, the visible units (input) are real-valued. The difference is in the hidden layer, where each hidden unit has a binary spike variable and a real-valued slab variable. A spike is a discreteprobability mass at zero, while a slab is adensity over continuous domain;^[13] their mixture forms aprior.^[14]

An extension of ssRBM called μ-ssRBM provides extra modeling capacity using additional terms in theenergy function. One of these terms enables the model to form aconditional distribution of the spike variables bymarginalizing out the slab variables given an observation.

In more general mathematical setting, the Boltzmann distribution is also known as theGibbs measure. Instatistics andmachine learning it is called alog-linear model. Indeep learning the Boltzmann distribution is used in the sampling distribution ofstochastic neural networks such as the Boltzmann machine.

The Boltzmann machine is based on the Sherrington–Kirkpatrickspin glass model byDavid Sherrington andScott Kirkpatrick.^[15] The seminal publication byJohn Hopfield (1982) applied methods of statistical mechanics, mainly the recently developed (1970s) theory of spin glasses, to studyassociative memory (later named the "Hopfield network").^[16]

The original contribution in applying such energy-based models in cognitive science appeared in papers byGeoffrey Hinton andTerry Sejnowski.^[17][18][19] In a 1995 interview, Hinton stated that in 1983 February or March, he was going to give a talk onsimulated annealing in Hopfield networks, so he had to design a learning algorithm for the talk, resulting in the Boltzmann machine learning algorithm.^[20]

The idea of applying the Ising model with annealedGibbs sampling was used inDouglas Hofstadter'sCopycat project (1984).^[21][22]

The explicit analogy drawn with statistical mechanics in the Boltzmann machine formulation led to the use of terminology borrowed from physics (e.g., "energy"), which became standard in the field. The widespread adoption of this terminology may have been encouraged by the fact that its use led to the adoption of a variety of concepts and methods from statistical mechanics. The various proposals to use simulated annealing for inference were apparently independent.

Similar ideas (with a change of sign in the energy function) are found inPaul Smolensky's "Harmony Theory".^[23] Ising models can be generalized toMarkov random fields, which find widespread application inlinguistics,robotics,computer vision andartificial intelligence.

In 2024, Hopfield and Hinton were awardedNobel Prize in Physics for their foundational contributions tomachine learning, such as the Boltzmann machine.^[24]

• Restricted Boltzmann machine
• Helmholtz machine
• Markov random field (MRF)
• Ising model (Lenz–Ising model)
• Hopfield network

• Hinton, G. E.;Sejnowski, T. J. (1986). D. E. Rumelhart; J. L. McClelland (eds.)."Learning and Relearning in Boltzmann Machines"(PDF).Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1: Foundations:282–317. Archived fromthe original(PDF) on 2010-07-05.
• Hinton, G. E. (2002)."Training Products of Experts by Minimizing Contrastive Divergence"(PDF).Neural Computation.14 (8):1771–1800.CiteSeerX 10.1.1.35.8613.doi:10.1162/089976602760128018.PMID 12180402.S2CID 207596505.
• Hinton, G. E.; Osindero, S.; Teh, Y. (2006)."A fast learning algorithm for deep belief nets"(PDF).Neural Computation.18 (7):1527–1554.CiteSeerX 10.1.1.76.1541.doi:10.1162/neco.2006.18.7.1527.PMID 16764513.S2CID 2309950.
• Kothari P (2020): https://www.forbes.com/sites/tomtaulli/2020/02/02/coronavirus-can-ai-artificial-intelligence-make-a-difference/?sh=1eca51e55817
• Montufar, Guido (2018)."Restricted Boltzmann Machines: Introduction and Review"(PDF).MPI MiS (Preprint). Retrieved1 August 2023.

• Scholarpedia article by Hinton about Boltzmann machines
• Talk at Google by Geoffrey Hinton

• Neural network architectures
• Ludwig Boltzmann
• Mathematical physics

• All articles with dead external links
• Articles with dead external links from June 2024
• Articles with permanently dead external links
• Articles with short description
• Short description is different from Wikidata
• All articles with unsourced statements
• Articles with unsourced statements from January 2013
• Articles with unsourced statements from August 2015