Arestricted Boltzmann machine (RBM) (also called arestricted Sherrington–Kirkpatrick model with external field orrestricted stochastic Ising–Lenz–Little model) is agenerativestochasticartificial neural network that can learn aprobability distribution over its set of inputs.^[1]

RBMs were initially proposed under the nameHarmonium byPaul Smolensky in 1986,^[2] and rose to prominence afterGeoffrey Hinton and collaborators used fast learning algorithms for them in the mid-2000s. RBMs have found applications indimensionality reduction,^[3]classification,^[4]collaborative filtering,^[5]feature learning,^[6]topic modelling,^[7]immunology,^[8] and evenmany‑body quantum mechanics.^[9][10][11]

They can be trained in eithersupervised orunsupervised ways, depending on the task.^{[citation needed]}

As their name implies, RBMs are a variant ofBoltzmann machines, with the restriction that theirneurons must form abipartite graph:

• a pair of nodes from each of the two groups of units (commonly referred to as the "visible" and "hidden" units respectively) may have a symmetric connection between them; and
• there are no connections between nodes within a group.

By contrast, "unrestricted" Boltzmann machines may have connections betweenhidden units. This restriction allows for more efficient trainingalgorithms than are available for the general class of Boltzmann machines, in particular thegradient-basedcontrastive divergence algorithm.^[12]

Restricted Boltzmann machines can also be used indeep learning networks. In particular,deep belief networks can be formed by "stacking" RBMs and optionally fine-tuning the resulting deep network withgradient descent andbackpropagation.^[13]

The standard type of RBM has binary-valued (Boolean) hidden and visible units, and consists of amatrix of weights${\displaystyle W}$ of size${\displaystyle m\times n}$. Each weight element${\displaystyle (w_{i,j})}$ of the matrix is associated with the connection between the visible (input) unit${\displaystyle v_i}$ and the hidden unit${\displaystyle h_j}$. In addition, there are bias weights (offsets)${\displaystyle a_i}$ for${\displaystyle v_i}$ and${\displaystyle b_j}$ for${\displaystyle h_j}$. Given the weights and biases, theenergy of a configuration (pair of Boolean vectors)(v,h) is defined as

${\displaystyle E(v,h)=-\sum _i{a}_i{v}_i-\sum _j{b}_j{h}_j-\sum _i\sum _j{v}_i{w}_{i,j}{h}_j}$

or, in matrix notation,

${\displaystyle E(v,h)=-a^{\mathrm{T}}v-b^{\mathrm{T}}h-v^{\mathrm{T}}Wh.}$

This energy function is analogous to that of aHopfield network. As with general Boltzmann machines, thejoint probability distribution for the visible and hidden vectors is defined in terms of the energy function as follows,^[14]

${\displaystyle P(v,h)=\frac{1}{Z}{e}^{-E(v,h)}}$

where${\displaystyle Z}$ is apartition function defined as the sum of${\displaystyle e^{-E(v,h)}}$ over all possible configurations, which can be interpreted as anormalizing constant to ensure that the probabilities sum to 1. Themarginal probability of a visible vector is the sum of${\displaystyle P(v,h)}$ over all possible hidden layer configurations,^[14]

${\displaystyle P(v)=\frac{1}{Z}\sum _{\{h\}}{e}^{-E(v,h)}}$,

and vice versa. Since the underlying graph structure of the RBM isbipartite (meaning there are no intra-layer connections), the hidden unit activations aremutually independent given the visible unit activations. Conversely, the visible unit activations are mutually independent given the hidden unit activations.^[12] That is, form visible units andn hidden units, theconditional probability of a configuration of the visible unitsv, given a configuration of the hidden unitsh, is

${\displaystyle P(v|h)=\prod _{i=1}^{m}P(v_i|h)}$.

Conversely, the conditional probability ofh givenv is

${\displaystyle P(h|v)=\prod _{j=1}^{n}P(h_j|v)}$.

The individual activation probabilities are given by

${\displaystyle P(h_j=1|v)=\sigma \left(b_j+\sum _{i=1}^m{w}_{i,j}{v}_i\right)}$ and${\displaystyle P(v_i=1|h)=\sigma \left(a_i+\sum _{j=1}^n{w}_{i,j}{h}_j\right)}$

where${\displaystyle \sigma }$ denotes thelogistic sigmoid.

The visible units of Restricted Boltzmann Machine can bemultinomial, although the hidden units areBernoulli.^{[clarification needed]} In this case, the logistic function for visible units is replaced by thesoftmax function

${\displaystyle P(v_i^k=1|h)=\frac{\exp (a_i^k+{\mathrm{\Sigma }}_j{W}_{ij}^k{h}_j)}{{\mathrm{\Sigma }}_{k^{\prime }=1}^K\exp (a_i^{k^{\prime }}+{\mathrm{\Sigma }}_j{W}_{ij}^{k^{\prime }}{h}_j)}}$

whereK is the number of discrete values that the visible values have. They are applied in topic modeling,^[7] andrecommender systems.^[5]

Restricted Boltzmann machines are a special case ofBoltzmann machines andMarkov random fields.^[15][16]

Thegraphical model of RBMs corresponds to that offactor analysis.^[17]

Restricted Boltzmann machines are trained to maximize the product of probabilities assigned to some training set${\displaystyle V}$ (a matrix, each row of which is treated as a visible vector${\displaystyle v}$),

${\displaystyle \arg \underset{W}{max}\prod _{v\in V}P(v)}$

or equivalently, to maximize theexpectedlog probability of a training sample${\displaystyle v}$ selected randomly from${\displaystyle V}$:^[15][16]

${\displaystyle \arg \underset{W}{max}\mathbb{E}\left[\log P(v)\right]}$

The algorithm most often used to train RBMs, that is, to optimize the weight matrix${\displaystyle W}$, is the contrastive divergence (CD) algorithm due toHinton, originally developed to train PoE (product of experts) models.^[18][19]The algorithm performsGibbs sampling and is used inside agradient descent procedure (similar to the way backpropagation is used inside such a procedure when training feedforward neural nets) to compute weight update.

The basic, single-step contrastive divergence (CD-1) procedure for a single sample can be summarized as follows:

1. Take a training samplev, compute the probabilities of the hidden units and sample a hidden activation vectorh from this probability distribution.
2. Compute theouter product ofv andh and call this thepositive gradient.
3. Fromh, sample a reconstructionv' of the visible units, then resample the hidden activationsh' from this. (Gibbs sampling step)
4. Compute theouter product ofv' andh' and call this thenegative gradient.
5. Let the update to the weight matrix${\displaystyle W}$ be the positive gradient minus the negative gradient, times some learning rate:${\displaystyle \mathrm{\Delta }W=ϵ(v{h}^{\mathsf{T}}-v^{\prime }{h}^{\prime \mathsf{T}})}$.
6. Update the biasesa andb analogously:${\displaystyle \mathrm{\Delta }a=ϵ(v-v^{\prime })}$,${\displaystyle \mathrm{\Delta }b=ϵ(h-h^{\prime })}$.

A Practical Guide to Training RBMs written by Hinton can be found on his homepage.^[14]

This sectionmay be too technical for most readers to understand. Pleasehelp improve it tomake it understandable to non-experts, without removing the technical details.(August 2023) (Learn how and when to remove this message)

This sectionneeds additional citations forverification. Please helpimprove this article byadding citations to reliable sources in this section. Unsourced material may be challenged and removed.(August 2023) (Learn how and when to remove this message)

• The difference between the Stacked Restricted Boltzmann Machines and RBM is that RBM has lateral connections within a layer that are prohibited to make analysis tractable. On the other hand, the Stacked Boltzmann consists of a combination of an unsupervised three-layer network with symmetric weights and a supervised fine-tuned top layer for recognizing three classes.
• The usage of Stacked Boltzmann is tounderstand Natural languages,retrieve documents, image generation, and classification. These functions are trained with unsupervised pre-training and/or supervised fine-tuning. Unlike the undirected symmetric top layer, with a two-way unsymmetric layer for connection for RBM. The restricted Boltzmann's connection is three-layers with asymmetric weights, and two networks are combined into one.
• Stacked Boltzmann does share similarities with RBM, the neuron for Stacked Boltzmann is a stochastic binary Hopfield neuron, which is the same as the Restricted Boltzmann Machine. The energy from both Restricted Boltzmann and RBM is given by Gibb's probability measure:${\displaystyle E=-\frac{1}{2}\sum _{i,j}{w}_{ij}{s}_i{s}_j+\sum _i{\theta }_i{s}_i}$. The training process of Restricted Boltzmann is similar to RBM. Restricted Boltzmann train one layer at a time and approximate equilibrium state with a 3-segment pass, not performing back propagation. Restricted Boltzmann uses both supervised and unsupervised on different RBM for pre-training for classification and recognition. The training uses contrastive divergence with Gibbs sampling: Δw_ij = e*(p_ij - p'_ij)
• The restricted Boltzmann's strength is it performs a non-linear transformation so it's easy to expand, and can give a hierarchical layer of features. The Weakness is that it has complicated calculations of integer and real-valued neurons. It does not follow the gradient of any function, so the approximation of Contrastive divergence to maximum likelihood is improvised.^[14]

• Fischer, Asja; Igel, Christian (2012), "An Introduction to Restricted Boltzmann Machines",Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, Lecture Notes in Computer Science, vol. 7441, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 14–36,doi:10.1007/978-3-642-33275-3_2,ISBN 978-3-642-33274-6

• Autoencoder
• Helmholtz machine

• Chen, Edwin (2011-07-18)."Introduction to Restricted Boltzmann Machines".Edwin Chen's blog.
• Nicholson, Chris; Gibson, Adam."A Beginner's Tutorial for Restricted Boltzmann Machines".Deeplearning4j Documentation. Archived from the original on 2017-02-11. Retrieved2018-11-15.{{cite web}}: CS1 maint: bot: original URL status unknown (link)
• Nicholson, Chris; Gibson, Adam."Understanding RBMs".Deeplearning4j Documentation. Archived fromthe original on 2016-09-20. Retrieved2014-12-29.

• Pythonimplementation of Bernoulli RBM andtutorial
• SimpleRBM is a very small RBM code (24kB) useful for you to learn about how RBMs learn and work.
• Julia implementation of Restricted Boltzmann machines:https://github.com/cossio/RestrictedBoltzmannMachines.jl

• Neural network architectures
• Stochastic models
• Supervised learning
• Unsupervised learning

• Webarchive template wayback links
• Articles with short description
• Short description is different from Wikidata
• All articles with unsourced statements
• Articles with unsourced statements from February 2024
• Wikipedia articles needing clarification from April 2021
• Wikipedia articles that are too technical from August 2023
• All articles that are too technical
• Articles needing additional references from August 2023
• All articles needing additional references
• CS1 maint: bot: original URL status unknown