Aresidual neural network (also referred to as aresidual network orResNet)^[1] is adeep learning architecture in which the layers learn residual functions with reference to the layer inputs. It was developed in 2015 forimage recognition, and won theImageNet Large Scale Visual Recognition Challenge (ILSVRC) of that year.^[2][3]

As a point of terminology, "residual connection" refers to the specific architectural motif of${\displaystyle x\mapsto f(x)+x}$, where${\displaystyle f}$ is an arbitrary neural network module. The motif had been used previously (see§History for details). However, the publication of ResNet made it widely popular forfeedforward networks, appearing in neural networks that are seemingly unrelated to ResNet.

The residual connection stabilizes the training and convergence of deep neural networks with hundreds of layers, and is a common motif in deep neural networks, such astransformer models (e.g.,BERT, andGPT models such asChatGPT), theAlphaGo Zero system, theAlphaStar system, and theAlphaFold system.

In a multilayer neural network model, consider a subnetwork with a certain number of stacked layers (e.g., 2 or 3). Denote the underlying function performed by this subnetwork as${\displaystyle H(x)}$, where${\displaystyle x}$ is the input to the subnetwork. Residual learning re-parameterizes this subnetwork and lets the parameter layers represent a "residual function"${\displaystyle F(x)=H(x)-x}$. The output${\displaystyle y}$ of this subnetwork is then represented as:

$${\displaystyle y=F(x)+x}$$

The operation of "${\displaystyle +x}$" is implemented via a "skip connection" that performs an identity mapping to connect the input of the subnetwork with its output. This connection is referred to as a "residual connection" in later work. The function${\displaystyle F(x)}$ is often represented by matrix multiplication interlaced withactivation functions and normalization operations (e.g.,batch normalization orlayer normalization). As a whole, one of these subnetworks is referred to as a "residual block".^[1] A deep residual network is constructed by simply stacking these blocks.

Long short-term memory (LSTM) has a memory mechanism that serves as a residual connection.^[4] In an LSTM without a forgetgate, an input${\displaystyle x_t}$ is processed by a function${\displaystyle F}$ and added to a memory cell${\displaystyle c_t}$, resulting in${\displaystyle c_{t+1}=c_t+F(x_t)}$. An LSTM with a forget gate essentially functions as ahighway network.

To stabilize thevariance of the layers' inputs, it is recommended to replace the residual connections${\displaystyle x+f(x)}$ with${\displaystyle x/L+f(x)}$, where${\displaystyle L}$ is the total number of residual layers.^[5]

If the function${\displaystyle F}$ is of type${\displaystyle F:{\mathbb{R}}^n\to {\mathbb{R}}^m}$ where${\displaystyle n\ne m}$, then${\displaystyle F(x)+x}$ is undefined. To handle this special case, a projection connection is used:

$${\displaystyle y=F(x)+P(x)}$$

where${\displaystyle P}$ is typically a linear projection, defined by${\displaystyle P(x)=Mx}$ where${\displaystyle M}$ is a${\displaystyle m\times n}$ matrix. The matrix is trained viabackpropagation, as is any other parameter of the model.

The introduction of identity mappings facilitates signal propagation in both forward and backward paths.^[6]

If the output of the${\displaystyle \ell }$-th residual block is the input to the${\displaystyle (\ell +1)}$-th residual block (assuming no activation function between blocks), then the${\displaystyle (\ell +1)}$-th input is:

$${\displaystyle x_{\ell +1}=F(x_{\ell })+x_{\ell }}$$

Applying this formulation recursively, e.g.:

yields the general relationship:

$${\displaystyle x_L=x_{\ell }+\sum _{i=\ell }^{L-1}F(x_i)}$$

where$L$ is the index of a residual block and$\ell$ is the index of some earlier block. This formulation suggests that there is always a signal that is directly sent from a shallower block$\ell$ to a deeper block$L$.

The residual learning formulation provides the added benefit of mitigating thevanishing gradient problem to some extent. However, it is crucial to acknowledge that the vanishing gradient issue is not the root cause of the degradation problem, which is tackled through the use of normalization. To observe the effect of residual blocks on backpropagation, consider the partial derivative of aloss function${\displaystyle \mathcal{E}}$ with respect to some residual block input${\displaystyle x_{\ell }}$. Using the equation above from forward propagation for a later residual block${\displaystyle L>\ell }$:^[6]

This formulation suggests that the gradient computation of a shallower layer,$\frac{\mathrm{\partial }\mathcal{E}}{\mathrm{\partial }{x}_{\ell }}$, always has a later term$\frac{\mathrm{\partial }\mathcal{E}}{\mathrm{\partial }{x}_L}$ that is directly added. Even if the gradients of the${\displaystyle F(x_i)}$ terms are small, the total gradient$\frac{\mathrm{\partial }\mathcal{E}}{\mathrm{\partial }{x}_{\ell }}$ resists vanishing due to the added term$\frac{\mathrm{\partial }\mathcal{E}}{\mathrm{\partial }{x}_L}$.

Abasic block is the simplest building block studied in the original ResNet.^[1] This block consists of two sequential 3x3convolutional layers and a residual connection. The input and output dimensions of both layers are equal.

Abottleneck block^[1] consists of three sequential convolutional layers and a residual connection. The first layer in this block is a 1×1 convolution for dimension reduction (e.g., to 1/2 of the input dimension); the second layer performs a 3×3 convolution; the last layer is another 1×1 convolution for dimension restoration. The models of ResNet-50, ResNet-101, and ResNet-152 are all based on bottleneck blocks.^[1]

Thepre-activation residual block^[6] applies activation functions before applying the residual function${\displaystyle F}$. Formally, the computation of a pre-activation residual block can be written as:

$${\displaystyle x_{\ell +1}=F(\varphi (x_{\ell }))+x_{\ell }}$$

where${\displaystyle \varphi }$ can be any activation (e.g.ReLU) or normalization (e.g.LayerNorm) operation. This design reduces the number of non-identity mappings between residual blocks, and allows an identity mapping directly from the input to the output. This design was used to train models with 200 to over 1000 layers, and was found to consistently outperform variants where the residual path is not an identity function. The pre-activation ResNet with 200 layers took 3 weeks to train forImageNet on 8GPUs in 2016.^[6]

SinceGPT-2,transformer blocks have been mostly implemented as pre-activation blocks. This is often referred to as "pre-normalization" in the literature of transformer models.^[7]

Originally, ResNet was designed forcomputer vision.^[1][8][9]

All transformer architectures include residual connections. Indeed, very deep transformers cannot be trained without them.^[10]

The original ResNet paper made no claim on being inspired by biological systems. However, later research has related ResNet to biologically-plausible algorithms.^[11][12]

A study published inScience in 2023^[13] disclosed the completeconnectome of an insect brain (specifically that of a fruit fly larva). This study discovered "multilayer shortcuts" that resemble the skip connections in artificial neural networks, including ResNets.

Residual connections were noticed inneuroanatomy, such asLorente de No (1938).^{[14]: Fig 3}McCulloch andPitts (1943) proposed artificial neural networks and considered those with residual connections.^{[15]: Fig 1.h}

In 1961,Frank Rosenblatt described a three-layermultilayer perceptron (MLP) model with skip connections.^{[16]: 313, Chapter 15} The model was referred to as a "cross-coupled system", and the skip connections were forms of cross-coupled connections.

During the late 1980s, "skip-layer" connections were sometimes used in neural networks. Examples include:^[17][18] Lang and Witbrock (1988)^[19] trained a fully connected feedforward network where each layer skip-connects to all subsequent layers, like the later DenseNet (2016). In this work, the residual connection was the form${\displaystyle x\mapsto F(x)+P(x)}$, where${\displaystyle P}$ is a randomly-initialized projection connection. They termed it a "short-cut connection". An early neural language model used residual connections and named them "direct connections".^[20]

Sepp Hochreiter discovered thevanishing gradient problem in 1991^[21] and argued that it explained why the then-prevalent forms ofrecurrent neural networks did not work for long sequences. He andSchmidhuber later designed the LSTM architecture to solve this problem,^[4][22] which has a "cell state"${\displaystyle c_t}$ that can function as a generalized residual connection. Thehighway network (2015)^[23][24] applied the idea of an LSTMunfolded in time tofeedforward neural networks, resulting in the highway network. ResNet is equivalent to an open-gated highway network.

During the early days of deep learning, there were attempts to train increasingly deep models. Notable examples included theAlexNet (2012), which had 8 layers, and theVGG-19 (2014), which had 19 layers.^[25] However, stacking too many layers led to a steep reduction intraining accuracy,^[26] known as the "degradation" problem.^[1] In theory, adding additional layers to deepen a network should not result in a higher trainingloss, but this is what happened withVGGNet.^[1] If the extra layers can be set asidentity mappings, however, then the deeper network would represent the same function as its shallower counterpart. There is some evidence that the optimizer is not able to approach identity mappings for the parameterized layers, and the benefit of residual connections was to allow identity mappings by default.^[6]

In 2014, the state of the art was training deep neural networks with 20 to 30 layers.^[25] The research team for ResNet attempted to train deeper ones by empirically testing various methods for training deeper networks, until they came upon the ResNet architecture.^[27]

Wide Residual Network (2016) found that using more channels and fewer layers than the original ResNet improves performance and GPU-computational efficiency, and that a block with two 3×3 convolutions is superior to other configurations of convolution blocks.^[28]

DenseNet (2016)^[29] connects the output of each layer to the input to each subsequent layer:

$${\displaystyle x_{\ell +1}=F(x_1,x_2,\dots ,x_{\ell -1},x_{\ell })}$$

Stochastic depth^[30] is aregularization method that randomly drops a subset of layers and lets the signal propagate through the identity skip connections. Also known asDropPath, this regularizes training for deep models, such asvision transformers.^[31]

ResNeXt (2017) combines theInception module with ResNet.^[32][8]

Squeeze-and-Excitation Networks (2018) added squeeze-and-excitation (SE) modules to ResNet.^[33] An SE module is applied after a convolution, and takes a tensor of shape${\displaystyle {\mathbb{R}}^{H\times W\times C}}$ (height, width, channels) as input. Each channel is averaged, resulting in a vector of shape${\displaystyle {\mathbb{R}}^C}$. This is then passed through amultilayer perceptron (with an architecture such aslinear-ReLU-linear-sigmoid) before it is multiplied with the original tensor. It won theILSVRC in 2017.^[34]

• Neural network architectures
• Deep learning

• Articles with short description
• Short description matches Wikidata