Inneural networks, thegating mechanism is an architectural motif for controlling the flow ofactivation andgradient signals. They are most prominently used inrecurrent neural networks (RNNs), but have also found applications in other architectures.

Gating mechanisms are the centerpiece oflong short-term memory (LSTM).^[1] They were proposed to mitigate thevanishing gradient problem often encountered by regular RNNs.

An LSTM unit contains three gates:

• Aninput gate, which controls the flow of new information into the memory cell
• Aforget gate, which controls how much information is retained from the previous time step
• Anoutput gate, which controls how much information is passed to the next layer.

The equations for LSTM are:^[2]

Here,${\displaystyle \odot }$ representselementwise multiplication.

• LSTM architecture, with gates
• 
• 
• 
• 

Thegated recurrent unit (GRU) simplifies the LSTM.^[3] Compared to the LSTM, the GRU has just two gates: areset gate and anupdate gate. GRU also merges the cell state and hidden state. The reset gate roughly corresponds to the forget gate, and the update gate roughly corresponds to the input gate. The output gate is removed.

There are several variants of GRU. One particular variant has these equations:^[4]

• Gated Recurrent Unit architecture, with gates
• 
• 
• 

Gated Linear Units (GLUs)^[5] adapt the gating mechanism for use infeedforward neural networks, often withintransformer-based architectures. They are defined as:

$${\displaystyle \mathrm{G}\mathrm{L}\mathrm{U}(a,b)=a\odot \sigma (b)}$$

where${\displaystyle a,b}$ are the first and second inputs, respectively.${\displaystyle \sigma }$ represents thesigmoidactivation function.

Replacing${\displaystyle \sigma }$ with other activation functions leads to variants of GLU:

whereReLU,GELU, andSwish are different activation functions.

In transformer models, such gating units are often used in thefeedforward modules. For a single vector input, this results in:^[6]

Gating mechanism is used inhighway networks, which were designed by unrolling an LSTM.

Channel gating^[7] uses a gate to control the flow of information through different channels inside aconvolutional neural network (CNN).

• Recurrent neural network
• Long short-term memory
• Gated recurrent unit
• Transformer
• Activation function

• Zhang, Aston; Lipton, Zachary; Li, Mu; Smola, Alexander J. (2024)."10.1. Long Short-Term Memory (LSTM)".Dive into deep learning. Cambridge New York Port Melbourne New Delhi Singapore: Cambridge University Press.ISBN 978-1-009-38943-3.

• Neural network architectures
• Deep learning

• Articles with short description
• Short description is different from Wikidata