Inartificial neural networks, thegated recurrent unit (GRU) is agating mechanism used inrecurrent neural networks, introduced in 2014 byKyunghyun Cho et al.^[1] The GRU is like along short-term memory (LSTM) with a gating mechanism to input or forget certain features,^[2] but lacks a context vector or output gate, resulting in fewer parameters than LSTM.^[3] GRU's performance on certain tasks of polyphonic music modeling, speech signal modeling and natural language processing was found to be similar to that of LSTM.^[4][5] GRUs showed that gating is indeed helpful in general, andBengio's team came to no concrete conclusion on which of the two gating units was better.^[6][7]

There are several variations on the full gated unit, with gating done using the previous hidden state and the bias in various combinations, and a simplified form called minimal gated unit.^[8]

In the following, the operator${\displaystyle \odot }$ denotes theHadamard product.

Initially, for${\displaystyle t=0}$, the output vector is${\displaystyle h_0=0}$.

Variables (${\displaystyle d}$ denotes the number of input features and${\displaystyle e}$ the number of output features):

• ${\displaystyle x_t\in {\mathbb{R}}^d}$: input vector
• ${\displaystyle h_t\in {\mathbb{R}}^e}$: output vector
• ${\displaystyle {\hat{h}}_t\in {\mathbb{R}}^e}$: candidate activation vector
• ${\displaystyle z_t\in (0,1{)}^e}$: update gate vector
• ${\displaystyle r_t\in (0,1{)}^e}$: reset gate vector
• ${\displaystyle W\in {\mathbb{R}}^{e\times d}}$,${\displaystyle U\in {\mathbb{R}}^{e\times e}}$ and${\displaystyle b\in {\mathbb{R}}^e}$: parameter matrices and vector which need to be learned during training

Activation functions

• ${\displaystyle \sigma }$: The original is alogistic function.
• ${\displaystyle \varphi }$: The original is ahyperbolic tangent.

Alternative activation functions are possible, provided that${\displaystyle \sigma (x)\in [0,1]}$.

Alternate forms can be created by changing${\displaystyle z_t}$ and${\displaystyle r_t}$^[9]

• Type 1: each gate depends only on the previous hidden state and the bias.

  

• Type 2: each gate depends only on the previous hidden state.

  

• Type 3: each gate is computed using only the bias.

  

The minimal gated unit (MGU) is similar to the fully gated unit, except the update and reset gate vector is merged into a forget gate. This also implies that the equation for the output vector must be changed:^[10]

Variables

• ${\displaystyle x_t}$: input vector
• ${\displaystyle h_t}$: output vector
• ${\displaystyle {\hat{h}}_t}$: candidate activation vector
• ${\displaystyle f_t}$: forget vector
• ${\displaystyle W}$,${\displaystyle U}$ and${\displaystyle b}$: parameter matrices and vector

The light gated recurrent unit (LiGRU)^[4] removes the reset gate altogether, replacestanh with theReLU activation, and appliesbatch normalization (BN):

LiGRU has been studied from a Bayesian perspective.^[11] This analysis yielded a variant called light Bayesian recurrent unit (LiBRU), which showed slight improvements over the LiGRU onspeech recognition tasks.

• Neural network architectures
• Artificial neural networks
• 2014 software
• 2014 in artificial intelligence

• Articles with short description
• Short description is different from Wikidata