GRU#

classtorch.nn.GRU(input_size,hidden_size,num_layers=1,bias=True,batch_first=False,dropout=0.0,bidirectional=False,device=None,dtype=None)[source]#

Apply a multi-layer gated recurrent unit (GRU) RNN to an input sequence.For each element in the input sequence, each layer computes the followingfunction:

$$\begin{array}{l}{r}_t=\sigma (W_{ir}{x}_t+b_{ir}+W_{hr}{h}_{(t-1)}+b_{hr})\\ z_t=\sigma (W_{iz}{x}_t+b_{iz}+W_{hz}{h}_{(t-1)}+b_{hz})\\ n_t=\tanh (W_{in}{x}_t+b_{in}+r_t\odot (W_{hn}{h}_{(t-1)}+b_{hn}))\\ h_t=(1-z_t)\odot n_t+z_t\odot h_{(t-1)}\end{array}$$rt​=σ(Wir​xt​+bir​+Whr​h(t−1)​+bhr​)zt​=σ(Wiz​xt​+biz​+Whz​h(t−1)​+bhz​)nt​=tanh(Win​xt​+bin​+rt​⊙(Whn​h(t−1)​+bhn​))ht​=(1−zt​)⊙nt​+zt​⊙h(t−1)​​

where$h_t$ht​ is the hidden state at timet,$x_t$xt​ is the inputat timet,$h_{(t-1)}$h(t−1)​ is the hidden state of the layerat timet-1 or the initial hidden state at time0, and$r_t$rt​,$z_t$zt​,$n_t$nt​ are the reset, update, and new gates, respectively.$\sigma$σ is the sigmoid function, and$\odot$⊙ is the Hadamard product.

In a multilayer GRU, the input$x_t^{(l)}$xt(l)​ of the$l$l -th layer($l\ge 2$l≥2) is the hidden state$h_t^{(l-1)}$ht(l−1)​ of the previous layer multiplied bydropout${\delta }_t^{(l-1)}$δt(l−1)​ where each${\delta }_t^{(l-1)}$δt(l−1)​ is a Bernoulli randomvariable which is$0$0 with probabilitydropout.

Parameters:

• input_size – The number of expected features in the inputx

• hidden_size – The number of features in the hidden stateh

• num_layers – Number of recurrent layers. E.g., settingnum_layers=2would mean stacking two GRUs together to form astacked GRU,with the second GRU taking in outputs of the first GRU andcomputing the final results. Default: 1

• bias – IfFalse, then the layer does not use bias weightsb_ih andb_hh.Default:True

• batch_first – IfTrue, then the input and output tensors are providedas(batch, seq, feature) instead of(seq, batch, feature).Note that this does not apply to hidden or cell states. See theInputs/Outputs sections below for details. Default:False

• dropout – If non-zero, introduces aDropout layer on the outputs of eachGRU layer except the last layer, with dropout probability equal todropout. Default: 0

• bidirectional – IfTrue, becomes a bidirectional GRU. Default:False

Inputs: input, h_0

• input: tensor of shape$(L,H_{in})$(L,Hin​) for unbatched input,$(L,N,H_{in})$(L,N,Hin​) whenbatch_first=False or$(N,L,H_{in})$(N,L,Hin​) whenbatch_first=True containing the features ofthe input sequence. The input can also be a packed variable length sequence.Seetorch.nn.utils.rnn.pack_padded_sequence() ortorch.nn.utils.rnn.pack_sequence() for details.

• h_0: tensor of shape$(D\ast \text{num\_layers},H_{out})$(D∗num_layers,Hout​) or$(D\ast \text{num\_layers},N,H_{out})$(D∗num_layers,N,Hout​)containing the initial hidden state for the input sequence. Defaults to zeros if not provided.

where:

N=L=D=Hin​=Hout​=​batch sizesequence length2 if bidirectional=True otherwise 1input_sizehidden_size​

Outputs: output, h_n

• output: tensor of shape$(L,D\ast H_{out})$(L,D∗Hout​) for unbatched input,$(L,N,D\ast H_{out})$(L,N,D∗Hout​) whenbatch_first=False or$(N,L,D\ast H_{out})$(N,L,D∗Hout​) whenbatch_first=True containing the output features(h_t) from the last layer of the GRU, for eacht. If atorch.nn.utils.rnn.PackedSequence has been given as the input, the outputwill also be a packed sequence.

• h_n: tensor of shape$(D\ast \text{num\_layers},H_{out})$(D∗num_layers,Hout​) or$(D\ast \text{num\_layers},N,H_{out})$(D∗num_layers,N,Hout​) containing the final hidden statefor the input sequence.

Variables:

• weight_ih_l[k] – the learnable input-hidden weights of the${\text{k}}^{th}$kth layer(W_ir|W_iz|W_in), of shape(3*hidden_size, input_size) fork = 0.Otherwise, the shape is(3*hidden_size, num_directions * hidden_size)

• weight_hh_l[k] – the learnable hidden-hidden weights of the${\text{k}}^{th}$kth layer(W_hr|W_hz|W_hn), of shape(3*hidden_size, hidden_size)

• bias_ih_l[k] – the learnable input-hidden bias of the${\text{k}}^{th}$kth layer(b_ir|b_iz|b_in), of shape(3*hidden_size)

• bias_hh_l[k] – the learnable hidden-hidden bias of the${\text{k}}^{th}$kth layer(b_hr|b_hz|b_hn), of shape(3*hidden_size)

Note

All the weights and biases are initialized from$\mathcal{U}(-\sqrt{k},\sqrt{k})$U(−k​,k​)where$k=\frac{1}{\text{hidden\_size}}$k=hidden_size1​

Note

For bidirectional GRUs, forward and backward are directions 0 and 1 respectively.Example of splitting the output layers whenbatch_first=False:output.view(seq_len,batch,num_directions,hidden_size).

Note

batch_first argument is ignored for unbatched inputs.

Note

The calculation of new gate$n_t$nt​ subtly differs from the original paper and other frameworks.In the original implementation, the Hadamard product$(\odot )$(⊙) between$r_t$rt​ and theprevious hidden state$h_{(t-1)}$h(t−1)​ is done before the multiplication with the weight matrixW and addition of bias:

$$\begin{array}{r}{\displaystyle n_t=\tanh (W_{in}{x}_t+b_{in}+W_{hn}(r_t\odot h_{(t-1)})+b_{hn})}\end{array}$$nt​=tanh(Win​xt​+bin​+Whn​(rt​⊙h(t−1)​)+bhn​)​

This is in contrast to PyTorch implementation, which is done after$W_{hn}{h}_{(t-1)}$Whn​h(t−1)​

$$\begin{array}{r}{\displaystyle n_t=\tanh (W_{in}{x}_t+b_{in}+r_t\odot (W_{hn}{h}_{(t-1)}+b_{hn}))}\end{array}$$nt​=tanh(Win​xt​+bin​+rt​⊙(Whn​h(t−1)​+bhn​))​

This implementation differs on purpose for efficiency.

Note

If the following conditions are satisfied:1) cudnn is enabled,2) input data is on the GPU3) input data has dtypetorch.float164) V100 GPU is used,5) input data is not inPackedSequence formatpersistent algorithm can be selected to improve performance.

Examples:

>>>rnn=nn.GRU(10,20,2)>>>input=torch.randn(5,3,10)>>>h0=torch.randn(2,3,20)>>>output,hn=rnn(input,h0)