RNN#

classtorch.nn.modules.rnn.RNN(input_size,hidden_size,num_layers=1,nonlinearity='tanh',bias=True,batch_first=False,dropout=0.0,bidirectional=False,device=None,dtype=None)[source]#

Apply a multi-layer Elman RNN with$\tanh$tanh or$\text{ReLU}$ReLUnon-linearity to an input sequence. For each element in the input sequence,each layer computes the following function:

$$h_t=\tanh (x_t{W}_{ih}^T+b_{ih}+h_{t-1}{W}_{hh}^T+b_{hh})$$ht​=tanh(xt​WihT​+bih​+ht−1​WhhT​+bhh​)

where$h_t$ht​ is the hidden state at timet,$x_t$xt​ isthe input at timet, and$h_{(t-1)}$h(t−1)​ is the hidden state of theprevious layer at timet-1 or the initial hidden state at time0.Ifnonlinearity is'relu', then$\text{ReLU}$ReLU is used instead of$\tanh$tanh.

# Efficient implementation equivalent to the following with bidirectional=Falsernn=nn.RNN(input_size,hidden_size,num_layers)params=dict(rnn.named_parameters())defforward(x,hx=None,batch_first=False):ifbatch_first:x=x.transpose(0,1)seq_len,batch_size,_=x.size()ifhxisNone:hx=torch.zeros(rnn.num_layers,batch_size,rnn.hidden_size)h_t_minus_1=hx.clone()h_t=hx.clone()output=[]fortinrange(seq_len):forlayerinrange(rnn.num_layers):input_t=x[t]iflayer==0elseh_t[layer-1]h_t[layer]=torch.tanh(input_t@params[f"weight_ih_l{layer}"].T+h_t_minus_1[layer]@params[f"weight_hh_l{layer}"].T+params[f"bias_hh_l{layer}"]+params[f"bias_ih_l{layer}"])output.append(h_t[-1].clone())h_t_minus_1=h_t.clone()output=torch.stack(output)ifbatch_first:output=output.transpose(0,1)returnoutput,h_t

Parameters

• input_size – The number of expected features in the inputx

• hidden_size – The number of features in the hidden stateh

• num_layers – Number of recurrent layers. E.g., settingnum_layers=2would mean stacking two RNNs together to form astacked RNN,with the second RNN taking in outputs of the first RNN andcomputing the final results. Default: 1

• nonlinearity – The non-linearity to use. Can be either'tanh' or'relu'. Default:'tanh'

• bias – IfFalse, then the layer does not use bias weightsb_ih andb_hh.Default:True

• batch_first – IfTrue, then the input and output tensors are providedas(batch, seq, feature) instead of(seq, batch, feature).Note that this does not apply to hidden or cell states. See theInputs/Outputs sections below for details. Default:False

• dropout – If non-zero, introduces aDropout layer on the outputs of eachRNN layer except the last layer, with dropout probability equal todropout. Default: 0

• bidirectional – IfTrue, becomes a bidirectional RNN. Default:False

Inputs: input, hx

• input: tensor of shape$(L,H_{in})$(L,Hin​) for unbatched input,$(L,N,H_{in})$(L,N,Hin​) whenbatch_first=False or$(N,L,H_{in})$(N,L,Hin​) whenbatch_first=True containing the features ofthe input sequence. The input can also be a packed variable length sequence.Seetorch.nn.utils.rnn.pack_padded_sequence() ortorch.nn.utils.rnn.pack_sequence() for details.

• hx: tensor of shape$(D\ast \text{num\_layers},H_{out})$(D∗num_layers,Hout​) for unbatched input or$(D\ast \text{num\_layers},N,H_{out})$(D∗num_layers,N,Hout​) containing the initial hiddenstate for the input sequence batch. Defaults to zeros if not provided.

where:

N=L=D=Hin​=Hout​=​batch sizesequence length2 if bidirectional=True otherwise 1input_sizehidden_size​

Outputs: output, h_n

• output: tensor of shape$(L,D\ast H_{out})$(L,D∗Hout​) for unbatched input,$(L,N,D\ast H_{out})$(L,N,D∗Hout​) whenbatch_first=False or$(N,L,D\ast H_{out})$(N,L,D∗Hout​) whenbatch_first=True containing the output features(h_t) from the last layer of the RNN, for eacht. If atorch.nn.utils.rnn.PackedSequence has been given as the input, the outputwill also be a packed sequence.

• h_n: tensor of shape$(D\ast \text{num\_layers},H_{out})$(D∗num_layers,Hout​) for unbatched input or$(D\ast \text{num\_layers},N,H_{out})$(D∗num_layers,N,Hout​) containing the final hidden statefor each element in the batch.

Variables

• weight_ih_l[k] – the learnable input-hidden weights of the k-th layer,of shape(hidden_size, input_size) fork = 0. Otherwise, the shape is(hidden_size, num_directions * hidden_size)

• weight_hh_l[k] – the learnable hidden-hidden weights of the k-th layer,of shape(hidden_size, hidden_size)

• bias_ih_l[k] – the learnable input-hidden bias of the k-th layer,of shape(hidden_size)

• bias_hh_l[k] – the learnable hidden-hidden bias of the k-th layer,of shape(hidden_size)

Note

All the weights and biases are initialized from$\mathcal{U}(-\sqrt{k},\sqrt{k})$U(−k​,k​)where$k=\frac{1}{\text{hidden\_size}}$k=hidden_size1​

Note

For bidirectional RNNs, forward and backward are directions 0 and 1 respectively.Example of splitting the output layers whenbatch_first=False:output.view(seq_len,batch,num_directions,hidden_size).

Note

batch_first argument is ignored for unbatched inputs.

Warning

There are known non-determinism issues for RNN functions on some versions of cuDNN and CUDA.You can enforce deterministic behavior by setting the following environment variables:

On CUDA 10.1, set environment variableCUDA_LAUNCH_BLOCKING=1.This may affect performance.

On CUDA 10.2 or later, set environment variable(note the leading colon symbol)CUBLAS_WORKSPACE_CONFIG=:16:8orCUBLAS_WORKSPACE_CONFIG=:4096:2.

See thecuDNN 8 Release Notes for more information.

Note

If the following conditions are satisfied:1) cudnn is enabled,2) input data is on the GPU3) input data has dtypetorch.float164) V100 GPU is used,5) input data is not inPackedSequence formatpersistent algorithm can be selected to improve performance.

Examples:

>>>rnn=nn.RNN(10,20,2)>>>input=torch.randn(5,3,10)>>>h0=torch.randn(2,3,20)>>>output,hn=rnn(input,h0)

forward(input:Tensor,hx:Optional[Tensor]=None)→tuple[torch.Tensor,torch.Tensor][source]#

forward(input:PackedSequence,hx:Optional[Tensor]=None)→tuple[torch.nn.utils.rnn.PackedSequence,torch.Tensor]

Runs the forward pass.