LSTM#

classtorch.nn.LSTM(input_size,hidden_size,num_layers=1,bias=True,batch_first=False,dropout=0.0,bidirectional=False,proj_size=0,device=None,dtype=None)[source]#

Apply a multi-layer long short-term memory (LSTM) RNN to an input sequence.For each element in the input sequence, each layer computes the followingfunction:

$$\begin{array}{l}\\ i_t=\sigma (W_{ii}{x}_t+b_{ii}+W_{hi}{h}_{t-1}+b_{hi})\\ f_t=\sigma (W_{if}{x}_t+b_{if}+W_{hf}{h}_{t-1}+b_{hf})\\ g_t=\tanh (W_{ig}{x}_t+b_{ig}+W_{hg}{h}_{t-1}+b_{hg})\\ o_t=\sigma (W_{io}{x}_t+b_{io}+W_{ho}{h}_{t-1}+b_{ho})\\ c_t=f_t\odot c_{t-1}+i_t\odot g_t\\ h_t=o_t\odot \tanh (c_t)\end{array}$$it​=σ(Wii​xt​+bii​+Whi​ht−1​+bhi​)ft​=σ(Wif​xt​+bif​+Whf​ht−1​+bhf​)gt​=tanh(Wig​xt​+big​+Whg​ht−1​+bhg​)ot​=σ(Wio​xt​+bio​+Who​ht−1​+bho​)ct​=ft​⊙ct−1​+it​⊙gt​ht​=ot​⊙tanh(ct​)​

where$h_t$ht​ is the hidden state at timet,$c_t$ct​ is the cellstate at timet,$x_t$xt​ is the input at timet,$h_{t-1}$ht−1​is the hidden state of the layer at timet-1 or the initial hiddenstate at time0, and$i_t$it​,$f_t$ft​,$g_t$gt​,$o_t$ot​ are the input, forget, cell, and output gates, respectively.$\sigma$σ is the sigmoid function, and$\odot$⊙ is the Hadamard product.

In a multilayer LSTM, the input$x_t^{(l)}$xt(l)​ of the$l$l -th layer($l\ge 2$l≥2) is the hidden state$h_t^{(l-1)}$ht(l−1)​ of the previous layer multiplied bydropout${\delta }_t^{(l-1)}$δt(l−1)​ where each${\delta }_t^{(l-1)}$δt(l−1)​ is a Bernoulli randomvariable which is$0$0 with probabilitydropout.

Ifproj_size>0 is specified, LSTM with projections will be used. This changesthe LSTM cell in the following way. First, the dimension of$h_t$ht​ will be changed fromhidden_size toproj_size (dimensions of$W_{hi}$Whi​ will be changed accordingly).Second, the output hidden state of each layer will be multiplied by a learnable projectionmatrix:$h_t=W_{hr}{h}_t$ht​=Whr​ht​. Note that as a consequence of this, the outputof LSTM network will be of different shape as well. See Inputs/Outputs sections below for exactdimensions of all variables. You can find more details inhttps://arxiv.org/abs/1402.1128.

Parameters

• input_size – The number of expected features in the inputx

• hidden_size – The number of features in the hidden stateh

• num_layers – Number of recurrent layers. E.g., settingnum_layers=2would mean stacking two LSTMs together to form astacked LSTM,with the second LSTM taking in outputs of the first LSTM andcomputing the final results. Default: 1

• bias – IfFalse, then the layer does not use bias weightsb_ih andb_hh.Default:True

• batch_first – IfTrue, then the input and output tensors are providedas(batch, seq, feature) instead of(seq, batch, feature).Note that this does not apply to hidden or cell states. See theInputs/Outputs sections below for details. Default:False

• dropout – If non-zero, introduces aDropout layer on the outputs of eachLSTM layer except the last layer, with dropout probability equal todropout. Default: 0

• bidirectional – IfTrue, becomes a bidirectional LSTM. Default:False

• proj_size – If>0, will use LSTM with projections of corresponding size. Default: 0

Inputs: input, (h_0, c_0)

• input: tensor of shape$(L,H_{in})$(L,Hin​) for unbatched input,$(L,N,H_{in})$(L,N,Hin​) whenbatch_first=False or$(N,L,H_{in})$(N,L,Hin​) whenbatch_first=True containing the features ofthe input sequence. The input can also be a packed variable length sequence.Seetorch.nn.utils.rnn.pack_padded_sequence() ortorch.nn.utils.rnn.pack_sequence() for details.

• h_0: tensor of shape$(D\ast \text{num\_layers},H_{out})$(D∗num_layers,Hout​) for unbatched input or$(D\ast \text{num\_layers},N,H_{out})$(D∗num_layers,N,Hout​) containing theinitial hidden state for each element in the input sequence.Defaults to zeros if (h_0, c_0) is not provided.

• c_0: tensor of shape$(D\ast \text{num\_layers},H_{cell})$(D∗num_layers,Hcell​) for unbatched input or$(D\ast \text{num\_layers},N,H_{cell})$(D∗num_layers,N,Hcell​) containing theinitial cell state for each element in the input sequence.Defaults to zeros if (h_0, c_0) is not provided.

where:

N=L=D=Hin​=Hcell​=Hout​=​batch sizesequence length2 if bidirectional=True otherwise 1input_sizehidden_sizeproj_size if proj_size>0 otherwise hidden_size​

Outputs: output, (h_n, c_n)

• output: tensor of shape$(L,D\ast H_{out})$(L,D∗Hout​) for unbatched input,$(L,N,D\ast H_{out})$(L,N,D∗Hout​) whenbatch_first=False or$(N,L,D\ast H_{out})$(N,L,D∗Hout​) whenbatch_first=True containing the output features(h_t) from the last layer of the LSTM, for eacht. If atorch.nn.utils.rnn.PackedSequence has been given as the input, the outputwill also be a packed sequence. Whenbidirectional=True,output will containa concatenation of the forward and reverse hidden states at each time step in the sequence.

• h_n: tensor of shape$(D\ast \text{num\_layers},H_{out})$(D∗num_layers,Hout​) for unbatched input or$(D\ast \text{num\_layers},N,H_{out})$(D∗num_layers,N,Hout​) containing thefinal hidden state for each element in the sequence. Whenbidirectional=True,h_n will contain a concatenation of the final forward and reverse hidden states, respectively.

• c_n: tensor of shape$(D\ast \text{num\_layers},H_{cell})$(D∗num_layers,Hcell​) for unbatched input or$(D\ast \text{num\_layers},N,H_{cell})$(D∗num_layers,N,Hcell​) containing thefinal cell state for each element in the sequence. Whenbidirectional=True,c_n will contain a concatenation of the final forward and reverse cell states, respectively.

Variables

• weight_ih_l[k] – the learnable input-hidden weights of the${\text{k}}^{th}$kth layer(W_ii|W_if|W_ig|W_io), of shape(4*hidden_size, input_size) fork = 0.Otherwise, the shape is(4*hidden_size, num_directions * hidden_size). Ifproj_size>0 was specified, the shape will be(4*hidden_size, num_directions * proj_size) fork > 0

• weight_hh_l[k] – the learnable hidden-hidden weights of the${\text{k}}^{th}$kth layer(W_hi|W_hf|W_hg|W_ho), of shape(4*hidden_size, hidden_size). Ifproj_size>0was specified, the shape will be(4*hidden_size, proj_size).

• bias_ih_l[k] – the learnable input-hidden bias of the${\text{k}}^{th}$kth layer(b_ii|b_if|b_ig|b_io), of shape(4*hidden_size)

• bias_hh_l[k] – the learnable hidden-hidden bias of the${\text{k}}^{th}$kth layer(b_hi|b_hf|b_hg|b_ho), of shape(4*hidden_size)

• weight_hr_l[k] – the learnable projection weights of the${\text{k}}^{th}$kth layerof shape(proj_size, hidden_size). Only present whenproj_size>0 wasspecified.

• weight_ih_l[k]_reverse – Analogous toweight_ih_l[k] for the reverse direction.Only present whenbidirectional=True.

• weight_hh_l[k]_reverse – Analogous toweight_hh_l[k] for the reverse direction.Only present whenbidirectional=True.

• bias_ih_l[k]_reverse – Analogous tobias_ih_l[k] for the reverse direction.Only present whenbidirectional=True.

• bias_hh_l[k]_reverse – Analogous tobias_hh_l[k] for the reverse direction.Only present whenbidirectional=True.

• weight_hr_l[k]_reverse – Analogous toweight_hr_l[k] for the reverse direction.Only present whenbidirectional=True andproj_size>0 was specified.

Note

All the weights and biases are initialized from$\mathcal{U}(-\sqrt{k},\sqrt{k})$U(−k​,k​)where$k=\frac{1}{\text{hidden\_size}}$k=hidden_size1​

Note

For bidirectional LSTMs, forward and backward are directions 0 and 1 respectively.Example of splitting the output layers whenbatch_first=False:output.view(seq_len,batch,num_directions,hidden_size).

Note

For bidirectional LSTMs,h_n is not equivalent to the last element ofoutput; theformer contains the final forward and reverse hidden states, while the latter contains thefinal forward hidden state and the initial reverse hidden state.

Note

batch_first argument is ignored for unbatched inputs.

Note

proj_size should be smaller thanhidden_size.

Warning

There are known non-determinism issues for RNN functions on some versions of cuDNN and CUDA.You can enforce deterministic behavior by setting the following environment variables:

On CUDA 10.1, set environment variableCUDA_LAUNCH_BLOCKING=1.This may affect performance.

On CUDA 10.2 or later, set environment variable(note the leading colon symbol)CUBLAS_WORKSPACE_CONFIG=:16:8orCUBLAS_WORKSPACE_CONFIG=:4096:2.

See thecuDNN 8 Release Notes for more information.

Note

If the following conditions are satisfied:1) cudnn is enabled,2) input data is on the GPU3) input data has dtypetorch.float164) V100 GPU is used,5) input data is not inPackedSequence formatpersistent algorithm can be selected to improve performance.

Examples:

>>>rnn=nn.LSTM(10,20,2)>>>input=torch.randn(5,3,10)>>>h0=torch.randn(2,3,20)>>>c0=torch.randn(2,3,20)>>>output,(hn,cn)=rnn(input,(h0,c0))