GELU#

classtorch.nn.GELU(approximate='none')[source]#

Applies the Gaussian Error Linear Units function.

$$\text{GELU}(x)=x\ast \mathrm{\Phi }(x)$$GELU(x)=x∗Φ(x)

where$\mathrm{\Phi }(x)$Φ(x) is the Cumulative Distribution Function for Gaussian Distribution.

When the approximate argument is ‘tanh’, Gelu is estimated with:

$$\text{GELU}(x)=0.5\ast x\ast (1+\text{Tanh}(\sqrt{2\mathrm{/}\pi }\ast (x+0.044715\ast x^3)))$$GELU(x)=0.5∗x∗(1+Tanh(2/π​∗(x+0.044715∗x3)))

Parameters:

approximate (str,optional) – the gelu approximation algorithm to use:'none' |'tanh'. Default:'none'

Shape:

• Input:$(\ast )$(∗), where$\ast$∗ means any number of dimensions.

• Output:$(\ast )$(∗), same shape as the input.

Examples:

>>>m=nn.GELU()>>>input=torch.randn(2)>>>output=m(input)

extra_repr()[source]#

Return the extra representation of the module.

Return type:

str

forward(input)[source]#

Runs the forward pass.

Return type:

Tensor