SiLU#

classtorch.nn.modules.activation.SiLU(inplace=False)[source]#

Applies the Sigmoid Linear Unit (SiLU) function, element-wise.

The SiLU function is also known as the swish function.

$$\text{silu}(x)=x\ast \sigma (x),\text{where }\sigma (x)\text{ is the logistic sigmoid.}$$silu(x)=x∗σ(x),where σ(x) is the logistic sigmoid.

Note

SeeGaussian Error Linear Units (GELUs)where the SiLU (Sigmoid Linear Unit) was originally coined, and seeSigmoid-Weighted Linear Units for Neural Network Function Approximationin Reinforcement Learning andSwish:a Self-Gated Activation Functionwhere the SiLU was experimented with later.

Shape:

• Input:$(\ast )$(∗), where$\ast$∗ means any number of dimensions.

• Output:$(\ast )$(∗), same shape as the input.

Examples:

>>>m=nn.SiLU()>>>input=torch.randn(2)>>>output=m(input)

extra_repr()[source]#

Return the extra representation of the module.

Return type

str

forward(input)[source]#

Runs the forward pass.

Return type

Tensor