RReLU#

classtorch.nn.modules.activation.RReLU(lower=0.125,upper=0.3333333333333333,inplace=False)[source]#

Applies the randomized leaky rectified linear unit function, element-wise.

Method described in the paper:Empirical Evaluation of Rectified Activations in Convolutional Network.

The function is defined as:

RReLU(x)={xax​if x≥0 otherwise ​

where$a$a is randomly sampled from uniform distribution$\mathcal{U}(\text{lower},\text{upper})$U(lower,upper) during training while duringevaluation$a$a is fixed with$a=\frac{\text{lower}+\text{upper}}{2}$a=2lower+upper​.

Parameters

• lower (float) – lower bound of the uniform distribution. Default:$\frac{1}{8}$81​

• upper (float) – upper bound of the uniform distribution. Default:$\frac{1}{3}$31​

• inplace (bool) – can optionally do the operation in-place. Default:False

Shape:

• Input:$(\ast )$(∗), where$\ast$∗ means any number of dimensions.

• Output:$(\ast )$(∗), same shape as the input.

Examples:

>>>m=nn.RReLU(0.1,0.3)>>>input=torch.randn(2)>>>output=m(input)

extra_repr()[source]#

Return the extra representation of the module.

Return type

str

forward(input)[source]#

Runs the forward pass.

Return type

Tensor