torch.nn.attention.bias.causal_lower_right#

torch.nn.attention.bias.causal_lower_right(*size)[source]#

Creates a lower-right triangular causal bias.

This function generates a lower-right triangular matrix to represent causal attention bias with adiagonal offset set so that the inclusive values are aligned to the lower right corner of the matrix.

The equivalent pytorch code for constructing this bias is:

diagonal_offset=size[1]-size[0]torch.tril(torch.ones(size,dtype=torch.bool),diagonal=diagonal_offset,)

For instance, withshape=(3,4), the materialized bias tensor will be:

[[1, 1, 0, 0], [1, 1, 1, 0], [1, 1, 1, 1]]

Parameters

size – The size of the bias matrix.

Returns

The LOWER_RIGHT triangular causal bias variant.

Return type

CausalBias