torch.nn.attention.bias.causal_upper_left#

torch.nn.attention.bias.causal_upper_left(*size)[source]#

Creates an upper-left triangular causal bias.

This function generates a upper-left triangular matrix to represent causal attention bias with adiagonal offset set so that the inclusive values are aligned to the upper left corner of the matrix.This equivalent to theis_causal=True argument inscaled_dot_product_attention.

The equivalent pytorch code for constructing this bias is:

torch.tril(torch.ones(size,dtype=torch.bool))

For instance, withshape=(3,4), the materialized bias tensor will be:

[[1, 0, 0, 0], [1, 1, 0, 0], [1, 1, 1, 0]]

Parameters

size – The size of the bias matrix.

Returns

The UPPER_LEFT triangular causal bias variant.

Return type

CausalBias