torch.nn.utils.parametrizations.orthogonal#

torch.nn.utils.parametrizations.orthogonal(module,name='weight',orthogonal_map=None,*,use_trivialization=True)[source]#

Apply an orthogonal or unitary parametrization to a matrix or a batch of matrices.

Letting$\mathbb{K}$K be$\mathbb{R}$R or$\mathbb{C}$C, the parametrizedmatrix$Q\in {\mathbb{K}}^{m\times n}$Q∈Km×n isorthogonal as

QHQQQH​=In​if m≥n=Im​if m<n​

where$Q^{\text{H}}$QH is the conjugate transpose when$Q$Q is complexand the transpose when$Q$Q is real-valued, and${\mathrm{I}}_n$In​ is then-dimensional identity matrix.In plain words,$Q$Q will have orthonormal columns whenever$m\ge n$m≥nand orthonormal rows otherwise.

If the tensor has more than two dimensions, we consider it as a batch of matrices of shape(…, m, n).

The matrix$Q$Q may be parametrized via three differentorthogonal_map in terms of the original tensor:

• "matrix_exp"/"cayley":thematrix_exp()$Q=\exp (A)$Q=exp(A) and theCayley map$Q=({\mathrm{I}}_n+A\mathrm{/}2)({\mathrm{I}}_n-A\mathrm{/}2{)}^{-1}$Q=(In​+A/2)(In​−A/2)−1 are applied to a skew-symmetric$A$A to give an orthogonal matrix.

• "householder": computes a product of Householder reflectors(householder_product()).

"matrix_exp"/"cayley" often make the parametrized weight converge faster than"householder", but they are slower to compute for very thin or very wide matrices.

Ifuse_trivialization=True (default), the parametrization implements the “Dynamic Trivialization Framework”,where an extra matrix$B\in {\mathbb{K}}^{n\times n}$B∈Kn×n is stored undermodule.parametrizations.weight[0].base. This helps theconvergence of the parametrized layer at the expense of some extra memory use.SeeTrivializations for Gradient-Based Optimization on Manifolds .

Initial value of$Q$Q:If the original tensor is not parametrized anduse_trivialization=True (default), the initial valueof$Q$Q is that of the original tensor if it is orthogonal (or unitary in the complex case)and it is orthogonalized via the QR decomposition otherwise (seetorch.linalg.qr()).Same happens when it is not parametrized andorthogonal_map="householder" even whenuse_trivialization=False.Otherwise, the initial value is the result of the composition of all the registeredparametrizations applied to the original tensor.

Note

This function is implemented using the parametrization functionalityinregister_parametrization().

Parameters

• module (nn.Module) – module on which to register the parametrization.

• name (str,optional) – name of the tensor to make orthogonal. Default:"weight".

• orthogonal_map (str,optional) – One of the following:"matrix_exp","cayley","householder".Default:"matrix_exp" if the matrix is square or complex,"householder" otherwise.

• use_trivialization (bool,optional) – whether to use the dynamic trivialization framework.Default:True.

Returns

The original module with an orthogonal parametrization registered to the specifiedweight

Return type

Module

Example:

>>>orth_linear=orthogonal(nn.Linear(20,40))>>>orth_linearParametrizedLinear(in_features=20, out_features=40, bias=True(parametrizations): ModuleDict(    (weight): ParametrizationList(    (0): _Orthogonal()    )))>>>Q=orth_linear.weight>>>torch.dist(Q.T@Q,torch.eye(20))tensor(4.9332e-07)