torch.use_deterministic_algorithms#

torch.use_deterministic_algorithms(mode,*,warn_only=False)[source]#

Sets whether PyTorch operations must use “deterministic”algorithms. That is, algorithms which, given the same input, and whenrun on the same software and hardware, always produce the same output.When enabled, operations will use deterministic algorithms when available,and if only nondeterministic algorithms are available they will throw aRuntimeError when called.

Note

This setting alone is not always enough to make an applicationreproducible. Refer toReproducibility for more information.

Note

torch.set_deterministic_debug_mode() offers an alternativeinterface for this feature.

The following normally-nondeterministic operations will actdeterministically whenmode=True:

• torch.nn.Conv1d when called on CUDA tensor

• torch.nn.Conv2d when called on CUDA tensor

• torch.nn.Conv3d when called on CUDA tensor

• torch.nn.ConvTranspose1d when called on CUDA tensor

• torch.nn.ConvTranspose2d when called on CUDA tensor

• torch.nn.ConvTranspose3d when called on CUDA tensor

• torch.nn.ReplicationPad1d when attempting to differentiate a CUDA tensor

• torch.nn.ReplicationPad2d when attempting to differentiate a CUDA tensor

• torch.nn.ReplicationPad3d when attempting to differentiate a CUDA tensor

• torch.bmm() when called on sparse-dense CUDA tensors

• torch.Tensor.__getitem__() when attempting to differentiate a CPU tensorand the index is a list of tensors

• torch.Tensor.index_put() withaccumulate=False

• torch.Tensor.index_put() withaccumulate=True when called on a CPUtensor

• torch.Tensor.put_() withaccumulate=True when called on a CPUtensor

• torch.Tensor.scatter_add_() when called on a CUDA tensor

• torch.gather() when called on a CUDA tensor that requires grad

• torch.index_add() when called on CUDA tensor

• torch.index_select() when attempting to differentiate a CUDA tensor

• torch.repeat_interleave() when attempting to differentiate a CUDA tensor

• torch.Tensor.index_copy() when called on a CPU or CUDA tensor

• torch.Tensor.scatter() whensrc type is Tensor and called on CUDA tensor

• torch.Tensor.scatter_reduce() whenreduce='sum' orreduce='mean' and called on CUDA tensor

The following normally-nondeterministic operations will throw aRuntimeError whenmode=True:

• torch.nn.AvgPool3d when attempting to differentiate a CUDA tensor

• torch.nn.AdaptiveAvgPool2d when attempting to differentiate a CUDA tensor

• torch.nn.AdaptiveAvgPool3d when attempting to differentiate a CUDA tensor

• torch.nn.MaxPool3d when attempting to differentiate a CUDA tensor

• torch.nn.AdaptiveMaxPool2d when attempting to differentiate a CUDA tensor

• torch.nn.FractionalMaxPool2d when attempting to differentiate a CUDA tensor

• torch.nn.FractionalMaxPool3d when attempting to differentiate a CUDA tensor

• torch.nn.MaxUnpool1d

• torch.nn.MaxUnpool2d

• torch.nn.MaxUnpool3d

• torch.nn.functional.interpolate() when attempting to differentiate a CUDA tensorand one of the following modes is used:

  • linear

  • bilinear

  • bicubic

  • trilinear

• torch.nn.ReflectionPad1d when attempting to differentiate a CUDA tensor

• torch.nn.ReflectionPad2d when attempting to differentiate a CUDA tensor

• torch.nn.ReflectionPad3d when attempting to differentiate a CUDA tensor

• torch.nn.NLLLoss when called on a CUDA tensor

• torch.nn.CTCLoss when attempting to differentiate a CUDA tensor

• torch.nn.EmbeddingBag when attempting to differentiate a CUDA tensor whenmode='max'

• torch.Tensor.put_() whenaccumulate=False

• torch.Tensor.put_() whenaccumulate=True and called on a CUDA tensor

• torch.histc() when called on a CUDA tensor

• torch.bincount() when called on a CUDA tensor andweightstensor is given

• torch.kthvalue() with called on a CUDA tensor

• torch.median() with indices output when called on a CUDA tensor

• torch.nn.functional.grid_sample() when attempting to differentiate a CUDA tensor

• torch.cumsum() when called on a CUDA tensor when dtype is floating point or complex

• torch.Tensor.scatter_reduce() whenreduce='prod' and called on CUDA tensor

• torch.Tensor.resize_() when called with a quantized tensor

In addition, several operations fill uninitialized memory when this settingis turned on and whentorch.utils.deterministic.fill_uninitialized_memory is turned on.See the documentation for that attribute for more information.

A handful of CUDA operations are nondeterministic if the CUDA version is10.2 or greater, unless the environment variableCUBLAS_WORKSPACE_CONFIG=:4096:8orCUBLAS_WORKSPACE_CONFIG=:16:8 is set. See the CUDA documentation for moredetails:https://docs.nvidia.com/cuda/cublas/index.html#results-reproducibilityIf one of these environment variable configurations is not set, aRuntimeErrorwill be raised from these operations when called with CUDA tensors:

• torch.mm()

• torch.mv()

• torch.bmm()

Note that deterministic operations tend to have worse performance thannondeterministic operations.

Note

This flag does not detect or prevent nondeterministic behavior causedby calling an inplace operation on a tensor with an internal memoryoverlap or by giving such a tensor as theout argument for anoperation. In these cases, multiple writes of different data may targeta single memory location, and the order of writes is not guaranteed.

Parameters

mode (bool) – If True, makes potentially nondeterministicoperations switch to a deterministic algorithm or throw a runtimeerror. If False, allows nondeterministic operations.

Keyword Arguments

warn_only (bool, optional) – If True, operations that do nothave a deterministic implementation will throw a warning instead ofan error. Default:False

Example:

>>>torch.use_deterministic_algorithms(True)# Forward mode nondeterministic error>>>torch.randn(10,device='cuda').kthvalue(1)...RuntimeError: kthvalue CUDA does not have a deterministic implementation...# Backward mode nondeterministic error>>>torch.nn.AvgPool3d(1)(torch.randn(3,4,5,6,requires_grad=True).cuda()).sum().backward()...RuntimeError: avg_pool3d_backward_cuda does not have a deterministic implementation...