Reproducibility

Completely reproducible results are not guaranteed across PyTorch releases,individual commits or different platforms. Furthermore, results need not bereproducible between CPU and GPU executions, even when using identical seeds.

However, in order to make computations deterministic on your specific problem onone specific platform and PyTorch release, there are a couple of steps to take.

There are two pseudorandom number generators involved in PyTorch, which you willneed to seed manually to make runs reproducible. Furthermore, you should ensurethat all other libraries your code relies on and which use random numbers alsouse a fixed seed.

PyTorch

You can usetorch.manual_seed() to seed the RNG for all devices (bothCPU and CUDA):

importtorchtorch.manual_seed(0)

There are some PyTorch functions that use CUDA functions that can be a sourceof non-determinism. One class of such CUDA functions are atomic operations,in particularatomicAdd, where the order of parallel additions to thesame value is undetermined and, for floating-point variables, a source ofvariance in the result. PyTorch functions that useatomicAdd in the forwardincludetorch.Tensor.index_add_(),torch.Tensor.scatter_add_(),torch.bincount().

A number of operations have backwards that useatomicAdd, in particulartorch.nn.functional.embedding_bag(),torch.nn.functional.ctc_loss() and many forms of pooling, padding, and sampling.There currently is no simple way of avoiding non-determinism in these functions.

CuDNN

When running on the CuDNN backend, two further options must be set:

torch.backends.cudnn.deterministic=Truetorch.backends.cudnn.benchmark=False

Warning

Deterministic mode can have a performance impact, depending on your model. This means that due to the deterministic nature of the model, the processing speed (i.e. processed batch items per second) can be lower than when the model is non-deterministic.

Numpy

If you or any of the libraries you are using rely on Numpy, you should seed theNumpy RNG as well. This can be done with:

importnumpyasnpnp.random.seed(0)