After studying#439,#2228,#2743,#6737 and the new FAQ about reproducibility, I was able to get consistent, reproducible result on my development machines using Theano. If I run my code twice, I get the exact same results.

The problem is that the results are reproducible only on thesame machine. In other words, if I

• Train a model on machine A
• Evaluate the model usingpredict
• Save the model (usingsave_model, ormodel_to_json andsave_weights)
• Transfer the model to machine B and load it
• Evaluate again the model on machine B usingpredict

The results of the twopredicts are different. Using CPU or GPU makes no difference - after I copy the model file(s) from a machine to another, the performance ofpredict changes dramatically.

The only difference on the two machines is the hardware (I use my laptop's 980M and a workstation with a Titan X Pascal) and the NVIDIA driver version, which is slightly older on the workstation. Both computers run Ubuntu 16.04 LTS and Cuda 8 with cuDNN. All libraries are on the same version on both machines, and the Python version is the same as well (3.6.1).

Is this behavior intended? I expect that running a pre-trained model on with the same architecture and weights on two different machines yields the same results, but this doesn't seem the case.

On a side note, a suggestion: on the FAQs about reproducibility, it should be explicitly stated that the development version of Theano is needed to get reproducible results.