Thank you@peterjc123 for the implementation. Actually I am working on writing a Unity native plugin(c++) on Windows to infer neural net results every frame, and CPU only is indeed much slower without this feature. I tried to the plugin with CUDA libtorch, however, Unity crashes at the exact line that doing neural net inference (forward(input_tensor)). I wrote a simple C++ program to load the plugin lib and dll and it works totally well. I suppose that the crash related to some library conflict between which libraries Unity uses and libtorch's CUDA prebuilt library on Windows. I found similar painful experience of OpenPose's Unity Plugin dealing with Neural Net library + CUDA + Unity Plugin setting.

So I think I will give up the CUDA version. My question is that is this feature finished in your fork branch(peterjc123:cpu_fused_win)? I only care about libtorch, CPU only version, on Windows.

If you've already finished it, may I try to build it from your last commit( on Windows and CPU only).
I would appreciate if you could share the pre-built binary, but I could also try build from your source code. Could you also link some building instruction for building libtorch CPU only on Windows with Visual Studio + cmake? Is there any difference in terms of building libtorch CPU only on Windwos between your code and the master pytorch branch?

Thank you very much!

@Immocat No, it is still in an early stage. There are some difficulties that I have to tackle before it can be merged into master.

1. Finding VS installation (Plan: usingvswhere and setting env vars. The current method is to activate the dev env every time we call the compiler.)
2. Tempfile under Windows (Plan: Using code from GCC and make some adaptations)
3. Some other things that I haven't considered (e.g. Some util functions and OS-dependent code logic)

For cuda jit fusion conflicts, maybe you could try building the static version of LibTorch. Below are the steps:

cmd:: EssentialsetBUILD_SHARED_LIBS=OFF:: [Optional] If you want to build with VS 2019 generator, please change the value in the next line to `Visual Studio 16 2019`.:: Note: This value is useless if Ninja is detected. However, you can force that by using `set USE_NINJA=OFF`.setCMAKE_GENERATOR=Visual Studio152017:: Read the content in the previous section carefully before you preceed.:: [Optional] If you want to override the underlying toolset used by Ninja and Visual Studio with CUDA, please run the following script block.:: "Visual Studio 2017 Developer Command Prompt" will be run automatically.:: Make sure you have CMake >= 3.12 before you do this when you use the Visual Studio generator.:: It's an essential step if you use Python 3.5.setCMAKE_GENERATOR_TOOLSET_VERSION=14.11setDISTUTILS_USE_SDK=1for /f"usebackq tokens=*"%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -version [15^,16^) -products * -latest -property installationPath`) do call "%i\VC\Auxiliary\Build\vcvarsall.bat" x64 -vcvars_ver=%CMAKE_GENERATOR_TOOLSET_VERSION%:: [Optional] If you want to override the cuda host compilersetCUDAHOSTCXX=C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.11.25503\bin\HostX64\x64\cl.exepython tools\build_libtorch.py

The basic functionality is working now. However, there are still some points to improve:

• Currently, we activate the dev env every time if it's not activated. It is very slow, and we may need to move it to the python frontend.
• OpenMP is not working. It complains that "index variable in OpenMP 'for' statement must have signed integral type".
• We should skip the tests when VS is not installed.
• Implement something like-march=native.

This is nifty stuff. Let us know if there is stuff we can do to help move it along.

@ezyang Could you please tell me where the jit frontend is? That is, how can I disable it in the Python side?

Could you please tell me where the jit frontend is? That is, how can I disable it in the Python side?

Are you talking about the TorchScript compiler? It's not really disableable; when you request a function to be compiled for torchscript, we recursively collect the source code reachable from it and compile it. Maybe you could tell us more about what's going on?

cc@suo for perhaps more comments

@ezyang
Some more questions:

1. What about usingPYTORCH_JIT=0 in this line:https://github.com/pytorch/pytorch/blob/master/torch/jit/__init__.py#L57?
2. You called it the TorchScript compiler, so it implies that JIT fusion is not working when we dotorch.jit.trace, right?
3. Could you please tell me a bit more about whattorch.jit.trace is currently doing on Windows?

The following is what I want to do now. First, I want to add a check for the VS env before every jit fuse call. If it is not activated, then we will try to activate it, but if we cannot find it, then we will skip the fusion step. Do you know where should I add these code?

What about using PYTORCH_JIT=0 in this line:https://github.com/pytorch/pytorch/blob/master/torch/jit/__init__.py#L57?

Ah yes, I forgot about that. That will indeed turn off JIT globally; it's meant as an easy way to turn off script if you're debugging an issue without having to edit source code.

You called it the TorchScript compiler, so it implies that JIT fusion is not working when we do torch.jit.trace, right?

Actually, fusion can apply to trace too. Trace versus script refers to different ways of getting the IR in question; trace means we run your program and record what happened; script means we parse the literal program text. The IR can be fused in both cases.

Could you please tell me a bit more about what torch.jit.trace is currently doing on Windows?

I am not aware of any Windows specific behavior for torch.jit.trace, and we don't seem to have any macros on MSVC that would affect this.

First, I want to add a check for the VS env before every jit fuse call. If it is not activated, then we will try to activate it, but if we cannot find it, then we will skip the fusion step. Do you know where should I add these code?

For cpu, it's going to be somewhere liketorch/csrc/jit/fuser/cpu/fused_kernel.cpp, probablyrunCompiler.

@pytorchbot rebase this please

@xscha Sure, it should be fairly easy to support clang or any other compilers, but not in this PR. And we may need a code refactory otherwise the code will look messy. As for android, I think it should be just be the same with the deskop OS, using interpreter to run the operators when using jit script, and for jit fusion, only gcc is supported.

I'm just worried about the fact we need a compiler on deployed systems where we do inferencing (JIT is only for the inferencing right?).
Is there an alternative to ahead-of-time compile with options like -mavx -mavx2, etc?

@xsacha Yes, I agree with you that we should use some lightweight cross-platform compilers like llvmlite used by numba.

@pytorchbot rebase this please

@ezyang Could you please take some time to review this PR?

@ezyang merged this pull request in2ce8c83.