Autoloading Out-of-Tree Extension

Created On: Oct 10, 2024 | Last Updated: Oct 10, 2024 | Last Verified: Oct 10, 2024

Author:Yuanhao Ji

The extension autoloading mechanism enables PyTorch to automaticallyload out-of-tree backend extensions without explicit import statements. Thisfeature is beneficial for users as it enhances theirexperience and enables them to follow the familiar PyTorch deviceprogramming model without having to explicitly load or import device-specificextensions. Additionally, it facilitates effortlessadoption of existing PyTorch applications with zero-code changes onout-of-tree devices. For further details, refer to the[RFC] Autoload Device Extension.

What you will learn

  • How to use out-of-tree extension autoloading in PyTorch

  • Review examples with Intel Gaudi HPU, Huawei Ascend NPU

Prerequisites

  • PyTorch v2.5 or later

Note

This feature is enabled by default and can be disabled by usingexportTORCH_DEVICE_BACKEND_AUTOLOAD=0.If you get an error like this: “Failed to load the backend extension”,this error is independent with PyTorch, you should disable this featureand ask the out-of-tree extension maintainer for help.

How to apply this mechanism to out-of-tree extensions?

For instance, suppose you have a backend namedfoo and a corresponding package namedtorch_foo. Ensure thatyour package is compatible with PyTorch 2.5 or later and includes the following snippet in its__init__.py file:

def_autoload():print("Check things are working with `torch.foo.is_available()`.")

Then, the only thing you need to do is define an entry point within your Python package:

setup(name="torch_foo",version="1.0",entry_points={"torch.backends":["torch_foo = torch_foo:_autoload",],})

Now you can import thetorch_foo module by simply adding theimporttorch statement without the need to addimporttorch_foo:

>>>importtorchCheck things are working with `torch.foo.is_available()`.>>>torch.foo.is_available()True

In some cases, you might encounter issues with circular imports. The examples below demonstrate how you can address them.

Examples

In this example, we will be using Intel Gaudi HPU and Huawei Ascend NPU to determine how tointegrate your out-of-tree extension with PyTorch using the autoloading feature.

habana_frameworks.torch is a Python package that enables users to runPyTorch programs on Intel Gaudi by using the PyTorchHPU device key.

habana_frameworks.torch is a submodule ofhabana_frameworks, we add an entry point to__autoload() inhabana_frameworks/setup.py:

setup(   name="habana_frameworks",   version="2.5",+   entry_points={+       'torch.backends': [+           "device_backend = habana_frameworks:__autoload",+       ],+   })

Inhabana_frameworks/init.py, we use a global variable to track if our module has been loaded:

importosis_loaded=False# A member variable of habana_frameworks module to track if our module has been importeddef__autoload():# This is an entrypoint for pytorch autoload mechanism# If the following condition is true, that means our backend has already been loaded, either explicitly# or by the autoload mechanism and importing it again should be skipped to avoid circular importsglobalis_loadedifis_loaded:returnimporthabana_frameworks.torch

Inhabana_frameworks/torch/init.py, we prevent circular imports by updating the state of the global variable:

importos# This is to prevent torch autoload mechanism from causing circular importsimporthabana_frameworkshabana_frameworks.is_loaded=True

torch_npu enables users to run PyTorch programs on Huawei Ascend NPU, itleverages thePrivateUse1 device key and exposes the device nameasnpu to the end users.

We define an entry point intorch_npu/setup.py:

setup(   name="torch_npu",   version="2.5",+   entry_points={+       'torch.backends': [+           'torch_npu = torch_npu:_autoload',+       ],+   })

Unlikehabana_frameworks,torch_npu uses the environment variableTORCH_DEVICE_BACKEND_AUTOLOADto control the autoloading process. For example, we set it to0 to disable autoloading to prevent circular imports:

# Disable autoloading before running 'import torch'os.environ['TORCH_DEVICE_BACKEND_AUTOLOAD']='0'importtorch

How it works

Autoloading implementation

Autoloading is implemented based on Python’sEntrypointsmechanism. We discover and load all of the specific entry pointsintorch/__init__.py that are defined by out-of-tree extensions.

As shown above, after installingtorch_foo, your Python module can be importedwhen loading the entrypoint that you have defined, and then you can do some necessary work whencalling it.

See the implementation in this pull request:[RFC] Add support for device extension autoloading.

Conclusion

In this tutorial, we learned about the out-of-tree extension autoloading mechanism in PyTorch, which automaticallyloads backend extensions eliminating the need to add additional import statements. We also learned how to applythis mechanism to out-of-tree extensions by defining an entry point and how to prevent circular imports.We also reviewed an example on how to use the autoloading mechanism with Intel Gaudi HPU and Huawei Ascend NPU.