Download (bzlmod)

See also

For WORKSPACE instructions seehere.

To add PyPI dependencies to yourMODULE.bazel file, use thepip.parseextension and call it to create the central external repo and individual wheelexternal repos. Include the toolchain extension in theMODULE.bazel file as shownin the first bzlmod example above.

pip=use_extension("@rules_python//python/extensions:pip.bzl","pip")pip.parse(hub_name="my_deps",python_version="3.13",requirements_lock="//:requirements_lock_3_11.txt",)use_repo(pip,"my_deps")

For more documentation, see the Bzlmod examples under theexamples folder or the documentationfor the@rules_python//python/extensions:pip.bzl extension.

We are using a host-platform compatible toolchain by default to setup pip dependencies.During the setup phase, we create some symlinks, which may be inefficient on Windowsby default. In that case use the following.bazelrc options to improve performance ifyou have admin privileges:

startup --windows_enable_symlinks

This will enable symlinks on Windows and help with bootstrap performance of setting up thehermetic host python interpreter on this platform. Linux and OSX users should see nodifference.

Interpreter selection

Thepip.parsebzlmod extension by default uses the hermetic Python toolchain for the hostplatform, but you can customize the interpreter usingpip.parse.python_interpreter andpip.parse.python_interpreter_target.

You can use the pip extension multiple times. This configuration will createmultiple external repos that have no relation to one another and may result indownloading the same wheels numerous times.

As with any repository rule or extension, if you would like to ensure thatpip_parse isre-executed to pick up a non-hermetic change to your environment (e.g., updating your systempython interpreter), you can force it to re-execute by runningbazelsync--only[pip_parsename].

Requirements for a specific OS/Architecture

In some cases, you may need to use different requirements files for different OS and architecture combinations.This is enabled via therequirements_by_platform attribute in thepip.parse extension and thepip.parse tag class. The keys of the dictionary are labels to the file, and the values are alist of comma-separated target (os, arch) tuples.

For example:

# ...requirements_by_platform={"requirements_linux_x86_64.txt":"linux_x86_64","requirements_osx.txt":"osx_*","requirements_linux_exotic.txt":"linux_exotic","requirements_some_platforms.txt":"linux_aarch64,windows_*",},# For the list of standard platforms that the rules_python has toolchains for, default to# the following requirements file.requirements_lock="requirements_lock.txt",

In case of duplicate platforms,rules_python will raise an error, as there hasto be an unambiguous mapping of the requirement files to the (os, arch) tuples.

An alternative way is to use per-OS requirement attributes.

# ...requirements_windows="requirements_windows.txt",requirements_darwin="requirements_darwin.txt",# For the remaining platforms (which is basically only linux OS), use this file.requirements_lock="requirements_lock.txt",)

Note

If you are using a universal lock file but want to restrict the list of platforms thatthe lock file will be evaluated against, consider using the aforementionedrequirements_by_platform attribute and listing the platforms explicitly.

Multi-platform support

Historically, thepip_parse andpip.parse have only been downloading/buildingPython dependencies for the host platform that thebazel commands are executed on. Overthe years, people started needing support for building containers, and usually, that involvesfetching dependencies for a particular target platform that may be different from the hostplatform.

Multi-platform support for cross-building the wheels can be done in two ways:

  1. usingexperimental_index_url for thepip.parse bzlmod tag class

  2. using thepip.parse.download_only setting.

Warning

This will not work for sdists with C extensions, but pure Python sdists may still work using the firstapproach.

Usingdownload_only attribute

Let’s say you have two requirements files:

# requirements.linux_x86_64.txt--platform=manylinux_2_17_x86_64--python-version=39--implementation=cp--abi=cp39foo==0.0.1--hash=sha256:deadbeefbar==0.0.1--hash=sha256:deadb00f
# requirements.osx_aarch64.txt contents--platform=macosx_10_9_arm64--python-version=39--implementation=cp--abi=cp39foo==0.0.3--hash=sha256:deadbaaf

With these 2 files yourpip.parse could look like:

pip.parse(hub_name="pip",python_version="3.9",# Tell `pip` to ignore sdistsdownload_only=True,requirements_by_platform={"requirements.linux_x86_64.txt":"linux_x86_64","requirements.osx_aarch64.txt":"osx_aarch64",},)

With this,pip.parse will create a hub repository that is going tosupport only two platforms -cp39_osx_aarch64 andcp39_linux_x86_64 - and itwill only usewheels and ignore any sdists that it may find on the PyPI-compatible indexes.

Warning

Because bazel is not aware what exactly is downloaded, the same wheel may be downloadedmultiple times.

Note

This will only work for wheel-only setups, i.e., all of your dependencies need to have wheelsavailable on the PyPI index that you use.

CustomizingRequires-Dist resolution

In order to understand what dependencies to pull for a particular package,rules_python parses thewhl fileMETADATA.Packages can express dependencies viaRequires-Dist, and they can add conditions using“environment markers”, which represent the Python version, OS, etc.

While the PyPI integration provides reasonable defaults to support mostplatforms and environment markers, the values it uses can be customized in casemore esoteric configurations are needed.

To customize the values used, you need to do two things:

  1. Define a target that returnsEnvMarkerInfo

  2. Set the//python/config_settings:pip_env_marker_config flag tothe target defined in (1).

The keys and values should be compatible with thePyPA dependency specifiersspecification.This is not strictly enforced, however, so you can return a subset of keys oradditional keys, which become available during dependency evaluation.

Bazel downloader and multi-platform wheel hub repository.

Warning

This is currently still experimental, and whilst it has been proven to work in quite a fewenvironments, the APIs are still being finalized, and there may be changes to the APIs for thisfeature without much notice.

The issues that you can subscribe to for updates are:

Thepip extension supports pulling information fromPyPI (or a compatible mirror), and itwill ensure that thebazel downloader is used for downloading the wheels.

This provides the following benefits:

  • Integration with thecredential_helper to authenticate with privatemirrors.

  • Cache the downloaded wheels speeding up the consecutive re-initialization of the repositories.

  • Reuse the same instance of the wheel for multiple target platforms.

  • Allow using transitions and targeting free-threaded and musl platforms more easily.

  • Avoidspip for wheel fetching and results in much faster dependency fetching.

To enable the feature specifypip.parse.experimental_index_url as shown intheexamples/bzlmod/MODULE.bazel example.

Similar touv, one can override theindex that is used for a single package. By default, we first search in the index specified bypip.parse.experimental_index_url, then we iterate through thepip.parse.experimental_extra_index_urls unless there are overrides specified viapip.parse.experimental_index_url_overrides.

When using this feature during thepip extension evaluation you will see the accessed indexes similar to below:

Loading: 0 packages loaded    Fetching module extension @@//python/extensions:pip.bzl%pip; Fetch package lists from PyPI index    Fetching https://pypi.org/simple/jinja2/

This does not mean thatrules_python is fetching the wheels eagerly; rather,it means that it is calling the PyPI server to get the Simple API responseto get the list of all available source and wheel distributions. Once it hasgotten all of the available distributions, it will select the right ones dependingon thesha256 values in yourrequirements_lock.txt file. Ifsha256 hashesare not present in the requirements file, we will fall back to matching by versionspecified in the lock file.

Fetching the distribution information from the PyPI allowsrules_python toknow whichwhl should be used on which target platform and it will determinethat by parsing thewhl filename based onPEP600,PEP656 standards. Thisallows the user to configure the behaviour by using the following publiclyavailable flags:

Credential Helper

TheBazel downloader usage allows for the BazelCredential Helper.Your Python artifact registry may provide a credential helper for you.Refer to your index’s docs to see if one is provided.

The simplest form of a credential helper is a bash script that accepts an argument and spits out JSON tostdout. For a service like Google Artifact Registry that uses‘Basic’ HTTP Auth and doesnot provide a credential helper that conforms to thespec, the script mightlook like:

#!/bin/bash# cred_helper.shARG=$1# but we don't do anything with it as it's always "get"# formatting is optionalecho'{'echo'  "headers": {'echo'    "Authorization": ["Basic dGVzdDoxMjPCow=="]'echo'  }'echo'}'

Configure Bazel to use this credential helper for your Python indexexample.com:

# .bazelrcbuild--credential_helper=example.com=/full/path/to/cred_helper.sh

Bazel will call this file likecred_helper.shget and use the returned JSON to inject headersinto whatever HTTP(S) request it performs againstexample.com.

See theCredential Helper Spec for more details.