Good question@oscarbenjamin. A few initial thoughts:

• I think we'd like to support this use case and want it to work. I'm thinking about doing this for SciPy too at some point, to avoid the pain of "can't find a BLAS library".
• Building a shared library and then using it from a Python extension module is supposed to work on Linux and macOS as of today. Not on Windows yet, see the feature request atLinking against libraries from meson on Windows #265.
  • However, the one test case we have (seetests/packages/link-against-local-lib/meson.build) useslink_with, which is not quite right for this use case.
  • We need a new test case usingsubproject, and then using it asdependency(...) rather than an explicitlink_with.
• What is then supposed to happen is that building a wheel completes. However, vendoring external dependencies into a wheel is not the job ofmeson-python but rather ofauditwheel/delocate/delvewheel.

Your test package looks quite good to me, the way it handles GMP as dependency seems right. The question is how to fix it - there's probably a better way than your patch above.

The reason why your patched meson-python produces a wheel that works is because either the Python extension module is compiled statically against GMP or because the Python extension is linked dynamically and you have GMP installed on the system where you are running it.

If you don't need GMP bundled with the Python wheel (if you can assume a compatible version of GMP is installed on the target system, or if you link statically to it), the way to make your current solution work is to do not install the GMP subproject. I'm not so sure how to do this in practice when you are dealing with anexternal_project subproject.

Looking a the source of the error reported by meson-python, this is the install manifest for the project:

$ meson introspect build --install-plan -i{    "targets": {        ".../mesontest/build/meson_test/_meson_test.cpython-311-darwin.so": {            "destination": "{py_platlib}/meson_test/_meson_test.cpython-311-darwin.so",            "tag": "runtime"        }    },    "python": {        ".../mesontest/meson_test/__init__.py": {            "destination": "{py_platlib}/meson_test/__init__.py",            "tag": "python-runtime"        }    },    "install_subdirs": {        ".../mesontest/build/subprojects/gmp-6.2.1/dist/usr/local": {            "destination": "{prefix}/.",            "tag": null,            "exclude_dirs": [],            "exclude_files": []        }    }}

meson-python does not know where to place the files that are copied into the generic{prefix} directory. Fixing this for projects that install only a shared library requires some assumptions, but is not too hard. However, something would need to be done about all other files installed by the subproject but that do not belong in the Python wheel. Even in the case of a simple (in terms of deployment) library like GMP, there are quite a few:

build/subprojects/gmp-6.2.1/dist/└── usr    └── local        ├── include        │   └── gmp.h        ├── lib        │   ├── libgmp.10.dylib        │   ├── libgmp.dylib -> libgmp.10.dylib        │   ├── libgmp.la        │   └── pkgconfig        │       └── gmp.pc        └── share            └── info                ├── dir                ├── gmp.info                ├── gmp.info-1                └── gmp.info-2

At the moment I don't see how a solution could look like.

You need to decide is whether you want to includelibgmp.so.10 in the Python wheel or require the users of your package to provide a system installedlibgmp.so.10.

The problem is I want different things in different situations:

• Building wheels for PyPI. This will take place in CI using cibuildwheel. There I will build GMP but I will do it in theCIBW_BEFORE_ALL stage and subsequently auditwheel, delocate, delvewheel will bundle the shared library with name mangling as needed. Since I will build GMP before building the wheels the situation is effectively like having a system installed GMP. However if I'm using meson/meson-python as my build system then I would ideally like to use those to manage doing this building and I don't see yet how to do that.
• The project will also be packaged for conda and linux distros etc and they will definitely want to link with the system installed dependencies.
• Some users will need to build from source for whatever reason and will start from the sdist (perhaps implicitly e.g. via pip). In this case I would like to use the system provided dependencies by default but I would also like to make it so that GMP can be built if needed. These users are not being given a wheel but rather the sdist that is generated by mesonpy and uses mesonpy as its build-system. That sdist potentially needs to be able to build a wheel that could include GMP so that it can be installed locally. Ideally the sdist contains the information needed to build a compatible version of GMP in a way that is compatible with the extension module(s) in the project.
• For development it really needs to be possible to build the main dependencies in a way that is isolated from the system installed libraries. I want to build and install them local to the venv/project somehow and I want to be able to switch to different versions of the dependencies e.g. to test new releases.

Potentially the answer to all of these is that is not really for mesonpy to build/bundle the library in the wheel and I am just not quite using it in the expected way. Perhaps I can open the question up a bit:

• If I am using meson and meson-python as the build system how can I do all of the things that are not just about building a wheel against system installed libs?
• How do I make it easy for a contributor to build the dependencies locally in a development setup?
• How do I provide configuration for someone building from source to be able to build the dependencies in the correct way?
• How can I have an editable install?
• How do I scope the versions/builds of these dependencies to a venv rather than installing them globally?

Looking at meson + meson-python made me wonder if it can somehow help with some of these things. The approach that I went for here seemed to give reasonable answers to some of the questions above because it means I can manage dependencies in the meson build configuration, users can have them be built if needed, the dependencies are contained to a venv etc.

Linking statically also answers those questions in the same way but has the downside that if I have many extension modules depending on GMP (and other C libraries) then I will end up duplicating the GMP etc library code across those many extension modules.

Having looked into this in more detail I don't think that the approach that I was attempting to follow of building and bundling external dependency shared libraries into the wheel with mesonpy is workable. At least I think that significant changes in mesonpy would be needed to make this work.

Currently the expectation is that building uses a PEP 517 build frontend likepython -m build orpip install .... The frontend creates an isolated environment and installs the Python build dependencies before invoking the build backend which is in mesonpy. The expectation in this setup is that any non-Python dependencies are installed "system-wide" before even the PEP 517 frontend is invoked.

When we want to ship wheels that bundle the dependencies we arrange to have the non-Python dependencies installed system-wide first, invoke the PEP 517 build and then repair afterwards. The repair step bundles the shared libraries but also performs the necessary name mangling or RPATH modifications so that the bundled libraries can be used without conflicting with system libs.

Even if the user does not have GMP installed when the wheel is built it is possible that they would install GMP later. It is not appropriate for mesonpy to install the libraries system-wide. It is also not enough just to bundle the shared library files without also taking steps to ensure that those libraries and the extension modules linking to them are isolated from system libs. If we want mesonpy to build the non-Python dependencies then we need it to include part of what the repair step is doing so it needs to perform a subset of what auditwheel/delocate/delvewheel currently do.

This seems like a change of scope for mesonpy and its role as a PEP 517 backend that is invoked by commands likepip install python-flint. Maybe that change of scope is reasonable but I don't think that I can make it work for python-flint without the support being added to mesonpy first.

What could work is if this was all using static linking but I think that is not a good fit for python-flint which has many extension modules all linking to the fairly largelibflint.so.

I would like to use meson to manage building the dependencies in a development checkout and also as an option for users who want to setup the dependencies easily when building from source. I think though that what we need for that is a separate meson build configuration like having a subdirectorydependencies with a separatedependencies/meson.build file. This build config would not be used by mesonpy and the PEP 517 build but could be run as a separate step beforehand (likeCIBW_BEFORE_ALL):

cd dependenciesmeson setup --flint-ver=3.1.0 --prefix=$(pwd)/../.localmeson installcd ..PKG_CONFIG_PATH=$(pwd)/.local pip install --no-build-isolation --editable .

Then perhaps if usingspin as a development frontend we could provide a coherent interface for managing this local dependency stack when installing/building python-flint or running the tests etc.

I am revisiting this after a discussion with someone trying to install python-flint (flintlib/python-flint#193).

A user wants to install a newer version of python-flint but we don't provide wheels for their platform (Linux aarch64 because GitHub Actions does not support this yet). They have previously built from source fine though and would be happy to do so again.

However they have the wrong version of Flint installed. Their Flint installation is part of a bundle of other software that uses an older version of Flint. They can't just install a newer version of Flint system-wide because it would break all of the other software.

We handle this situation nicely with the binary wheels on PyPI because auditwheel mangles and bundleslibflint.so into the wheel. We have all of the code needed to be able to build Flint for the user and the build would succeed on their system but meson-python does not have any facility to mangle and bundle the binary into a wheel.

I said in my last comment that it seems like meson-python should possibly not be expected to do this building and bundling of non-Python libraries. However I should amend that statement at least to say that it is perhaps not a good idea to do thatby default. At the very least it would be good to be able to have an option like:

#Build and bundle Flint 3.1 into the Python installation:pip install --config-settings=setup-args=-Dbuild_flint==3.1 python-flint

I think that this can't work right now though because meson-python does not provide a facility to mangle and bundle the shared libraries but it is clear to me now that this would be a useful thing to be able to do.

Hi, has there been any progress on this?

I have a similar use case.

I have a subproject (mylib), which always builds a shared library dependency (mylib_dep). I want to link my Python extension against it and include it in the wheel.

Example:

python=import('python').find_installation(pure:false)mylib_proj=subproject('mylib', ...)mylib_dep= mylib_proj.get_variable('mylib_dep')# Shared mylib lib dependencypython.extension_module(...,dependency: mylib_dep,install:true,subdir:'myModule')

The subproject's call tolibrary(), which produces themylib_lib variable, does setinstall: true.mylib_lib is then used to produce themylib_dep variable viadeclare_dependency(). Then, as seen above,mylib_deb is extracted from the subproject and used in the current project.

However, when I build the wheel only the Python extension.so is present. How can I include the shared library from the subproject?