Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Splitting a large Meson project into multiple pyprojects#685

Unanswered
tobiasdiez asked this question inQ&A
Discussion options

At Sagemath, we have the following wish-list for our Meson build:

  • Have one monolithic project that compiles all Cython files and includes all Python files. This is mostly used for local development.
  • Have one "standard" project (shipped to pypi) that includes almost all of the monolithic project, except for certain addons or extensions.
  • Each of these addons should be shipped to pypi as a separate project.

One important aspect is that these addon's have additional external dependencies that are not (necessarily) shared by the "standard" installation.

I wanted to ask if someone has experience with a similar setup and can recommend an approach for implementing it?
The following options come to mind:

  • Extract the addons to Meson subprojects that manage their own dependencies and installation instructions. The monolytic and standard project then include these subjects based on a "toggle".
  • Usetags to mark the files belonging to a certain addon, and then toggle their installation by providing the correct--tags argument. I don't know though if it is possible to check for the presence of a dependency conditional on the provided install tag.
  • Introduce a boolean variable for each addon, and then guard the installation of the relevant Cython/Python files using if-statements.
You must be logged in to vote

Replies: 1 comment 4 replies

Comment options

Interesting question - and great to see Sagemath working on a Meson build!

I don't know though if it is possible to check for the presence of a dependency conditional on the provided install tag.

It isn't. Install tags are purely an install-time feature; all targets gets built even if they don't get installed. It works well for something like "don't include the test suite in the wheel", but not for conditional dependency detection or other conditional build steps.

Extract the addons to Meson subprojects that manage their own dependencies and installation instructions.

Subprojects are designed for external dependencies; my gut feel here is that they can be made to work for your use case, but aren't the best choice given that your project is in a single large repo and the addons you're trying to enable/disable aren't really external.

Introduce a boolean variable for each addon, and then guard the installation of the relevant Cython/Python files using if-statements.

This seems like the way to go. Important question: are these addons each in their own subdirectory? If so, then I think this shouldn't be hard: move thedependency() calls that you now all have in the top-levelmeson.build to themeson.build file in the addons subdirectory, and then based on the boolean variable usesubdir('addon1') one level up to include it in the full build yes or no. This will then also solve the problem that for publishing the addon as a separate package, you need a separatepyproject.toml file (or amend the existing one) with a new project name, description, license, etc.

You must be logged in to vote
4 replies
@tobiasdiez
Comment options

Thanks a lot! These are very valuable insights.

These addons are mostly in their own subfolders, but not always (we might or might not want to change this - the jury is not yet out on this). There are also ideas floating in the room to extract at least some of the addons to completely independent repositories. I have the feeling that subprojects would serve as a temporary middle way where the addons are still in the same repo but a bit more independent from the main code. Does this change your gutt feeling about using subprojects for this purpose or do you still prefer the boolean-variable approach?

This will then also solve the problem that for publishing the addon as a separate package, you need a separate pyproject.toml file

Just to make sure that I parse this sentence correctly: one still needs to have a separate pyproject.toml + main meson.build file for each addon - just that the latter is really simple and sets only the project metadata and then includes the meson build file of the addon, right?

@rgommers
Comment options

Does this change your gutt feeling about using subprojects for this purpose or do you still prefer the boolean-variable approach?

Not sure without trying. Both methods are easy to evolve into moving the add-on to a separate repo. And for the add-ons that aren't completely isolated, I think subprojects aren't an option. One advantage of subprojects is that it's slightly cleaner to have a separateproject() call (see code outline below). I'd probably still try the boolean flag option first, but the differences may be minor/subtle.

one still needs to have a separate pyproject.toml + main meson.build file for each addon - just that the latter is really simple and sets only the project metadata and then includes the meson build file of the addon, right?

I think yes, but now I am unsure of parsing that sentence, so let me sketch it out. The repo will look like:

# top-level of the repomeson.build       # contains a project() call for the main projectmeson.options   # defines the `addon1_standalone` option (default: false)pyproject.toml   # contains all metadata for the main project    src/        meson.build     # contains `if use_addon1; subdir(addon1)`         addon1/            meson.build      # contains `dependency(my-dep-for-addon1)`            meson.options   # also defines the `addon1_standalone` option (default: true)            pyproject.toml  # contains all metadata for addon1

Thesrc/addon1/meson.build should have a separateproject() call to ensure that thatmeson.build file can be used standalone:

ifget_option('addon1_standalone')# build option, defined in meson.optionsproject('addon1','c','cython'  )# add meson_version, default_options, etc. as neededendif

Thesrc/addon1/pyproject.toml file should contain that same flag[build-system] and[project] tables with metadata foraddon1.

Then to build your wheels:

# from top-level directorypython -m build --wheel  # build wheel for main projectcd src/addon1python -m build --wheel  # build wheel for addon1
@tobiasdiez
Comment options

Ah, this is an interesting idea. But sadly it doesn't seem to work. With theif get_option(...) aroundproject, Meson complains thatproject is not the first statement in the meson file if I compile the addon separately. But without it the compilation of the main project fails because of a "Second call to project()". Any idea how to workaround this limitation? Eg, is it possible to have a main meson build file that is not calledmeson.build?

I did a bit more research and it seems people recommend subprojects in this case (e.g.mesonbuild/meson#6404 andmesonbuild/meson#9929). An alternative approach would be to have an<root>/addons folder that only contains the pyproject.toml + main meson build file for each addon, and a symlink to the main addon source code.

@rgommers
Comment options

Ahh that's too bad. The table inmeson#6406-comment summarizes it well:

image

There will be some limitations though I expect, for the "have one monolithic project for local development" use case. For one, the add-on has to be a namespace package for it to show up in-tree in the monolithic mode (but that's okay still). More importantly, I suspect editable installs for the monolithic mode won't work for the add-ons in subprojects.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Category
Q&A
Labels
None yet
2 participants
@tobiasdiez@rgommers

[8]ページ先頭

©2009-2025 Movatter.jp