Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

CI, BLD: Push NumPy's Emscripten/Pyodide wheels nightly to Anaconda.org PyPI index#26134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged

Conversation

@agriyakhetarpal
Copy link
Contributor

@agriyakhetarpalagriyakhetarpal commentedMar 25, 2024
edited
Loading

Description

This PR adds a schedule to push WASM wheels that are compiled via the Emscripten toolchain and Pyodide ecosystem to NumPy's PyPI-like index on Anaconda.org athttps://anaconda.org/scientific-python-nightly-wheels/numpy.

Key changes

  1. The addition of a schedule to the job, which is the same as the schedule thatwheels.yml runs on
  2. Aworkflow_dispatch: trigger has been added to push the wheels manually if needed
    a. The input is false by default to not accidentally upload and subsequently overwrite any existing wheels.
  3. A step has been added that runs after the tests run and succeed, which uses theNUMPY_NIGHTLY_UPLOAD_TOKEN repository secret. These uploads will not be attempted on forks or on workflow run contexts outside of the provided condition(s) in this newly added step.

@agriyakhetarpal
Copy link
ContributorAuthor

For more context to offer to reviewers, here:@rgommers suggested using thescientific-python/upload-nightly-action GitHub Action and to keep the workflow and the upload in a single place, to establish consistency with other repositories that are uploading (or will be uploading) these WASM wheels. This should be ready for review as-is, and it does not need to accessNUMPY_STAGING_UPLOAD_TOKEN or other repository secrets, since themultibuild-wheels-staging organisation is not being uploaded to at this time.

@lesteve
Copy link
Contributor

lesteve commentedMar 28, 2024
edited
Loading

Just curious, is there a way to actually conveniently use the uploaded wheels e.g. through%pip install inside JupyterLite (ormicropip.install inside Pyodide console)?

It seems like there is still the issue that CORS headers are still to be added to anaconda.org. I haven't tried in a while but this one week old commentpyodide/micropip#101 (comment) seems to say this is still the case. Related topyodide/pyodide#3049 (comment).

@agriyakhetarpal
Copy link
ContributorAuthor

It seems like there is still the issue that CORS headers are still to be added to anaconda.org. I haven't tried in a while but this one week old commentpyodide/micropip#101 (comment) seems to say this is still the case. Related topyodide/pyodide#3049 (comment).

Hi@lesteve – I have been in the works for adding these nightly wheels to PyWavelets (seePyWavelets/pywt#728) where I put up the linked request. But since there has been a response there from the Anaconda side, maybe the third time's the charm?

For alternatives, we had previously considered using thejupyterlite-xeus kernel instead ofjupyterlite-pyodide-kernel because the former has support forpre-installing packages before the documentation is built, but that right now requires setting up and maintaining a conda recipe where artifacts will then be downloaded off ofemscripten-forge. Doing that does not sound viable at this time, so maybexeus-kernel can include support for Python wheels sometime soon.

Just curious, is there a way to actually conveniently use the uploaded wheels e.g. through %pip install inside JupyterLite (or micropip.install inside Pyodide console)?

However, to answer this – you should—in theory—be able to use these nightly wheels by downloading them from GitHub Actions artifacts and lettingmicropip handle it from a local server?

@agriyakhetarpal
Copy link
ContributorAuthor

Converting this PR to a draft until there is a fix for#26164

@agriyakhetarpalagriyakhetarpal marked this pull request as draftMarch 29, 2024 19:18
This commit adds a schedule to push WASM wheelsthat are compiled via the Emscripten toolchain andPyodide ecosystem to NumPy's PyPI-like index onAnaconda.org. The key changes here, are:1. A schedule has been added to the job2. A workflow_dispatch trigger has been addedto push the wheels manually if needed3. A step has been added that runs after the testsrun and succeed, which uses a repository secretthat is NUMPY_NIGHTLY_UPLOAD_TOKEN. Theartifacts can then be found on this link:https://anaconda.org/scientific-python-nightly-wheels/numpyThe wheels uploads will not be attempted on forksor on workflow run contexts outside of the providedcondition(s) in the newly added step.[skip cirrus] [skip circle] [skip azp] [skip travis]
@agriyakhetarpal
Copy link
ContributorAuthor

agriyakhetarpal commentedMar 31, 2024
edited
Loading

Since there is a fix now, rebasing and marking the PR as ready for review again.

@agriyakhetarpalagriyakhetarpalforce-pushed theupload-emscripten-wheels-nightly branch from5709af0 toaf099a7CompareMarch 31, 2024 20:34
@agriyakhetarpalagriyakhetarpal marked this pull request as ready for reviewMarch 31, 2024 20:34
@lesteve
Copy link
Contributor

lesteve commentedApr 2, 2024
edited
Loading

Context : I was interested as well to have an easily installable Pyodide wheel for the scikit-learn documentation, one for the latest scikit-learn release (independently on Pyodide release cycles) one for the scikit-learn development version.

It would be nice if anaconda.org had CORS headers set-up, but the fact that it has not happened yet, may indicate that there are some complications. One of the nice thing with anaconda.org is that they havepypi.anaconda.org so that would allow to use the Pyodide wheel conveniently likemicropip.install('scikit-learn', index_urls=['https://pypi.anaconda.org/...')

As for the alternatives I have personally considered but left for later for now:

  • it seems like the Pyodide kernel can also ship Pyodide wheels at build-timehttps://jupyterlite.readthedocs.io/en/latest/howto/pyodide/wheels.html. The wheel would be part of the doc website, which adds a few MB but maybe OK enough
  • have the Pyodide wheel in another github repo so you can do things like this in a notebook. The URL is quite long which is not ideal:
    %pip install https://cdn.jsdelivr.net/gh/lesteve/scikit-learn-tests-pyodide@scikit-learn-wheel/dist/scikit_learn-1.2.dev0-cp310-cp310-emscripten_3_1_21_wasm32.whl
  • @rth did a quick-and-dirty CORS proxy with Cloudfare at one point. I guess if someone (maybe from Scientific Python?) wanted to maintain such a CORS proxy this could be an option?

@agriyakhetarpal
Copy link
ContributorAuthor

it seems like the Pyodide kernel can also ship Pyodide wheels at build-timejupyterlite.readthedocs.io/en/latest/howto/pyodide/wheels.html. The wheel would be part of the doc website, which adds a few MB but maybe OK enough

We were thinking of doing this, too – however, we did not get to a point where we could reliably download the wheel to be shipped at build time (most likely one would have to do that by overriding the build commands of the Read the Docs build inside thereadthedocs.yaml file). I imagine that doing so on a docs site that uses GitHub Pages would be a bit easier, though.

have the Pyodide wheel in another github repo so you can do things like this in a notebook. The URL is quite long which is not ideal:

Would this be easier to track with a GitHub Release? GitHub sets CORS headers, so tools likepip andpipx are generally able to download and install wheels from URLs like this, and a release URL would be a bit shorter if that helps. I do not have much experience with web dev, so I am not sure how setting up a CORS proxy goes :)

In the best case, yes, having a wheel come pre-installed/pre-loaded withjupyterlite-pyodide-kernel should solve those problems, but I cannot comment on how easy that would be, or where one would start with the implementation. If the Xeus kernel can do it, then I suppose the Pyodide kernel can, too.

[skip cirrus] [skip circle] [skip azp] [skip travis]
@agriyakhetarpalagriyakhetarpalforce-pushed theupload-emscripten-wheels-nightly branch frome728282 to31411afCompareApril 2, 2024 19:21
@charrischarris merged commite191a5f intonumpy:mainApr 3, 2024
@charris
Copy link
Member

Thanks@agriyakhetarpal .

agriyakhetarpal reacted with hooray emoji

@agriyakhetarpalagriyakhetarpal deleted the upload-emscripten-wheels-nightly branchApril 3, 2024 00:38
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@charrischarrischarris left review comments

Assignees

No one assigned

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@agriyakhetarpal@lesteve@charris@rgommers

[8]ページ先頭

©2009-2025 Movatter.jp