Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-95299: Stop installing setuptools as a part of ensurepip and venv#101039

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
pradyunsg merged 21 commits intopython:mainfrompradyunsg:remove-setuptools-ensurepip
Apr 18, 2023

Conversation

pradyunsg
Copy link
Member

@pradyunsgpradyunsg commentedJan 14, 2023
edited
Loading

This PR removes the bundled setuptools wheel from ensurepip, and stops installing setuptools in environments created by venv.


I based this PR off of my understanding of venv and ensurepip, and using the following search to locate all the relevant bits of code + ensuring that tests are happy locally.

Screenshot 2023-01-14 at 17 20 01

In this PR, I've intentionally not touched the example inhttps://docs.python.org/3.12/library/venv.html#an-example-of-extending-envbuilder. That example is largely outdated and is likely better served being updated in a dedicated PR for it.

AlexWaygood, andriyor, Avasam, groodt, SpecLad, and edmorley reacted with hooray emoji
@pradyunsgpradyunsg added stdlibPython modules in the Lib dir 3.12only security fixes labelsJan 14, 2023
@pradyunsg
Copy link
MemberAuthor

pradyunsg commentedJan 14, 2023
edited
Loading

Hmm...@vstinner@encukou any opinions on what I should do withtest_cppext (added in#91321) here?

That test has an implicit assumption thatsetuptools will be present in the virtual environment created byvenv (added in#92639, as part of removing reliance on distutils).

Notably, some leading questions that affect what direction I could take for updating that test:

  • Does this test need to use packaging tooling? (otherwise, I guess we can hand-craft building the extension and rely on that)
  • If itshould use setuptools, would it be OK for this test to require reaching out over the network to fetch setuptools via pip?
  • Would it be OK to build the extension in pip-installable package, rather thanusing the deprecated approach of directlyinvoking setup.py?

@pradyunsgpradyunsgforce-pushed theremove-setuptools-ensurepip branch from17a121e to32b3cccCompareJanuary 14, 2023 22:37
@pradyunsg
Copy link
MemberAuthor

pradyunsg commentedJan 14, 2023
edited
Loading

Since it's cheap to revert commits/remove them from a PR, and nice to have a code change to discuss details over, I've gone ahead and pushed a change that assumes that the answer to the questions I raised above is yes, yes and yes. :)

Mentioning a GitHub UI thing that I find very useful (and non-initutive): you can click "force-pushed" on the event above to see what changed in the force pushes between the hashes that GitHub lists in the from ... to ... at the end of the event.

@pradyunsgpradyunsgforce-pushed theremove-setuptools-ensurepip branch from32b3ccc to6939d48CompareJanuary 14, 2023 22:42
Copy link
Member

@vsajipvsajip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The changes seem reasonable to me, though I'm only the code owner for the venv and test_venv bits.

pradyunsg reacted with heart emoji
@encukou
Copy link
Member

I guess we can hand-craft building the extension and rely on that

IMO, that would be best. Relatedly, I've been thinking how to handledistutils removal inthe extension tutorial -- it should definitely mention setuptools, but a simple but working example command line, because setuptools is not always available.
Having that in the tests -- even in a way that only satisfies our CI on select platforms -- would help :)
CPython tests depending on setuptools isn't great in general, and neither is downloading code as part of the test run.

Anyway, if that sounds like more than you signed up for, go ahead with Setuptools.
I don't have time for a full PR review now, sorry! Whoever gets around to it, pleasetest-with-buildbots before merging.

zware reacted with thumbs up emojipradyunsg and Avasam reacted with heart emoji

@pradyunsg
Copy link
MemberAuthor

Anyway, if that sounds like more than you signed up for, go ahead with Setuptools.

I spent some time today looking into this after my work day, and I've not figured out the details in a portable manner. I'd prefer if this could be tackled in a follow up instead. :)

@pradyunsg
Copy link
MemberAuthor

/cc@dstufft@ncoghlan as other experts on ensurepip.

I'mtechnically listed as an expert on ensurepip already and I'm ~sure that the changes I've made to it are correct; but it certainly won't hurt if y'all can review this. :)

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by@pradyunsg for commit7457fd3 🤖

If you want to schedule another build, you need to add the:hammer: test-with-buildbots label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-buildbotsTest PR w/ buildbots; report in status section labelJan 31, 2023
@pfmoore
Copy link
Member

I added a review comment that should fix the buildbot errors.

@hugovk
Copy link
Member

Should this also go in What's New, underRemoved?

pradyunsg reacted with thumbs up emoji

@pfmoore
Copy link
Member

Should this also go in What's New, underRemoved?

The inclusion of setuptools was always explicitly documented as an internal implementation detail, and as such, I don't think we have any obligation to report its removal. It could be mentioned in the same way that we might document the removal of an internal API - I did a quick check and the 3.11 What's New included "Removed the undocumented privatefloat.__set_format__() method". So I guess there's a precedent.

@encukou
Copy link
Member

Please add it as a convenience to users, even if there's technically no obligation. It's beenvery easy to end up depending on setuptools accidentally.

IMO, for removals/breaks, it's best to write What's New entries for the target audience of a poor soul who isn't too well-versed in Python but inherited some complex system with a “temporary” hack. Point them to a solution, don't apologize, but don't berate them either.
With that in mind, please mentionpkg_resources, as that shows up in ImportErrors but the connection to setuptools may not be clear.

(cc @python/proofreaders -- hope the above is good general advice)

AlexWaygood, pradyunsg, pfmoore, CAM-Gerlach, hroncok, JelleZijlstra, and FFY00 reacted with thumbs up emoji

@pradyunsg
Copy link
MemberAuthor

Thanks all! Holler on the issue, if there's any concerns! :)

carljm added a commit to carljm/cpython that referenced this pull requestApr 20, 2023
* main: (24 commits)pythongh-98040: Move the Single-Phase Init Tests Out of test_imp (pythongh-102561)pythongh-83861: Fix datetime.astimezone() method (pythonGH-101545)pythongh-102856: Clean some of the PEP 701 tokenizer implementation (python#103634)pythongh-102856: Skip test_mismatched_parens in WASI builds (python#103633)pythongh-102856: Initial implementation of PEP 701 (python#102855)pythongh-103583: Add ref. dependency between multibytecodec modules (python#103589)pythongh-83004: Harden msvcrt further (python#103420)pythonGH-88342: clarify that `asyncio.as_completed` accepts generators yielding tasks (python#103626)pythongh-102778: IDLE - make sys.last_exc available in Shell after traceback (python#103314)pythongh-103582: Remove last references to `argparse.REMAINDER` from docs (python#103586)pythongh-103583: Always pass multibyte codec structs as const (python#103588)pythongh-103617: Fix compiler warning in _iomodule.c (python#103618)pythongh-103596: [Enum] do not shadow mixed-in methods/attributes (pythonGH-103600)pythonGH-100530: Change the error message for non-class class patterns (pythonGH-103576)pythongh-95299: Remove lingering setuptools reference in installer scripts (pythonGH-103613)  [Doc] Fix a typo in optparse.rst (python#103504)pythongh-101100: Fix broken reference `__format__` in `string.rst` (python#103531)pythongh-95299: Stop installing setuptools as a part of ensurepip and venv (python#101039)pythonGH-103484: Docs: add linkcheck allowed redirects entries for most cases (python#103569)pythongh-67230: update whatsnew note for csv changes (python#103598)  ...
webknjaz added a commit to webknjaz/cpython that referenced this pull requestSep 10, 2023
This is a refactoring change that aims to simplify ``ensurepip``.Before it, this module had legacy infrastructure that made anassumption that ``ensurepip`` would be provisioning more then just asingle wheel. That assumption is no longer true since [[1]][[2]][[3]].In this change, the improvement is done around removing unnecessaryloops and supporting structures to change the assumptions to expectonly the bundled or replacement ``pip`` wheel.[1]:python@ece20db[2]:python#101039[2]:python#95299
webknjaz added a commit to webknjaz/cpython that referenced this pull requestSep 10, 2023
This is a refactoring change that aims to simplify ``ensurepip``.Before it, this module had legacy infrastructure that made anassumption that ``ensurepip`` would be provisioning more then just asingle wheel. That assumption is no longer true since [[1]][[2]][[3]].In this change, the improvement is done around removing unnecessaryloops and supporting structures to change the assumptions to expectonly the bundled or replacement ``pip`` wheel.[1]:python@ece20db[2]:python#101039[3]:python#95299
webknjaz added a commit to webknjaz/cpython that referenced this pull requestSep 10, 2023
This is a refactoring change that aims to simplify ``ensurepip``.Before it, this module had legacy infrastructure that made anassumption that ``ensurepip`` would be provisioning more then just asingle wheel. That assumption is no longer true since [[1]][[2]][[3]].In this change, the improvement is done around removing unnecessaryloops and supporting structures to change the assumptions to expectonly the bundled or replacement ``pip`` wheel.[1]:python@ece20db[2]:python#101039[3]:python#95299
webknjaz added a commit to webknjaz/cpython that referenced this pull requestSep 10, 2023
This is a refactoring change that aims to simplify ``ensurepip``.Before it, this module had legacy infrastructure that made anassumption that ``ensurepip`` would be provisioning more then just asingle wheel. That assumption is no longer true since [[1]][[2]][[3]].In this change, the improvement is done around removing unnecessaryloops and supporting structures to change the assumptions to expectonly the bundled or replacement ``pip`` wheel.[1]:python@ece20db[2]:python#101039[3]:python#95299
webknjaz added a commit to webknjaz/cpython that referenced this pull requestJan 25, 2024
This is a refactoring change that aims to simplify ``ensurepip``.Before it, this module had legacy infrastructure that made anassumption that ``ensurepip`` would be provisioning more then just asingle wheel. That assumption is no longer true since [[1]][[2]][[3]].In this change, the improvement is done around removing unnecessaryloops and supporting structures to change the assumptions to expectonly the bundled or replacement ``pip`` wheel.[1]:python@ece20db[2]:python#101039[3]:python#95299
edmorley added a commit to edmorley/get-pip that referenced this pull requestJul 7, 2024
Currently `get-pip.py` installs not only pip, but also setuptools andwheel by default, unless the `--no-setuptools` / `--no-wheel` (or`PIP_NO_SETUPTOOLS` / `PIP_NO_WHEEL` env vars) are used.This has historically been necessary, however, modern versions of pipwill now fallback to `pyproject.toml` (PEP 517: [1]) based builds (whichwill default to a setuptools backend, and thus automatically installsetuptools and wheel in the isolated build environment) if eithersetuptools is not installed (as of pip 22.1: [2]), or if wheel is notinstalled (as of pip 23.1: [3]).In addition, as of Python 3.12, the stdlib's `ensurepip` and `venv`modules no longer install setuptools, and only install pip ([4]).As such, it is now time for `get-pip.py` to stop installing setuptoolsand wheel by default on Python 3.12+, in order to:- Act as another small step towards `pyproject.toml` / PEP 517  based builds eventually becoming the pip default.- Improve parity with the behaviour of `ensurepip` / `venv` on  Python 3.12+.- Allow `get-pip.py` to focus on its primary responsibility:  bootstrapping Pip.Closespypa#200.[1]:https://peps.python.org/pep-0517/[2]:pypa/pip#10717[3]:pypa/pip#11871[4]:python/cpython#101039
edmorley added a commit to edmorley/get-pip that referenced this pull requestJul 7, 2024
Currently `get-pip.py` installs not only pip, but also setuptools andwheel by default, unless the `--no-setuptools` / `--no-wheel` (or`PIP_NO_SETUPTOOLS` / `PIP_NO_WHEEL` env vars) are used.This has historically been necessary, however, modern versions of pipwill now fallback to `pyproject.toml` (PEP 517: [1]) based builds (whichwill default to a setuptools backend, and thus automatically installsetuptools and wheel in the isolated build environment) if eithersetuptools is not installed (as of pip 22.1: [2]), or if wheel is notinstalled (as of pip 23.1: [3]).In addition, as of Python 3.12, the stdlib's `ensurepip` and `venv`modules no longer install setuptools, and only install pip ([4]).As such, it is now time for `get-pip.py` to stop installing setuptoolsand wheel by default on Python 3.12+, in order to:- Act as another small step towards `pyproject.toml` / PEP 517  based builds eventually becoming the pip default.- Improve parity with the behaviour of `ensurepip` / `venv` on  Python 3.12+.- Allow `get-pip.py` to focus on its primary responsibility:  bootstrapping Pip.Closespypa#200.[1]:https://peps.python.org/pep-0517/[2]:pypa/pip#10717[3]:pypa/pip#11871[4]:python/cpython#101039
edmorley added a commit to edmorley/get-pip that referenced this pull requestJul 29, 2024
Currently `get-pip.py` installs not only pip, but also setuptools andwheel by default, unless the `--no-setuptools` / `--no-wheel` (or`PIP_NO_SETUPTOOLS` / `PIP_NO_WHEEL` env vars) are used.This has historically been necessary, however, modern versions of pipwill now fallback to `pyproject.toml` (PEP 517: [1]) based builds (whichwill default to a setuptools backend, and thus automatically installsetuptools and wheel in the isolated build environment) if eithersetuptools is not installed (as of pip 22.1: [2]), or if wheel is notinstalled (as of pip 23.1: [3]).In addition, as of Python 3.12, the stdlib's `ensurepip` and `venv`modules no longer install setuptools, and only install pip ([4]).As such, it is now time for `get-pip.py` to stop installing setuptoolsand wheel by default on Python 3.12+, in order to:- Act as another small step towards `pyproject.toml` / PEP 517  based builds eventually becoming the pip default.- Improve parity with the behaviour of `ensurepip` / `venv` on  Python 3.12+.- Allow `get-pip.py` to focus on its primary responsibility:  bootstrapping Pip.Closespypa#200.[1]:https://peps.python.org/pep-0517/[2]:pypa/pip#10717[3]:pypa/pip#11871[4]:python/cpython#101039
edmorley added a commit to edmorley/get-pip that referenced this pull requestJul 29, 2024
Currently `get-pip.py` installs not only pip, but also setuptools andwheel by default, unless the `--no-setuptools` / `--no-wheel` (or`PIP_NO_SETUPTOOLS` / `PIP_NO_WHEEL` env vars) are used.This has historically been necessary, however, modern versions of pipwill now fallback to `pyproject.toml` (PEP 517: [1]) based builds (whichwill default to a setuptools backend, and thus automatically installsetuptools and wheel in the isolated build environment) if eithersetuptools is not installed (as of pip 22.1: [2]), or if wheel is notinstalled (as of pip 23.1: [3]).In addition, as of Python 3.12, the stdlib's `ensurepip` and `venv`modules no longer install setuptools, and only install pip ([4]).As such, it is now time for `get-pip.py` to stop installing setuptoolsand wheel by default on Python 3.12+, in order to:- Act as another small step towards `pyproject.toml` / PEP 517  based builds eventually becoming the pip default.- Improve parity with the behaviour of `ensurepip` / `venv` on  Python 3.12+.- Allow `get-pip.py` to focus on its primary responsibility:  bootstrapping Pip.Closespypa#200.[1]:https://peps.python.org/pep-0517/[2]:pypa/pip#10717[3]:pypa/pip#11871[4]:python/cpython#101039
edmorley added a commit to heroku/buildpacks-python that referenced this pull requestAug 7, 2024
Currently the buildpack performs a system site-packages install of notonly pip, but also setuptools and wheel. This has historically beennecessary for pip to be able to build source distributions (sdists) forpackages that don't ship with compatible wheels.However:- Thanks to PEP 518, packages can now (and many already do) specify an  explicit build backend using `[build-system]` in their  `pyproject.toml`. The dependencies specified in that config (such as  setuptools and wheel) will be installed by pip into an isolated and  ephemeral build environment as part of the source distribution build  process. Such packages therefore don't need/use globally installed  setuptools/wheel versions.- As of pip v22.1, pip will now default to the isolated build environment  mode (along with a fallback legacy setuptools build backend), if the  setuptools package isn't installed globally. This means that packages  that haven't yet migrated to a PEP 518 `pyproject.toml` build backend  config can still build even if setuptools isn't installed globally.There are a small number of rarely used packages in the wild thataren't compatible with build isolation mode, however, these typicallyrequire more build dependencies than just setuptools, which means theywouldn't have worked with this buildpack anyway.As such, it's no longer necessary for us to install setuptools and wheelglobally. This matches the behaviour of the `venv` and `ensurepip` modulesin Python 3.12+, where setuptools and wheel installation has also beenremoved. And it also matches the default behaviour of Poetry too, whose`install --sync` command removes any implicitly installed packages in thecurrent environment (other than pip).See:https://peps.python.org/pep-0518/https://pip.pypa.io/en/stable/reference/build-system/https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolationpypa/pip#10717python/cpython#101039pypa/get-pip#218astral-sh/uv#2252GUS-W-16437776.
edmorley added a commit to heroku/buildpacks-python that referenced this pull requestAug 7, 2024
Currently the buildpack performs a system site-packages install of notonly pip, but also setuptools and wheel. This has historically beennecessary for pip to be able to build source distributions (sdists) forpackages that don't ship with compatible wheels.However:- Thanks to PEP 518, packages can now (and many already do) specify an  explicit build backend using `[build-system]` in their  `pyproject.toml`. The dependencies specified in that config (such as  setuptools and wheel) will be installed by pip into an isolated and  ephemeral build environment as part of the source distribution build  process. Such packages therefore don't need/use globally installed  setuptools/wheel versions.- As of pip v22.1, pip will now default to the isolated build environment  mode (along with a fallback legacy setuptools build backend), if the  setuptools package isn't installed globally. This means that packages  that haven't yet migrated to a PEP 518 `pyproject.toml` build backend  config can still build even if setuptools isn't installed globally.There are a small number of rarely used packages in the wild thataren't compatible with build isolation mode, however, these typicallyrequire more build dependencies than just setuptools, which means theywouldn't have worked with this buildpack anyway.As such, it's no longer necessary for us to install setuptools and wheelglobally. This matches the behaviour of the `venv` and `ensurepip` modulesin Python 3.12+, where setuptools and wheel installation has also beenremoved. And it also matches the default behaviour of Poetry too, whose`install --sync` command removes any implicitly installed packages in thecurrent environment (other than pip).See:https://peps.python.org/pep-0518/https://pip.pypa.io/en/stable/reference/build-system/https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolationpypa/pip#10717python/cpython#101039pypa/get-pip#218astral-sh/uv#2252GUS-W-16437776.
edmorley added a commit to heroku/buildpacks-python that referenced this pull requestAug 7, 2024
Currently the buildpack performs a system site-packages install of notonly pip, but also setuptools and wheel. This has historically beennecessary for pip to be able to build source distributions (sdists) forpackages that don't ship with compatible wheels.However:- Thanks to PEP 518, packages can now (and many already do) specify an  explicit build backend using `[build-system]` in their  `pyproject.toml`. The dependencies specified in that config (such as  setuptools and wheel) will be installed by pip into an isolated and  ephemeral build environment as part of the source distribution build  process. Such packages therefore don't need/use globally installed  setuptools/wheel versions.- As of pip v22.1, pip will now default to the isolated build environment  mode (along with a fallback legacy setuptools build backend), if the  setuptools package isn't installed globally. This means that packages  that haven't yet migrated to a PEP 518 `pyproject.toml` build backend  config can still build even if setuptools isn't installed globally.There are a small number of rarely used packages in the wild thataren't compatible with build isolation mode, however, these typicallyrequire more build dependencies than just setuptools, which means theywouldn't have worked with this buildpack anyway.As such, it's no longer necessary for us to install setuptools and wheelglobally. This matches the behaviour of the `venv` and `ensurepip` modulesin Python 3.12+, where setuptools and wheel installation has also beenremoved. And it also matches the default behaviour of Poetry too, whose`install --sync` command removes any implicitly installed packages in thecurrent environment (other than pip).See:https://peps.python.org/pep-0518/https://pip.pypa.io/en/stable/reference/build-system/https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolationpypa/pip#10717python/cpython#101039pypa/get-pip#218astral-sh/uv#2252GUS-W-16437776.
Tenzer added a commit to Tenzer/aws-lambda-python-runtime-interface-client that referenced this pull requestOct 10, 2024
It is necessary for building the package when running the integration tests,but more recent Python versions no longer include it by default:python/cpython#101039.
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@vstinnervstinnervstinner left review comments

@merwokmerwokmerwok left review comments

@pfmoorepfmoorepfmoore approved these changes

@hugovkhugovkhugovk left review comments

@CAM-GerlachCAM-GerlachCAM-Gerlach left review comments

@vsajipvsajipvsajip approved these changes

@dstufftdstufftdstufft approved these changes

@ezio-melottiezio-melottiAwaiting requested review from ezio-melotti

Assignees
No one assigned
Labels
3.12only security fixesstdlibPython modules in the Lib dirtopic-ensurepip
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

12 participants
@pradyunsg@encukou@bedevere-bot@pfmoore@hugovk@AlexWaygood@CAM-Gerlach@vstinner@vsajip@dstufft@merwok@arhadthedev

[8]ページ先頭

©2009-2025 Movatter.jp