Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork32k
gh-95299: Stop installing setuptools as a part of ensurepip and venv#101039
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
pradyunsg commentedJan 14, 2023 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Hmm...@vstinner@encukou any opinions on what I should do with That test has an implicit assumption that Notably, some leading questions that affect what direction I could take for updating that test:
|
17a121e
to32b3ccc
Comparepradyunsg commentedJan 14, 2023 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Since it's cheap to revert commits/remove them from a PR, and nice to have a code change to discuss details over, I've gone ahead and pushed a change that assumes that the answer to the questions I raised above is yes, yes and yes. :) Mentioning a GitHub UI thing that I find very useful (and non-initutive): you can click "force-pushed" on the event above to see what changed in the force pushes between the hashes that GitHub lists in the from ... to ... at the end of the event. |
32b3ccc
to6939d48
CompareUh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The changes seem reasonable to me, though I'm only the code owner for the venv and test_venv bits.
IMO, that would be best. Relatedly, I've been thinking how to handle Anyway, if that sounds like more than you signed up for, go ahead with Setuptools. |
I spent some time today looking into this after my work day, and I've not figured out the details in a portable manner. I'd prefer if this could be tackled in a follow up instead. :) |
bedevere-bot commentedJan 31, 2023
🤖 New build scheduled with the buildbot fleet by@pradyunsg for commit7457fd3 🤖 If you want to schedule another build, you need to add the |
Uh oh!
There was an error while loading.Please reload this page.
I added a review comment that should fix the buildbot errors. |
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Should this also go in What's New, underRemoved? |
The inclusion of setuptools was always explicitly documented as an internal implementation detail, and as such, I don't think we have any obligation to report its removal. It could be mentioned in the same way that we might document the removal of an internal API - I did a quick check and the 3.11 What's New included "Removed the undocumented private |
Please add it as a convenience to users, even if there's technically no obligation. It's beenvery easy to end up depending on setuptools accidentally. IMO, for removals/breaks, it's best to write What's New entries for the target audience of a poor soul who isn't too well-versed in Python but inherited some complex system with a “temporary” hack. Point them to a solution, don't apologize, but don't berate them either. (cc @python/proofreaders -- hope the above is good general advice) |
Thanks all! Holler on the issue, if there's any concerns! :) |
* main: (24 commits)pythongh-98040: Move the Single-Phase Init Tests Out of test_imp (pythongh-102561)pythongh-83861: Fix datetime.astimezone() method (pythonGH-101545)pythongh-102856: Clean some of the PEP 701 tokenizer implementation (python#103634)pythongh-102856: Skip test_mismatched_parens in WASI builds (python#103633)pythongh-102856: Initial implementation of PEP 701 (python#102855)pythongh-103583: Add ref. dependency between multibytecodec modules (python#103589)pythongh-83004: Harden msvcrt further (python#103420)pythonGH-88342: clarify that `asyncio.as_completed` accepts generators yielding tasks (python#103626)pythongh-102778: IDLE - make sys.last_exc available in Shell after traceback (python#103314)pythongh-103582: Remove last references to `argparse.REMAINDER` from docs (python#103586)pythongh-103583: Always pass multibyte codec structs as const (python#103588)pythongh-103617: Fix compiler warning in _iomodule.c (python#103618)pythongh-103596: [Enum] do not shadow mixed-in methods/attributes (pythonGH-103600)pythonGH-100530: Change the error message for non-class class patterns (pythonGH-103576)pythongh-95299: Remove lingering setuptools reference in installer scripts (pythonGH-103613) [Doc] Fix a typo in optparse.rst (python#103504)pythongh-101100: Fix broken reference `__format__` in `string.rst` (python#103531)pythongh-95299: Stop installing setuptools as a part of ensurepip and venv (python#101039)pythonGH-103484: Docs: add linkcheck allowed redirects entries for most cases (python#103569)pythongh-67230: update whatsnew note for csv changes (python#103598) ...
This is a refactoring change that aims to simplify ``ensurepip``.Before it, this module had legacy infrastructure that made anassumption that ``ensurepip`` would be provisioning more then just asingle wheel. That assumption is no longer true since [[1]][[2]][[3]].In this change, the improvement is done around removing unnecessaryloops and supporting structures to change the assumptions to expectonly the bundled or replacement ``pip`` wheel.[1]:python@ece20db[2]:python#101039[2]:python#95299
This is a refactoring change that aims to simplify ``ensurepip``.Before it, this module had legacy infrastructure that made anassumption that ``ensurepip`` would be provisioning more then just asingle wheel. That assumption is no longer true since [[1]][[2]][[3]].In this change, the improvement is done around removing unnecessaryloops and supporting structures to change the assumptions to expectonly the bundled or replacement ``pip`` wheel.[1]:python@ece20db[2]:python#101039[3]:python#95299
This is a refactoring change that aims to simplify ``ensurepip``.Before it, this module had legacy infrastructure that made anassumption that ``ensurepip`` would be provisioning more then just asingle wheel. That assumption is no longer true since [[1]][[2]][[3]].In this change, the improvement is done around removing unnecessaryloops and supporting structures to change the assumptions to expectonly the bundled or replacement ``pip`` wheel.[1]:python@ece20db[2]:python#101039[3]:python#95299
This is a refactoring change that aims to simplify ``ensurepip``.Before it, this module had legacy infrastructure that made anassumption that ``ensurepip`` would be provisioning more then just asingle wheel. That assumption is no longer true since [[1]][[2]][[3]].In this change, the improvement is done around removing unnecessaryloops and supporting structures to change the assumptions to expectonly the bundled or replacement ``pip`` wheel.[1]:python@ece20db[2]:python#101039[3]:python#95299
This is a refactoring change that aims to simplify ``ensurepip``.Before it, this module had legacy infrastructure that made anassumption that ``ensurepip`` would be provisioning more then just asingle wheel. That assumption is no longer true since [[1]][[2]][[3]].In this change, the improvement is done around removing unnecessaryloops and supporting structures to change the assumptions to expectonly the bundled or replacement ``pip`` wheel.[1]:python@ece20db[2]:python#101039[3]:python#95299
Currently `get-pip.py` installs not only pip, but also setuptools andwheel by default, unless the `--no-setuptools` / `--no-wheel` (or`PIP_NO_SETUPTOOLS` / `PIP_NO_WHEEL` env vars) are used.This has historically been necessary, however, modern versions of pipwill now fallback to `pyproject.toml` (PEP 517: [1]) based builds (whichwill default to a setuptools backend, and thus automatically installsetuptools and wheel in the isolated build environment) if eithersetuptools is not installed (as of pip 22.1: [2]), or if wheel is notinstalled (as of pip 23.1: [3]).In addition, as of Python 3.12, the stdlib's `ensurepip` and `venv`modules no longer install setuptools, and only install pip ([4]).As such, it is now time for `get-pip.py` to stop installing setuptoolsand wheel by default on Python 3.12+, in order to:- Act as another small step towards `pyproject.toml` / PEP 517 based builds eventually becoming the pip default.- Improve parity with the behaviour of `ensurepip` / `venv` on Python 3.12+.- Allow `get-pip.py` to focus on its primary responsibility: bootstrapping Pip.Closespypa#200.[1]:https://peps.python.org/pep-0517/[2]:pypa/pip#10717[3]:pypa/pip#11871[4]:python/cpython#101039
Currently `get-pip.py` installs not only pip, but also setuptools andwheel by default, unless the `--no-setuptools` / `--no-wheel` (or`PIP_NO_SETUPTOOLS` / `PIP_NO_WHEEL` env vars) are used.This has historically been necessary, however, modern versions of pipwill now fallback to `pyproject.toml` (PEP 517: [1]) based builds (whichwill default to a setuptools backend, and thus automatically installsetuptools and wheel in the isolated build environment) if eithersetuptools is not installed (as of pip 22.1: [2]), or if wheel is notinstalled (as of pip 23.1: [3]).In addition, as of Python 3.12, the stdlib's `ensurepip` and `venv`modules no longer install setuptools, and only install pip ([4]).As such, it is now time for `get-pip.py` to stop installing setuptoolsand wheel by default on Python 3.12+, in order to:- Act as another small step towards `pyproject.toml` / PEP 517 based builds eventually becoming the pip default.- Improve parity with the behaviour of `ensurepip` / `venv` on Python 3.12+.- Allow `get-pip.py` to focus on its primary responsibility: bootstrapping Pip.Closespypa#200.[1]:https://peps.python.org/pep-0517/[2]:pypa/pip#10717[3]:pypa/pip#11871[4]:python/cpython#101039
Currently `get-pip.py` installs not only pip, but also setuptools andwheel by default, unless the `--no-setuptools` / `--no-wheel` (or`PIP_NO_SETUPTOOLS` / `PIP_NO_WHEEL` env vars) are used.This has historically been necessary, however, modern versions of pipwill now fallback to `pyproject.toml` (PEP 517: [1]) based builds (whichwill default to a setuptools backend, and thus automatically installsetuptools and wheel in the isolated build environment) if eithersetuptools is not installed (as of pip 22.1: [2]), or if wheel is notinstalled (as of pip 23.1: [3]).In addition, as of Python 3.12, the stdlib's `ensurepip` and `venv`modules no longer install setuptools, and only install pip ([4]).As such, it is now time for `get-pip.py` to stop installing setuptoolsand wheel by default on Python 3.12+, in order to:- Act as another small step towards `pyproject.toml` / PEP 517 based builds eventually becoming the pip default.- Improve parity with the behaviour of `ensurepip` / `venv` on Python 3.12+.- Allow `get-pip.py` to focus on its primary responsibility: bootstrapping Pip.Closespypa#200.[1]:https://peps.python.org/pep-0517/[2]:pypa/pip#10717[3]:pypa/pip#11871[4]:python/cpython#101039
Currently `get-pip.py` installs not only pip, but also setuptools andwheel by default, unless the `--no-setuptools` / `--no-wheel` (or`PIP_NO_SETUPTOOLS` / `PIP_NO_WHEEL` env vars) are used.This has historically been necessary, however, modern versions of pipwill now fallback to `pyproject.toml` (PEP 517: [1]) based builds (whichwill default to a setuptools backend, and thus automatically installsetuptools and wheel in the isolated build environment) if eithersetuptools is not installed (as of pip 22.1: [2]), or if wheel is notinstalled (as of pip 23.1: [3]).In addition, as of Python 3.12, the stdlib's `ensurepip` and `venv`modules no longer install setuptools, and only install pip ([4]).As such, it is now time for `get-pip.py` to stop installing setuptoolsand wheel by default on Python 3.12+, in order to:- Act as another small step towards `pyproject.toml` / PEP 517 based builds eventually becoming the pip default.- Improve parity with the behaviour of `ensurepip` / `venv` on Python 3.12+.- Allow `get-pip.py` to focus on its primary responsibility: bootstrapping Pip.Closespypa#200.[1]:https://peps.python.org/pep-0517/[2]:pypa/pip#10717[3]:pypa/pip#11871[4]:python/cpython#101039
Currently the buildpack performs a system site-packages install of notonly pip, but also setuptools and wheel. This has historically beennecessary for pip to be able to build source distributions (sdists) forpackages that don't ship with compatible wheels.However:- Thanks to PEP 518, packages can now (and many already do) specify an explicit build backend using `[build-system]` in their `pyproject.toml`. The dependencies specified in that config (such as setuptools and wheel) will be installed by pip into an isolated and ephemeral build environment as part of the source distribution build process. Such packages therefore don't need/use globally installed setuptools/wheel versions.- As of pip v22.1, pip will now default to the isolated build environment mode (along with a fallback legacy setuptools build backend), if the setuptools package isn't installed globally. This means that packages that haven't yet migrated to a PEP 518 `pyproject.toml` build backend config can still build even if setuptools isn't installed globally.There are a small number of rarely used packages in the wild thataren't compatible with build isolation mode, however, these typicallyrequire more build dependencies than just setuptools, which means theywouldn't have worked with this buildpack anyway.As such, it's no longer necessary for us to install setuptools and wheelglobally. This matches the behaviour of the `venv` and `ensurepip` modulesin Python 3.12+, where setuptools and wheel installation has also beenremoved. And it also matches the default behaviour of Poetry too, whose`install --sync` command removes any implicitly installed packages in thecurrent environment (other than pip).See:https://peps.python.org/pep-0518/https://pip.pypa.io/en/stable/reference/build-system/https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolationpypa/pip#10717python/cpython#101039pypa/get-pip#218astral-sh/uv#2252GUS-W-16437776.
Currently the buildpack performs a system site-packages install of notonly pip, but also setuptools and wheel. This has historically beennecessary for pip to be able to build source distributions (sdists) forpackages that don't ship with compatible wheels.However:- Thanks to PEP 518, packages can now (and many already do) specify an explicit build backend using `[build-system]` in their `pyproject.toml`. The dependencies specified in that config (such as setuptools and wheel) will be installed by pip into an isolated and ephemeral build environment as part of the source distribution build process. Such packages therefore don't need/use globally installed setuptools/wheel versions.- As of pip v22.1, pip will now default to the isolated build environment mode (along with a fallback legacy setuptools build backend), if the setuptools package isn't installed globally. This means that packages that haven't yet migrated to a PEP 518 `pyproject.toml` build backend config can still build even if setuptools isn't installed globally.There are a small number of rarely used packages in the wild thataren't compatible with build isolation mode, however, these typicallyrequire more build dependencies than just setuptools, which means theywouldn't have worked with this buildpack anyway.As such, it's no longer necessary for us to install setuptools and wheelglobally. This matches the behaviour of the `venv` and `ensurepip` modulesin Python 3.12+, where setuptools and wheel installation has also beenremoved. And it also matches the default behaviour of Poetry too, whose`install --sync` command removes any implicitly installed packages in thecurrent environment (other than pip).See:https://peps.python.org/pep-0518/https://pip.pypa.io/en/stable/reference/build-system/https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolationpypa/pip#10717python/cpython#101039pypa/get-pip#218astral-sh/uv#2252GUS-W-16437776.
Currently the buildpack performs a system site-packages install of notonly pip, but also setuptools and wheel. This has historically beennecessary for pip to be able to build source distributions (sdists) forpackages that don't ship with compatible wheels.However:- Thanks to PEP 518, packages can now (and many already do) specify an explicit build backend using `[build-system]` in their `pyproject.toml`. The dependencies specified in that config (such as setuptools and wheel) will be installed by pip into an isolated and ephemeral build environment as part of the source distribution build process. Such packages therefore don't need/use globally installed setuptools/wheel versions.- As of pip v22.1, pip will now default to the isolated build environment mode (along with a fallback legacy setuptools build backend), if the setuptools package isn't installed globally. This means that packages that haven't yet migrated to a PEP 518 `pyproject.toml` build backend config can still build even if setuptools isn't installed globally.There are a small number of rarely used packages in the wild thataren't compatible with build isolation mode, however, these typicallyrequire more build dependencies than just setuptools, which means theywouldn't have worked with this buildpack anyway.As such, it's no longer necessary for us to install setuptools and wheelglobally. This matches the behaviour of the `venv` and `ensurepip` modulesin Python 3.12+, where setuptools and wheel installation has also beenremoved. And it also matches the default behaviour of Poetry too, whose`install --sync` command removes any implicitly installed packages in thecurrent environment (other than pip).See:https://peps.python.org/pep-0518/https://pip.pypa.io/en/stable/reference/build-system/https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolationpypa/pip#10717python/cpython#101039pypa/get-pip#218astral-sh/uv#2252GUS-W-16437776.
It is necessary for building the package when running the integration tests,but more recent Python versions no longer include it by default:python/cpython#101039.
Uh oh!
There was an error while loading.Please reload this page.
This PR removes the bundled setuptools wheel from ensurepip, and stops installing setuptools in environments created by venv.
I based this PR off of my understanding of venv and ensurepip, and using the following search to locate all the relevant bits of code + ensuring that tests are happy locally.
In this PR, I've intentionally not touched the example inhttps://docs.python.org/3.12/library/venv.html#an-example-of-extending-envbuilder. That example is largely outdated and is likely better served being updated in a dedicated PR for it.