Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-130094: Fix race conditions inimportlib#130101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
colesbury merged 6 commits intopython:mainfromcolesbury:gh-130094-sys-meta-path
Feb 18, 2025

Conversation

@colesbury
Copy link
Contributor

@colesburycolesbury commentedFeb 14, 2025
edited
Loading

Entries may be added or removed fromsys.meta_path concurrently. For example, setuptools temporarily adds and removes thedistutils finder from the beginning of the list. The local copy ensures that we don't skip over any entries.

Some packages modifysys.modules during import. For example,collections inserts the entry forcollections.abc intosys.modules during import. We need to ensure that we re-checksys.modulesafter the parent module is fully initialized.

Entries may be added or removed from `sys.meta_path` concurrently. Forexample, setuptools temporarily adds and removes the `distutils`finder from the beginning of the list. The local copy ensures that wedon't skip over any entries.
@eendebakpt
Copy link
Contributor

This does not resolve the issue for me, although it is more difficult to trigger now. It is a bit hard to determine what is going wrong as adding simpleprint statements toimportlib._bootstrap.py causes the compilation to fail (something related to the deep freezing or the printing not being available during bootstrap).

The error I get is now always incollections.abc. I suspect that there is a race with two threads are concurrently loadingcollections andcollections.abc. Traceback is here:

Exception in thread Thread-7 (work):Traceback (most recent call last):  File "c:\projects\misc\cpython\Lib\threading.py", line 1054, in _bootstrap_inner    self.run()    ~~~~~~~~^^  File "c:\projects\misc\cpython\Lib\threading.py", line 996, in run    self._target(*self._args, **self._kwargs)    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "c:\projects\tmp\ft_import.py", line 21, in work    mod = import_module(m)  File "c:\projects\misc\cpython\Lib\importlib\__init__.py", line 88, in import_module    return _bootstrap._gcd_import(name[level:], package, level)           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "<frozen importlib._bootstrap>", line 1389, in _gcd_import  File "<frozen importlib._bootstrap>", line 1362, in _find_and_load  File "<frozen importlib._bootstrap>", line 1326, in _find_and_load_unlockedModuleNotFoundError: No module named 'collections.abc'

A minimal reproducer (no exeternal packages involved now):

import sysfrom threading import Thread, Barrierfrom importlib import import_moduleimport timenumber_of_threads = 2barrier = Barrier(number_of_threads)pmods = []def work(ii):    # Method doing the actual import    barrier.wait()    while True:        try:            m = pmods.pop()            mod = import_module(m)        except IndexError:            returnworker_threads = []for ii in range(number_of_threads):    worker_threads.append(Thread(target=work, args=[ii]))def parallel_import(modules: list[str]):    global pmods    pmods += modules    for t in worker_threads:        t.start()    for t in worker_threads:        t.join()mods = [    "collections.abc",    "collections",]parallel_import(mods)print("parallel_import complete")

When running the minimal example with a batch file 80 times, the issue is triggered with very high probability.

@colesburycolesbury changed the titlegh-130094: Fix race condition in_find_specgh-130094: Fix race conditions in_find_specFeb 14, 2025
@colesbury
Copy link
ContributorAuthor

@eendebakpt - I fixed another race condition. Would you please try again?

@colesburycolesbury changed the titlegh-130094: Fix race conditions in_find_specgh-130094: Fix race conditions inimportlibFeb 14, 2025
@eendebakpt
Copy link
Contributor

@colesbury With the latest commits I see no problems any more.

I created a unit test for this issue based onimportlib.reload in

3dc2bee

The test works, but is a bit slow. If I reduce the number of iterations (so that total running time is less than 0.1), odds of triggering the problem are rather low.

Co-authored-by: Brett Cannon <brett@python.org>
@colesburycolesbury merged commit857bdba intopython:mainFeb 18, 2025
39 checks passed
@colesburycolesbury deleted the gh-130094-sys-meta-path branchFebruary 18, 2025 23:02
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@brettcannonbrettcannonbrettcannon approved these changes

@ericsnowcurrentlyericsnowcurrentlyAwaiting requested review from ericsnowcurrentlyericsnowcurrently is a code owner

@ncoghlanncoghlanAwaiting requested review from ncoghlanncoghlan is a code owner

@warsawwarsawAwaiting requested review from warsawwarsaw is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@colesbury@eendebakpt@brettcannon

[8]ページ先頭

©2009-2025 Movatter.jp