Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-71936: Fix race condition in multiprocessing.Pool#98274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Closed
hattya wants to merge2 commits intopython:mainfromhattya:fix-issue-71936

Conversation

@hattya
Copy link
Contributor

@hattyahattya commentedOct 15, 2022
edited
Loading

Proxes of shared objects register a Finalizer in BaseProxy._incref(), and it will call BaseProxy._decref() when it is GCed. This may cause a race condition with Pool(maxtasksperchild=None) on Windows.

A connection will be closed and raised TypeError when a GC occurs between _ConnectionBase._check_writable() and _ConnectionBase._send_bytes() in _ConnectionBase.send() in the second or later task.

BaseProxy does not count references well.

Proxes of shared objects register a Finalizer in BaseProxy._incref(), and itwill call BaseProxy._decref() when it is GCed. This may cause a race conditionwith Pool(maxtasksperchild=None) on Windows.A connection will be closed and raised TypeError when a GC occurs between_ConnectionBase._check_writable() and _ConnectionBase._send_bytes() in_ConnectionBase.send() in the second or later task.BaseProxy does not count references well.
@bedevere-bot
Copy link

Most changes to Pythonrequire a NEWS entry.

Please add it using theblurb_it web app or theblurb command-line tool.

@ghost
Copy link

ghost commentedOct 15, 2022
edited by ghost
Loading

All commit authors signed the Contributor License Agreement.
CLA signed

@hattya
Copy link
ContributorAuthor

Steps to reproduce:

>xcopy /I"%ProgramFiles%\Python310\Lib\multiprocessing" multiprocessing>type a.patch--- a/multiprocessing/connection.py+++ b/multiprocessing/connection.py@@ -282,6 +282,8 @@             _CloseHandle(self._handle)         def _send_bytes(self, buf):+            if (10, 1) in util._finalizer_registry:+                import gc; gc.collect()             ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)             try:                 if err == _winapi.ERROR_IO_PENDING:>patch -p1< a.patch>type a.py#!/usr/bin/env pythonimport multiprocessing as mpdef foo(lock, i):    def func(x):        # for circular reference        with lock:            pass        if x > 0:            return func(x - 1)    func(i)if __name__ == '__main__':    m = mp.Manager()    lock = m.Lock()    with mp.Pool(2) as pool:        for _ in pool.starmap(foo, ((lock, i) for i in range(10))):            pass>a.py

@bedevere-bot
Copy link

Most changes to Pythonrequire a NEWS entry.

Please add it using theblurb_it web app or theblurb command-line tool.

@basnijholt
Copy link

basnijholt commentedJun 12, 2024
edited
Loading

Like I mentioned here#71936 (comment), this patch resolves the problem I am running into on Python 3.12.3.

@hattya, could you add a news entry like the bot suggests? I see you already did that.

WaterHyacinthInNANHU reacted with thumbs up emoji

@basnijholt
Copy link

basnijholt commentedSep 11, 2024
edited
Loading

I just ran into this exact problem again in a slightly different context thanmy previous comment.

@encukou,@gpshead,@vstinner, and@AlexWaygood (most recent reviewers of this module – apologies for direct tag), would any of you be able to take a look at this?

This PR solves the race condition I keep running into. For example, seethis build of the documentation.

@encukou
Copy link
Member

My review led to a “competing” PR:#124973
@basnijholt, could you check if that also solves the issue for you?

@encukou
Copy link
Member

I have merged the PR that builds on this one.
Thank you for the fix,@hattya!

@basnijholt
Copy link

basnijholt commentedNov 15, 2024
edited
Loading

@encukou, I also replied in#71936 (comment) and it indeed solves my problem!

With this fix hopefully Readthedocs builds forhttps://github.com/pipefunc/pipefunc will not randomly fail now 👍

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@gpsheadgpsheadAwaiting requested review from gpsheadgpshead is a code owner

Assignees

No one assigned

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hattya@bedevere-bot@basnijholt@encukou@AlexWaygood

[8]ページ先頭

©2009-2025 Movatter.jp