Movatterモバイル変換


[0]ホーム

URL:


homepage

Issue5331

This issue trackerhas been migrated toGitHub, and is currentlyread-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title:multiprocessing hangs when Pool used within Process
Type:crashStage:
Components:Library (Lib)Versions:Python 3.0, Python 2.6
process
Status:closedResolution:fixed
Dependencies:Superseder:
Assigned To: jnollerNosy List: LambertDW, OG7, jnoller, zzzeek
Priority:normalKeywords:

Created on2009-02-20 16:06 byzzzeek, last changed2022-04-11 14:56 byadmin. This issue is nowclosed.

Messages (6)
msg82534 -(view)Author: mike bayer (zzzeek)*Date: 2009-02-20 16:06
this occurs for me running on Mac OSX Leopard.   The equivalent codeusing "processing" in python 2.5 works fine, which is how I found thisbug - my code hung when upgraded to 2.6.    Basically initiating amultiprocessing.Pool inside of multiprocessing.Process hangs theapplication.  Below is code which illustrates the issue on both py2.6and py3.0:    from multiprocessing import Pool, Process    import sys    def f(x):        return x*x    def go():        pool = Pool(processes=4)                      x = pool.map(f, [1, 2, 3, 4, 5, 6, 7])        sys.stdout.write("result: %s\n" % x)    def go2():        x = map(f, [1,2,3,4,5,6,7])        sys.stdout.write("result: %s\n" % x)    # pool alone, fine    go()    # process alone, fine    p = Process(target=go2)    p.start()    p.join()    # use both, hangs    p = Process(target=go)    p.start()    p.join()
msg82540 -(view)Author: David W. Lambert (LambertDW)Date: 2009-02-20 18:57
Fails also on this system.$ p3Python 3.0.1 (r301:69556, Feb 13 2009, 23:52:55) [GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2(...)>>> # use both hangs(...)>>> p.start()>>> p.join()(hmm la  do diii   laaa...)^CTraceback (most recent call last):  File "<stdin>", line 1, in <module>  File"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/process.py",line 119, in join    res = self._popen.wait(timeout)  File"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/forking.py",line 118, in wait    return self.poll(0)  File"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/forking.py",line 107, in poll    pid, sts = os.waitpid(self.pid, flag)KeyboardInterrupt>>>
msg89192 -(view)Author: OG7 (OG7)Date: 2009-06-10 08:58
Apparently the pool workers die all at once, just after the pool createsthem and before the pool is used.I added a few lines to multiprocessing/pool.py to get the stack and theexception backtrace.        except (EOFError, IOError):            import traceback            debug(traceback.format_exc())            debug(''.join(traceback.format_stack()))            debug('worker got EOFError or IOError -- exiting')            breakINFO::Rule dispatcher::multiprocessing::child process calling self.run()DEBUG::Rule dispatcher::multiprocessing::created semlock with handle3082559488DEBUG::Rule dispatcher::multiprocessing::created semlock with handle3082104832DEBUG::Rule dispatcher::multiprocessing::created semlock with handle3081826304DEBUG::Rule dispatcher::multiprocessing::created semlock with handle3081822208INFO::PoolWorker-3:1::multiprocessing::child process calling self.run()DEBUG::PoolWorker-3:1::multiprocessing::Traceback (most recent call last):  File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",line 57, in worker    task = get()  File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/queues.py",line 339, in get    return recv()IOError: [Errno 9] Bad file descriptorDEBUG::PoolWorker-3:1::multiprocessing::  File"/home/who/var/co/git-svn/what-base/correlator/bin/correlator", line 30,in <module>    what.corr.actors.main.main()  File"/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/main.py",line 47, in main    rrp.start()  File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",line 109, in start    self._popen = Popen(self)  File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py",line 99, in __init__    code = process_obj._bootstrap()  File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",line 236, in _bootstrap    self.run()  File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",line 93, in run    self._target(*self._args, **self._kwargs)  File"/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/rule_dispatcher.py",line 26, in main    initargs=(manager.agg_msgs_queue, ))  File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/__init__.py",line 232, in Pool    return Pool(processes, initializer, initargs)  File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",line 107, in __init__    w.start()  File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",line 109, in start    self._popen = Popen(self)  File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py",line 99, in __init__    code = process_obj._bootstrap()  File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",line 236, in _bootstrap    self.run()  File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",line 93, in run    self._target(*self._args, **self._kwargs)  File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",line 61, in worker    debug(''.join(traceback.format_stack()))DEBUG::PoolWorker-3:1::multiprocessing::worker got EOFError or IOError-- exitingINFO::PoolWorker-3:1::multiprocessing::process shutting downDEBUG::PoolWorker-3:1::multiprocessing::running all "atexit" finalizerswith priority >= 0
msg89194 -(view)Author: OG7 (OG7)Date: 2009-06-10 10:42
It seems the root cause is athttp://bugs.python.org/issue5155 .A workaround is to use a duplex Pipe in SimpleQueue.
msg89901 -(view)Author: Jesse Noller (jnoller)*(Python committer)Date: 2009-06-30 03:04
Patch attached toissue 5313, please review
msg89945 -(view)Author: Jesse Noller (jnoller)*(Python committer)Date: 2009-06-30 17:14
Committed inr73708 on trunk
History
DateUserActionArgs
2022-04-11 14:56:45adminsetgithub: 49581
2009-06-30 17:14:56jnollersetstatus: open -> closed
resolution: fixed
messages: +msg89945
2009-06-30 03:04:14jnollersetmessages: +msg89901
2009-06-10 10:42:31OG7setmessages: +msg89194
2009-06-10 08:58:09OG7setnosy: +OG7
messages: +msg89192
2009-03-29 14:38:33jnollersetpriority: normal
2009-02-21 01:53:21benjamin.petersonsetassignee:jnoller
nosy: +jnoller
2009-02-20 18:57:27LambertDWsetnosy: +LambertDW
messages: +msg82540
2009-02-20 16:06:50zzzeekcreate
Supported byThe Python Software Foundation,
Powered byRoundup
Copyright © 1990-2022,Python Software Foundation
Legal Statements

[8]ページ先頭

©2009-2026 Movatter.jp