
This issue trackerhas been migrated toGitHub, and is currentlyread-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.
Created on2009-02-20 16:06 byzzzeek, last changed2022-04-11 14:56 byadmin. This issue is nowclosed.
| Messages (6) | |||
|---|---|---|---|
| msg82534 -(view) | Author: mike bayer (zzzeek)* | Date: 2009-02-20 16:06 | |
this occurs for me running on Mac OSX Leopard. The equivalent codeusing "processing" in python 2.5 works fine, which is how I found thisbug - my code hung when upgraded to 2.6. Basically initiating amultiprocessing.Pool inside of multiprocessing.Process hangs theapplication. Below is code which illustrates the issue on both py2.6and py3.0: from multiprocessing import Pool, Process import sys def f(x): return x*x def go(): pool = Pool(processes=4) x = pool.map(f, [1, 2, 3, 4, 5, 6, 7]) sys.stdout.write("result: %s\n" % x) def go2(): x = map(f, [1,2,3,4,5,6,7]) sys.stdout.write("result: %s\n" % x) # pool alone, fine go() # process alone, fine p = Process(target=go2) p.start() p.join() # use both, hangs p = Process(target=go) p.start() p.join() | |||
| msg82540 -(view) | Author: David W. Lambert (LambertDW) | Date: 2009-02-20 18:57 | |
Fails also on this system.$ p3Python 3.0.1 (r301:69556, Feb 13 2009, 23:52:55) [GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2(...)>>> # use both hangs(...)>>> p.start()>>> p.join()(hmm la do diii laaa...)^CTraceback (most recent call last): File "<stdin>", line 1, in <module> File"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/process.py",line 119, in join res = self._popen.wait(timeout) File"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/forking.py",line 118, in wait return self.poll(0) File"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/forking.py",line 107, in poll pid, sts = os.waitpid(self.pid, flag)KeyboardInterrupt>>> | |||
| msg89192 -(view) | Author: OG7 (OG7) | Date: 2009-06-10 08:58 | |
Apparently the pool workers die all at once, just after the pool createsthem and before the pool is used.I added a few lines to multiprocessing/pool.py to get the stack and theexception backtrace. except (EOFError, IOError): import traceback debug(traceback.format_exc()) debug(''.join(traceback.format_stack())) debug('worker got EOFError or IOError -- exiting') breakINFO::Rule dispatcher::multiprocessing::child process calling self.run()DEBUG::Rule dispatcher::multiprocessing::created semlock with handle3082559488DEBUG::Rule dispatcher::multiprocessing::created semlock with handle3082104832DEBUG::Rule dispatcher::multiprocessing::created semlock with handle3081826304DEBUG::Rule dispatcher::multiprocessing::created semlock with handle3081822208INFO::PoolWorker-3:1::multiprocessing::child process calling self.run()DEBUG::PoolWorker-3:1::multiprocessing::Traceback (most recent call last): File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",line 57, in worker task = get() File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/queues.py",line 339, in get return recv()IOError: [Errno 9] Bad file descriptorDEBUG::PoolWorker-3:1::multiprocessing:: File"/home/who/var/co/git-svn/what-base/correlator/bin/correlator", line 30,in <module> what.corr.actors.main.main() File"/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/main.py",line 47, in main rrp.start() File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",line 109, in start self._popen = Popen(self) File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py",line 99, in __init__ code = process_obj._bootstrap() File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",line 236, in _bootstrap self.run() File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",line 93, in run self._target(*self._args, **self._kwargs) File"/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/rule_dispatcher.py",line 26, in main initargs=(manager.agg_msgs_queue, )) File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/__init__.py",line 232, in Pool return Pool(processes, initializer, initargs) File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",line 107, in __init__ w.start() File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",line 109, in start self._popen = Popen(self) File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py",line 99, in __init__ code = process_obj._bootstrap() File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",line 236, in _bootstrap self.run() File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",line 93, in run self._target(*self._args, **self._kwargs) File"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",line 61, in worker debug(''.join(traceback.format_stack()))DEBUG::PoolWorker-3:1::multiprocessing::worker got EOFError or IOError-- exitingINFO::PoolWorker-3:1::multiprocessing::process shutting downDEBUG::PoolWorker-3:1::multiprocessing::running all "atexit" finalizerswith priority >= 0 | |||
| msg89194 -(view) | Author: OG7 (OG7) | Date: 2009-06-10 10:42 | |
It seems the root cause is athttp://bugs.python.org/issue5155 .A workaround is to use a duplex Pipe in SimpleQueue. | |||
| msg89901 -(view) | Author: Jesse Noller (jnoller)*![]() | Date: 2009-06-30 03:04 | |
Patch attached toissue 5313, please review | |||
| msg89945 -(view) | Author: Jesse Noller (jnoller)*![]() | Date: 2009-06-30 17:14 | |
Committed inr73708 on trunk | |||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:56:45 | admin | set | github: 49581 |
| 2009-06-30 17:14:56 | jnoller | set | status: open -> closed resolution: fixed messages: +msg89945 |
| 2009-06-30 03:04:14 | jnoller | set | messages: +msg89901 |
| 2009-06-10 10:42:31 | OG7 | set | messages: +msg89194 |
| 2009-06-10 08:58:09 | OG7 | set | nosy: +OG7 messages: +msg89192 |
| 2009-03-29 14:38:33 | jnoller | set | priority: normal |
| 2009-02-21 01:53:21 | benjamin.peterson | set | assignee:jnoller nosy: +jnoller |
| 2009-02-20 18:57:27 | LambertDW | set | nosy: +LambertDW messages: +msg82540 |
| 2009-02-20 16:06:50 | zzzeek | create | |