Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork32k
Closed
Description
Bug report
Checklist
- I am confident this is a bug in CPython, not a bug in a third-party project
- I have searched theCPython issue tracker,
and am confident this bug has not been reported before
CPython versions tested on:
3.11
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.11.5 (main, Aug 26 2023, 00:26:34) [GCC 12.2.1 20220924]
A clear and concise description of the bug:
Using nestedmultiprocessing (i.e. spawn a child process inside a child process) is broken as of Python 3.11.5, leading to an attribute error.
ProcessProcess-1:Traceback (mostrecentcalllast):File"/usr/local/lib/python3.11/multiprocessing/process.py",line314,in_bootstrapself.run()File"/usr/local/lib/python3.11/multiprocessing/process.py",line108,inrunself._target(*self._args,**self._kwargs)File"/io/pyi_multiprocessing_nested_process.py",line15,inprocess_functionprocess.start()File"/usr/local/lib/python3.11/multiprocessing/process.py",line121,instartself._popen=self._Popen(self)^^^^^^^^^^^^^^^^^File"/usr/local/lib/python3.11/multiprocessing/context.py",line224,in_Popenreturn_default_context.get_context().Process._Popen(process_obj)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File"/usr/local/lib/python3.11/multiprocessing/context.py",line288,in_PopenreturnPopen(process_obj)^^^^^^^^^^^^^^^^^^File"/usr/local/lib/python3.11/multiprocessing/popen_spawn_posix.py",line32,in__init__super().__init__(process_obj)File"/usr/local/lib/python3.11/multiprocessing/popen_fork.py",line19,in__init__self._launch(process_obj)File"/usr/local/lib/python3.11/multiprocessing/popen_spawn_posix.py",line47,in_launchreduction.dump(process_obj,fp)File"/usr/local/lib/python3.11/multiprocessing/reduction.py",line60,indumpForkingPickler(file,protocol).dump(obj)File"/usr/local/lib/python3.11/multiprocessing/synchronize.py",line106,in__getstate__ifself.is_fork_ctx:^^^^^^^^^^^^^^^^AttributeError:'Lock'objecthasnoattribute'is_fork_ctx'Results: [1]
Minimal code example below. Invoke with argumentfork,forkserver orspawn.fork will work.forkserver andspawn will both raise the above error. All three variants work with Python 3.11.4.
importsysimportmultiprocessingdefnested_process_function(queue):print("Running nested sub-process!")queue.put(2)defprocess_function(queue):print("Running sub-process!")queue.put(1)process=multiprocessing.Process(target=nested_process_function,args=(queue,))process.start()process.join()defmain(start_method):multiprocessing.set_start_method(start_method)queue=multiprocessing.Queue()process=multiprocessing.Process(target=process_function,args=(queue,))process.start()process.join()results= []whilenotqueue.empty():results.append(queue.get())print(f"Results:{results}")assertresults== [1,2]if__name__=='__main__':iflen(sys.argv)!=2:raiseSystemExit(f"Usage:{sys.argv[0]} fork|forkserver|spawn")main(sys.argv[1])
I believe that the source of this regression is34ef75d which adds the attributeis_fork_ctx tomultiprocessing.Lock() but doesn't update the pickle methods (__getstate__() and__setstate__()) so after being serialised and deserialised, theLock() object looses that attribute.
The following patch, addingis_fork_ctx to the pickle methods, makes the above work again.
diff --git a/Lib/multiprocessing/synchronize.py b/Lib/multiprocessing/synchronize.pyindex 2328d33212..9c5c2aada6 100644--- a/Lib/multiprocessing/synchronize.py+++ b/Lib/multiprocessing/synchronize.py@@ -109,10 +109,11 @@ def __getstate__(self): 'not supported. Please use the same context to create ' 'multiprocessing objects and Process.') h = sl.handle- return (h, sl.kind, sl.maxvalue, sl.name)+ return (h, sl.kind, sl.maxvalue, sl.name, self.is_fork_ctx) def __setstate__(self, state):- self._semlock = _multiprocessing.SemLock._rebuild(*state)+ self._semlock = _multiprocessing.SemLock._rebuild(*state[:4])+ self.is_fork_ctx = state[4] util.debug('recreated blocker with handle %r' % state[0]) self._make_methods()
### Tasks