Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.7k
Description
Bug report
multiprocessing.managers usesconvert_to_error(kind, result) to make a raisable exception out ofresult when a call has responded with some sort of error. Ifkind == "#ERROR", thenresult is already an exception and the caller raises it directly—but becauseresult was created in a frame at or under the caller, this creates a reference cycleresult →result.__traceback__ → (some frame).f_locals['result'].
In particular, every time I've used a manager queue I've expected frequent occurrences ofqueue.Empty, and the buildup of reference cycles sporadically wakes up the garbage collector and wrecks my hopes of consistent latency.
I'm including an example script below. PR coming in a moment, so please let me know if I should expand the example into a test and bundle that in. (Please also feel free to tell me if this is a misuse of queue.Empty and I should buzz off.)
Your environment
- CPython versions tested on: 3.11.3
- Operating system and architecture:
uname -asaysLinux delia 6.3.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 11 May 2023 16:40:42 +0000 x86_64 GNU/Linux
Minimal example
Output
net allocations: 0got from queue: [0, 1, 2, 3, 4]net allocations: 23garbage produced: Counter({<class 'traceback'>: 3, <class 'frame'>: 3, <class 'list'>: 1, <class '_queue.Empty'>: 1})net allocations: 0Script
#!/usr/bin/env pythonimportcollectionsimportgcimportmultiprocessingimportqueueimporttimedefsender(q):foriinrange(5):q.put_nowait(i)defget_all_available(q):result= []try:whileTrue:result.append(q.get_nowait())exceptqueue.Empty: ...returnresultdefmain():q=multiprocessing.Manager().Queue()p=multiprocessing.Process(target=sender,args=(q,))p.start()# take control of gcgc.disable()gc.collect()gc.set_debug(gc.DEBUG_SAVEALL)time.sleep(0.1)# just in case the new process took a while to createprint('net allocations: ',gc.get_count()[0])# trigger a queue.Emptyprint('got from queue: ',get_all_available(q))# check for collectable garbage and print itprint('net allocations: ',gc.get_count()[0])gc.collect()print('garbage produced:',collections.Counter(type(x)forxingc.garbage))gc.set_debug(0)gc.garbage.clear()gc.collect()print('net allocations: ',gc.get_count()[0])# clean upp.join()if__name__=='__main__':main()