Bug report

Bug description:

From@corona10:

importconcurrent.futuresfromconcurrent.futuresimportInterpreterPoolExecutor,ThreadPoolExecutordeffib(n):ifn<2:returnnreturnfib(n-1)+fib(n-2)withInterpreterPoolExecutor(max_workers=5)asexecutor:futures= {executor.submit(fib,n):nforninrange(10)}forfutureinconcurrent.futures.as_completed(futures):n=futures[future]data=future.result()print(f"fib({n}):{data}")

This fails with:

Exception: AttributeError: module '__main__' has no attribute 'fib'The above exception was the direct cause of the following exception:concurrent.interpreters.NotShareableError: object could not be unpickledThe above exception was the direct cause of the following exception:...

The failure goes away if the "executable" part of the script is wrapped in anif __name__ == '__main__': block.

This is because of howInterpreter.call() sends functions between interpreters. It mostly pickles them. On the other side, functions are looked up by name in the defining module. For the__main__ module, this is a problem.

The original script only runs in the main interpreter, so the function will not be found in the other interpreter's__main__ module. We must run the script in the other interpreter, but without polluting__main__ and avoiding executing anything but the desired function definition. We partially solve this by callingrunpy.run_path() with a different module name than__main__.

multiprocessing faces a similar situation. See the "Safe importing of main module" entry in thehttps://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods section.

The solution here is probably the same: document that users should applyif __name__ == '__main__':. Wemay also be able to accommodate this in the code somewhat, since we know when we are running the script in the other interpreter. However, there's a limit to how much we can do so without introducing nontrivial complexity.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response