Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork32k
Description
Bug report
Bug description:
I must admit that this issue only happens in very specific conditions, namely when using the multiprocessing lib on a system where the temp folder is set to a long path by the user. One could argue this is a user error, but bear with me a second.
The problem is inmultiprocessing.util.get_temp_dir
defget_temp_dir():# get name of a temp directory which will be automatically cleaned uptempdir=process.current_process()._config.get('tempdir')# This could be set to an arbitrary long path by usersiftempdirisNone:importshutil,tempfiletempdir=tempfile.mkdtemp(prefix='pymp-')info('created temp directory %s',tempdir)# keep a strong reference to shutil.rmtree(), since the finalizer# can be called late during Python shutdownFinalize(None,_remove_temp_dir,args=(shutil.rmtree,tempdir),exitpriority=-100)process.current_process()._config['tempdir']=tempdirreturntempdir
This function is used inmultiprocessing.connect.arbitrary_address
defarbitrary_address(family):''' Return an arbitrary free address for the given family '''iffamily=='AF_INET':return ('localhost',0)eliffamily=='AF_UNIX':returntempfile.mktemp(prefix='listener-',dir=util.get_temp_dir())eliffamily=='AF_PIPE':returntempfile.mktemp(prefix=r'\\.\pipe\pyc-%d-%d-'% (os.getpid(),next(_mmap_counter)),dir="")else:raiseValueError('unrecognized family')
Where this is problematic is that arbitrary_address is used inmultiprocessing.forkserver.Forkserver.ensure_running
which uses the address provided by arbitrary_address to create and bind to a socket
withsocket.socket(socket.AF_UNIX)aslistener:address=connection.arbitrary_address('AF_UNIX')listener.bind(address)ifnotutil.is_abstract_socket_namespace(address):os.chmod(address,0o600)listener.listen()
Since UNIX sockets path have a limited character count between 92 and 108 [1], it would make sense for the std lib to have safe guards against generating paths that are too long to begin with. Sure, users can use work arounds such as setting TMP_DIR="/tmp" [2] but this requires each and every user of the multiprocessing library to be aware of such limitations.
I propose we fix the problem once and for all at the source, by checking the size of the path created bytempfile.mktemp(prefix='listener-', dir=util.get_temp_dir())
and should it be greater or equal to the max length supported by the platform, reverting to whatget_temp_dir()
does whentempdir=None
[1]https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_un.h.html#tag_13_67_04
[2]https://patchwork.ozlabs.org/project/qemu-devel/patch/20220722182508.89761-2-peter@pjd.dev/#2938322
CPython versions tested on:
3.11
Operating systems tested on:
Linux