Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit0ca67e6

Browse files
authored
GH-84559: Deprecate fork being the multiprocessing default. (#100618)
This starts the process. Users who don't specify their own start methodand use the default on platforms where it is 'fork' will see aDeprecationWarning upon multiprocessing.Pool() construction or uponmultiprocessing.Process.start() or concurrent.futures.ProcessPool use.See the related issue and documentation within this change for details.
1 parent618b7a8 commit0ca67e6

16 files changed

+284
-63
lines changed

‎Doc/library/concurrent.futures.rst

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -250,9 +250,10 @@ to a :class:`ProcessPoolExecutor` will result in deadlock.
250250
then:exc:`ValueError` will be raised. If *max_workers* is ``None``, then
251251
the default chosen will be at most ``61``, even if more processors are
252252
available.
253-
*mp_context* can be a multiprocessing context or None. It will be used to
254-
launch the workers. If *mp_context* is ``None`` or not given, the default
255-
multiprocessing context is used.
253+
*mp_context* can be a:mod:`multiprocessing` context or ``None``. It will be
254+
used to launch the workers. If *mp_context* is ``None`` or not given, the
255+
default:mod:`multiprocessing` context is used.
256+
See:ref:`multiprocessing-start-methods`.
256257

257258
*initializer* is an optional callable that is called at the start of
258259
each worker process; *initargs* is a tuple of arguments passed to the
@@ -284,6 +285,13 @@ to a :class:`ProcessPoolExecutor` will result in deadlock.
284285
The *max_tasks_per_child* argument was added to allow users to
285286
control the lifetime of workers in the pool.
286287

288+
..versionchanged::3.12
289+
The implicit use of the:mod:`multiprocessing` *fork* start method as a
290+
platform default (see:ref:`multiprocessing-start-methods`) now raises a
291+
:exc:`DeprecationWarning`. The default will change in Python 3.14.
292+
Code that requires *fork* should explicitly specify that when creating
293+
their:class:`ProcessPoolExecutor` by passing a
294+
``mp_context=multiprocessing.get_context('fork')`` parameter.
287295

288296
.. _processpoolexecutor-example:
289297

‎Doc/library/multiprocessing.rst

Lines changed: 64 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ offers both local and remote concurrency, effectively side-stepping the
1919
:term:`Global Interpreter Lock <global interpreter lock>` by using
2020
subprocesses instead of threads. Due
2121
to this, the:mod:`multiprocessing` module allows the programmer to fully
22-
leverage multiple processors on a given machine. It runs on bothUnix and
22+
leverage multiple processors on a given machine. It runs on bothPOSIX and
2323
Windows.
2424

2525
The:mod:`multiprocessing` module also introduces APIs which do not have
@@ -99,11 +99,11 @@ necessary, see :ref:`multiprocessing-programming`.
9999

100100

101101

102+
.. _multiprocessing-start-methods:
103+
102104
Contexts and start methods
103105
~~~~~~~~~~~~~~~~~~~~~~~~~~
104106

105-
.. _multiprocessing-start-methods:
106-
107107
Depending on the platform,:mod:`multiprocessing` supports three ways
108108
to start a process. These *start methods* are
109109

@@ -115,7 +115,7 @@ to start a process. These *start methods* are
115115
will not be inherited. Starting a process using this method is
116116
rather slow compared to using *fork* or *forkserver*.
117117

118-
Available onUnix and Windows. The default on Windows and macOS.
118+
Available onPOSIX and Windows platforms. The default on Windows and macOS.
119119

120120
*fork*
121121
The parent process uses:func:`os.fork` to fork the Python
@@ -124,32 +124,39 @@ to start a process. These *start methods* are
124124
inherited by the child process. Note that safely forking a
125125
multithreaded process is problematic.
126126

127-
Available onUnix only.Thedefault onUnix.
127+
Available onPOSIX systems.Currently thedefault onPOSIX except macOS.
128128

129129
*forkserver*
130130
When the program starts and selects the *forkserver* start method,
131-
a server process isstarted. From then on, whenever a new process
131+
a server process isspawned. From then on, whenever a new process
132132
is needed, the parent process connects to the server and requests
133-
that it fork a new process. The fork server process is single
134-
threaded so it is safe for it to use:func:`os.fork`. No
135-
unnecessary resources are inherited.
133+
that it fork a new process. The fork server process is single threaded
134+
unless system libraries or preloaded imports spawn threads as a
135+
side-effect so it is generally safe for it to use:func:`os.fork`.
136+
No unnecessary resources are inherited.
136137

137-
Available on Unix platforms which support passing file descriptors
138-
over Unix pipes.
138+
Available on POSIX platforms which support passing file descriptors
139+
over Unix pipes such as Linux.
140+
141+
..versionchanged::3.12
142+
Implicit use of the *fork* start method as the default now raises a
143+
:exc:`DeprecationWarning`. Code that requires it should explicitly
144+
specify *fork* via:func:`get_context` or:func:`set_start_method`.
145+
The default will change away from *fork* in 3.14.
139146

140147
..versionchanged::3.8
141148

142149
On macOS, the *spawn* start method is now the default. The *fork* start
143150
method should be considered unsafe as it can lead to crashes of the
144-
subprocess. See:issue:`33725`.
151+
subprocess as macOS system libraries may start threads. See:issue:`33725`.
145152

146153
..versionchanged::3.4
147-
*spawn* added on allUnix platforms, and *forkserver* added for
148-
someUnix platforms.
154+
*spawn* added on allPOSIX platforms, and *forkserver* added for
155+
somePOSIX platforms.
149156
Child processes no longer inherit all of the parents inheritable
150157
handles on Windows.
151158

152-
OnUnix using the *spawn* or *forkserver* start methods will also
159+
OnPOSIX using the *spawn* or *forkserver* start methods will also
153160
start a *resource tracker* process which tracks the unlinked named
154161
system resources (such as named semaphores or
155162
:class:`~multiprocessing.shared_memory.SharedMemory` objects) created
@@ -211,10 +218,10 @@ library user.
211218

212219
..warning::
213220

214-
The ``'spawn'`` and ``'forkserver'`` start methodscannot currently
221+
The ``'spawn'`` and ``'forkserver'`` start methodsgenerally cannot
215222
be used with "frozen" executables (i.e., binaries produced by
216-
packages like **PyInstaller** and **cx_Freeze**) onUnix.
217-
The ``'fork'`` start methoddoes work.
223+
packages like **PyInstaller** and **cx_Freeze**) onPOSIX systems.
224+
The ``'fork'`` start methodmay work if code does not use threads.
218225

219226

220227
Exchanging objects between processes
@@ -629,14 +636,14 @@ The :mod:`multiprocessing` package mostly replicates the API of the
629636
calling:meth:`join()` is simpler.
630637

631638
On Windows, this is an OS handle usable with the ``WaitForSingleObject``
632-
and ``WaitForMultipleObjects`` family of API calls. OnUnix, this is
639+
and ``WaitForMultipleObjects`` family of API calls. OnPOSIX, this is
633640
a file descriptor usable with primitives from the:mod:`select` module.
634641

635642
..versionadded::3.3
636643

637644
..method::terminate()
638645

639-
Terminate the process. OnUnix this is done using the ``SIGTERM`` signal;
646+
Terminate the process. OnPOSIX this is done using the ``SIGTERM`` signal;
640647
on Windows:c:func:`TerminateProcess` is used. Note that exit handlers and
641648
finally clauses, etc., will not be executed.
642649

@@ -653,7 +660,7 @@ The :mod:`multiprocessing` package mostly replicates the API of the
653660

654661
..method::kill()
655662

656-
Same as:meth:`terminate()` but using the ``SIGKILL`` signal onUnix.
663+
Same as:meth:`terminate()` but using the ``SIGKILL`` signal onPOSIX.
657664

658665
..versionadded::3.7
659666

@@ -676,16 +683,17 @@ The :mod:`multiprocessing` package mostly replicates the API of the
676683
..doctest::
677684

678685
>>>import multiprocessing, time, signal
679-
>>>p= multiprocessing.Process(target=time.sleep,args=(1000,))
686+
>>>mp_context= multiprocessing.get_context('spawn')
687+
>>>p= mp_context.Process(target=time.sleep,args=(1000,))
680688
>>>print(p, p.is_alive())
681-
<Process ... initial> False
689+
<...Process ... initial> False
682690
>>>p.start()
683691
>>>print(p, p.is_alive())
684-
<Process ... started> True
692+
<...Process ... started> True
685693
>>>p.terminate()
686694
>>>time.sleep(0.1)
687695
>>>print(p, p.is_alive())
688-
<Process ... stopped exitcode=-SIGTERM> False
696+
<...Process ... stopped exitcode=-SIGTERM> False
689697
>>>p.exitcode==-signal.SIGTERM
690698
True
691699

@@ -815,7 +823,7 @@ For an example of the usage of queues for interprocess communication see
815823
Return the approximate size of the queue. Because of
816824
multithreading/multiprocessing semantics, this number is not reliable.
817825

818-
Note that this may raise:exc:`NotImplementedError` onUnixplatforms like
826+
Note that this may raise:exc:`NotImplementedError` on platforms like
819827
macOS where ``sem_getvalue()`` is not implemented.
820828

821829
..method::empty()
@@ -1034,9 +1042,8 @@ Miscellaneous
10341042

10351043
Returns a list of the supported start methods, the first of which
10361044
is the default. The possible start methods are ``'fork'``,
1037-
``'spawn'`` and ``'forkserver'``. On Windows only ``'spawn'`` is
1038-
available. On Unix ``'fork'`` and ``'spawn'`` are always
1039-
supported, with ``'fork'`` being the default.
1045+
``'spawn'`` and ``'forkserver'``. Not all platforms support all
1046+
methods. See:ref:`multiprocessing-start-methods`.
10401047

10411048
..versionadded::3.4
10421049

@@ -1048,7 +1055,7 @@ Miscellaneous
10481055
If *method* is ``None`` then the default context is returned.
10491056
Otherwise *method* should be ``'fork'``, ``'spawn'``,
10501057
``'forkserver'``.:exc:`ValueError` is raised if the specified
1051-
start method is not available.
1058+
start method is not available. See:ref:`multiprocessing-start-methods`.
10521059

10531060
..versionadded::3.4
10541061

@@ -1062,8 +1069,7 @@ Miscellaneous
10621069
is true then ``None`` is returned.
10631070

10641071
The return value can be ``'fork'``, ``'spawn'``, ``'forkserver'``
1065-
or ``None``. ``'fork'`` is the default on Unix, while ``'spawn'`` is
1066-
the default on Windows and macOS.
1072+
or ``None``. See:ref:`multiprocessing-start-methods`.
10671073

10681074
..versionchanged::3.8
10691075

@@ -1084,11 +1090,26 @@ Miscellaneous
10841090
before they can create child processes.
10851091

10861092
..versionchanged::3.4
1087-
Now supported onUnix when the ``'spawn'`` start method is used.
1093+
Now supported onPOSIX when the ``'spawn'`` start method is used.
10881094

10891095
..versionchanged::3.11
10901096
Accepts a:term:`path-like object`.
10911097

1098+
..function::set_forkserver_preload(module_names)
1099+
1100+
Set a list of module names for the forkserver main process to attempt to
1101+
import so that their already imported state is inherited by forked
1102+
processes. Any:exc:`ImportError` when doing so is silently ignored.
1103+
This can be used as a performance enhancement to avoid repeated work
1104+
in every process.
1105+
1106+
For this to work, it must be called before the forkserver process has been
1107+
launched (before creating a:class:`Pool` or starting a:class:`Process`).
1108+
1109+
Only meaningful when using the ``'forkserver'`` start method.
1110+
1111+
..versionadded::3.4
1112+
10921113
..function::set_start_method(method, force=False)
10931114

10941115
Set the method which should be used to start child processes.
@@ -1102,6 +1123,8 @@ Miscellaneous
11021123
protected inside the ``if __name__ == '__main__'`` clause of the
11031124
main module.
11041125

1126+
See:ref:`multiprocessing-start-methods`.
1127+
11051128
..versionadded::3.4
11061129

11071130
..note::
@@ -1906,7 +1929,8 @@ their parent process exits. The manager classes are defined in the
19061929

19071930
..doctest::
19081931

1909-
>>>manager= multiprocessing.Manager()
1932+
>>>mp_context= multiprocessing.get_context('spawn')
1933+
>>>manager= mp_context.Manager()
19101934
>>>Global= manager.Namespace()
19111935
>>>Global.x=10
19121936
>>>Global.y='hello'
@@ -2018,8 +2042,8 @@ the proxy). In this way, a proxy can be used just like its referent can:
20182042

20192043
..doctest::
20202044

2021-
>>>from multiprocessingimport Manager
2022-
>>>manager= Manager()
2045+
>>>mp_context= multiprocessing.get_context('spawn')
2046+
>>>manager=mp_context.Manager()
20232047
>>>l= manager.list([i*ifor iinrange(10)])
20242048
>>>print(l)
20252049
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
@@ -2520,7 +2544,7 @@ multiple connections at the same time.
25202544
*timeout* is ``None`` then it will block for an unlimited period.
25212545
A negative timeout is equivalent to a zero timeout.
25222546

2523-
For bothUnix and Windows, an object can appear in *object_list* if
2547+
For bothPOSIX and Windows, an object can appear in *object_list* if
25242548
it is
25252549

25262550
* a readable:class:`~multiprocessing.connection.Connection` object;
@@ -2531,7 +2555,7 @@ multiple connections at the same time.
25312555
A connection or socket object is ready when there is data available
25322556
to be read from it, or the other end has been closed.
25332557

2534-
**Unix**: ``wait(object_list, timeout)`` almost equivalent
2558+
**POSIX**: ``wait(object_list, timeout)`` almost equivalent
25352559
``select.select(object_list, [], [], timeout)``. The difference is
25362560
that, if:func:`select.select` is interrupted by a signal, it can
25372561
raise:exc:`OSError` with an error number of ``EINTR``, whereas
@@ -2803,7 +2827,7 @@ Thread safety of proxies
28032827

28042828
Joining zombie processes
28052829

2806-
OnUnix when a process finishes but has not been joined it becomes a zombie.
2830+
OnPOSIX when a process finishes but has not been joined it becomes a zombie.
28072831
There should never be very many because each time a new process starts (or
28082832
:func:`~multiprocessing.active_children` is called) all completed processes
28092833
which have not yet been joined will be joined. Also calling a finished
@@ -2866,7 +2890,7 @@ Joining processes that use queues
28662890

28672891
Explicitly pass resources to child processes
28682892

2869-
OnUnix using the *fork* start method, a child process can make
2893+
OnPOSIX using the *fork* start method, a child process can make
28702894
use of a shared resource created in a parent process using a
28712895
global resource. However, it is better to pass the object as an
28722896
argument to the constructor for the child process.

‎Doc/whatsnew/3.12.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -440,6 +440,11 @@ Deprecated
440440
warning at compile time. This field will be removed in Python 3.14.
441441
(Contributed by Ramvikrams and Kumar Aditya in:gh:`101193`. PEP by Ken Jin.)
442442

443+
* Use of the implicit default ``'fork'`` start method for
444+
:mod:`multiprocessing` and:class:`concurrent.futures.ProcessPoolExecutor`
445+
now emits a:exc:`DeprecationWarning` on Linux and other non-macOS POSIX
446+
systems. Avoid this by explicitly specifying a start method.
447+
See:ref:`multiprocessing-start-methods`.
443448

444449
Pending Removal in Python 3.13
445450
------------------------------
@@ -505,6 +510,9 @@ Pending Removal in Python 3.14
505510
* Testing the truth value of an:class:`xml.etree.ElementTree.Element`
506511
is deprecated and will raise an exception in Python 3.14.
507512

513+
* The default:mod:`multiprocessing` start method will change to one of either
514+
``'forkserver'`` or ``'spawn'`` on all platforms for which ``'fork'`` remains
515+
the default per:gh:`84559`.
508516

509517
Pending Removal in Future Versions
510518
----------------------------------

‎Lib/compileall.py

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,9 +97,15 @@ def compile_dir(dir, maxlevels=None, ddir=None, force=False,
9797
files=_walk_dir(dir,quiet=quiet,maxlevels=maxlevels)
9898
success=True
9999
ifworkers!=1andProcessPoolExecutorisnotNone:
100+
importmultiprocessing
101+
ifmultiprocessing.get_start_method()=='fork':
102+
mp_context=multiprocessing.get_context('forkserver')
103+
else:
104+
mp_context=None
100105
# If workers == 0, let ProcessPoolExecutor choose
101106
workers=workersorNone
102-
withProcessPoolExecutor(max_workers=workers)asexecutor:
107+
withProcessPoolExecutor(max_workers=workers,
108+
mp_context=mp_context)asexecutor:
103109
results=executor.map(partial(compile_file,
104110
ddir=ddir,force=force,
105111
rx=rx,quiet=quiet,

‎Lib/concurrent/futures/process.py

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@
5757
importitertools
5858
importsys
5959
fromtracebackimportformat_exception
60+
importwarnings
6061

6162

6263
_threads_wakeups=weakref.WeakKeyDictionary()
@@ -616,9 +617,9 @@ def __init__(self, max_workers=None, mp_context=None,
616617
max_workers: The maximum number of processes that can be used to
617618
execute the given calls. If None or not given then as many
618619
worker processes will be created as the machine has processors.
619-
mp_context: A multiprocessing context to launch the workers. This
620-
object should provide SimpleQueue, Queue and Process. Useful
621-
to allow specific multiprocessing start methods.
620+
mp_context: A multiprocessing context to launch the workers created
621+
using the multiprocessing.get_context('start method') API. This
622+
object should provide SimpleQueue, Queue and Process.
622623
initializer: A callable used to initialize worker processes.
623624
initargs: A tuple of arguments to pass to the initializer.
624625
max_tasks_per_child: The maximum number of tasks a worker process
@@ -650,6 +651,22 @@ def __init__(self, max_workers=None, mp_context=None,
650651
mp_context=mp.get_context("spawn")
651652
else:
652653
mp_context=mp.get_context()
654+
if (mp_context.get_start_method()=="fork"and
655+
mp_context==mp.context._default_context._default_context):
656+
warnings.warn(
657+
"The default multiprocessing start method will change "
658+
"away from 'fork' in Python >= 3.14, per GH-84559. "
659+
"ProcessPoolExecutor uses multiprocessing. "
660+
"If your application requires the 'fork' multiprocessing "
661+
"start method, explicitly specify that by passing a "
662+
"mp_context= parameter. "
663+
"The safest start method is 'spawn'.",
664+
category=mp.context.DefaultForkDeprecationWarning,
665+
stacklevel=2,
666+
)
667+
# Avoid the equivalent warning from multiprocessing itself via
668+
# a non-default fork context.
669+
mp_context=mp.get_context("fork")
653670
self._mp_context=mp_context
654671

655672
# https://github.com/python/cpython/issues/90622

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp