Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commitff81b63

Browse files
committed
gh-89240: limit multiprocessing.Pool to 61 workers on windows
1 parentd1a89ce commitff81b63

File tree

4 files changed

+28
-0
lines changed

4 files changed

+28
-0
lines changed

‎Doc/library/multiprocessing.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2209,6 +2209,10 @@ with the :class:`Pool` class.
22092209

22102210
*processes* is the number of worker processes to use. If *processes* is
22112211
``None`` then the number returned by:func:`os.cpu_count` is used.
2212+
On Windows, *processes* must be equal or lower than ``61``. If it is not
2213+
then:exc:`ValueError` will be raised. If *processes* is ``None``, then
2214+
the default chosen will be at most ``61``, even if more processors are
2215+
available.
22122216

22132217
If *initializer* is not ``None`` then each worker process will call
22142218
``initializer(*initargs)`` when it starts.

‎Lib/multiprocessing/pool.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
importitertools
1818
importos
1919
importqueue
20+
importsys
2021
importthreading
2122
importtime
2223
importtraceback
@@ -175,6 +176,10 @@ class Pool(object):
175176
Class which supports an async version of applying functions to arguments.
176177
'''
177178
_wrap_exception=True
179+
# On Windows, WaitForMultipleObjects is used to wait for processes to
180+
# finish. It can wait on, at most, 64 objects. There is an overhead of three
181+
# objects.
182+
_MAX_WINDOWS_WORKERS=64-3
178183

179184
@staticmethod
180185
defProcess(ctx,*args,**kwds):
@@ -201,8 +206,12 @@ def __init__(self, processes=None, initializer=None, initargs=(),
201206

202207
ifprocessesisNone:
203208
processes=os.cpu_count()or1
209+
ifsys.platform=='win32':
210+
processes=min(processes,self._MAX_WINDOWS_WORKERS)
204211
ifprocesses<1:
205212
raiseValueError("Number of processes must be at least 1")
213+
ifsys.platform=='win32'andprocesses>self._MAX_WINDOWS_WORKERS:
214+
raiseValueError(f"max_workers must be <={self._MAX_WINDOWS_WORKERS}")
206215
ifmaxtasksperchildisnotNone:
207216
ifnotisinstance(maxtasksperchild,int)ormaxtasksperchild<=0:
208217
raiseValueError("maxtasksperchild must be a positive int or None")
@@ -920,6 +929,7 @@ def _set(self, i, obj):
920929

921930
classThreadPool(Pool):
922931
_wrap_exception=False
932+
_MAX_WINDOWS_WORKERS=float("inf")
923933

924934
@staticmethod
925935
defProcess(ctx,*args,**kwds):

‎Lib/test/_test_multiprocessing.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2772,6 +2772,18 @@ def test_resource_warning(self):
27722772
pool=None
27732773
support.gc_collect()
27742774

2775+
classTestPoolMaxWorkers(unittest.TestCase):
2776+
@unittest.skipUnless(sys.platform=='win32','Windows-only process limit')
2777+
deftest_max_workers_too_large(self):
2778+
withself.assertRaisesRegex(ValueError,"max_workers must be <= 61"):
2779+
multiprocessing.pool.Pool(62)
2780+
2781+
# ThreadPool have no limit.
2782+
p=multiprocessing.pool.ThreadPool(62)
2783+
p.close()
2784+
p.join()
2785+
2786+
27752787
defraising():
27762788
raiseKeyError("key")
27772789

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Limit:class:`processes` in ``multiprocessing.Pool`` to 61 to work around a
2+
WaitForMultipleObjects limitation.

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp