Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit62d79d8

Browse files
authored
Merge branch 'main' intogh-109719
2 parents6a5e675 +3e8fcb7 commit62d79d8

File tree

16 files changed

+277
-146
lines changed

16 files changed

+277
-146
lines changed

‎Doc/library/itertools.rst‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -845,7 +845,6 @@ which incur interpreter overhead.
845845

846846
def quantify(iterable, pred=bool):
847847
"Given a predicate that returns True or False, count the True results."
848-
"Count how many times the predicate is True"
849848
return sum(map(pred, iterable))
850849

851850
def all_equal(iterable):

‎Doc/whatsnew/3.12.rst‎

Lines changed: 23 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -291,9 +291,11 @@ can be used to customize buffer creation.
291291
PEP 684: A Per-Interpreter GIL
292292
------------------------------
293293

294-
Sub-interpreters may now be created with a unique GIL per interpreter.
294+
:pep:`684` introduces a per-interpreter:term:`GIL <global interpreter lock>`,
295+
so that sub-interpreters may now be created with a unique GIL per interpreter.
295296
This allows Python programs to take full advantage of multiple CPU
296-
cores.
297+
cores. This is currently only available through the C-API,
298+
though a Python API is:pep:`anticipated for 3.13 <554>`.
297299

298300
Use the new:c:func:`Py_NewInterpreterFromConfig` function to
299301
create an interpreter with its own GIL::
@@ -312,22 +314,22 @@ create an interpreter with its own GIL::
312314
For further examples how to use the C-API for sub-interpreters with a
313315
per-interpreter GIL, see:source:`Modules/_xxsubinterpretersmodule.c`.
314316

315-
A Python API is anticipated for 3.13. (See:pep:`554`.)
316-
317317
(Contributed by Eric Snow in:gh:`104210`, etc.)
318318

319319
.. _whatsnew312-pep669:
320320

321321
PEP 669: Low impact monitoring for CPython
322322
------------------------------------------
323323

324-
CPython 3.12 now supports the ability to monitor calls,
325-
returns, lines, exceptions and other events using instrumentation.
324+
:pep:`669` defines a new:mod:`API <sys.monitoring>` for profilers,
325+
debuggers, and other tools to monitor events in CPython.
326+
It covers a wide range of events, including calls,
327+
returns, lines, exceptions, jumps, and more.
326328
This means that you only pay for what you use, providing support
327329
for near-zero overhead debuggers and coverage tools.
328-
329330
See:mod:`sys.monitoring` for details.
330331

332+
(Contributed by Mark Shannon in:gh:`103083`.)
331333

332334
New Features Related to Type Hints
333335
==================================
@@ -459,12 +461,12 @@ and others in :gh:`103764`.)
459461
Other Language Changes
460462
======================
461463

462-
* Add:ref:`perf_profiling` through the new
463-
environment variable:envvar:`PYTHONPERFSUPPORT`,
464-
the new command-line option:option:`-X perf <-X>`,
464+
* Add:ref:`support for the perf profiler<perf_profiling>` through the new
465+
environment variable:envvar:`PYTHONPERFSUPPORT`
466+
and command-line option:option:`-X perf <-X>`,
465467
as well as the new:func:`sys.activate_stack_trampoline`,
466468
:func:`sys.deactivate_stack_trampoline`,
467-
and:func:`sys.is_stack_trampoline_active`APIs.
469+
and:func:`sys.is_stack_trampoline_active`functions.
468470
(Design by Pablo Galindo. Contributed by Pablo Galindo and Christian Heimes
469471
with contributions from Gregory P. Smith [Google] and Mark Shannon
470472
in:gh:`96123`.)
@@ -473,7 +475,7 @@ Other Language Changes
473475
have a new a *filter* argument that allows limiting tar features than may be
474476
surprising or dangerous, such as creating files outside the destination
475477
directory.
476-
See:ref:`tarfile-extraction-filter` for details.
478+
See:ref:`tarfile extraction filters<tarfile-extraction-filter>` for details.
477479
In Python 3.14, the default will switch to ``'data'``.
478480
(Contributed by Petr Viktorin in:pep:`706`.)
479481

@@ -501,8 +503,8 @@ Other Language Changes
501503
* A backslash-character pair that is not a valid escape sequence now generates
502504
a:exc:`SyntaxWarning`, instead of:exc:`DeprecationWarning`.
503505
For example, ``re.compile("\d+\.\d+")`` now emits a:exc:`SyntaxWarning`
504-
(``"\d"`` is an invalid escape sequence), use raw strings for regular
505-
expression: ``re.compile(r"\d+\.\d+")``.
506+
(``"\d"`` is an invalid escape sequence, use raw strings for regular
507+
expression: ``re.compile(r"\d+\.\d+")``).
506508
In a future Python version,:exc:`SyntaxError` will eventually be raised,
507509
instead of:exc:`SyntaxWarning`.
508510
(Contributed by Victor Stinner in:gh:`98401`.)
@@ -531,7 +533,7 @@ Other Language Changes
531533
when summing floats or mixed ints and floats.
532534
(Contributed by Raymond Hettinger in:gh:`100425`.)
533535

534-
* Exceptions raised in atypeobject's ``__set_name__`` method are no longer
536+
* Exceptions raised in aclass or type's ``__set_name__`` method are no longer
535537
wrapped by a:exc:`RuntimeError`. Context information is added to the
536538
exception as a:pep:`678` note. (Contributed by Irit Katriel in:gh:`77757`.)
537539

@@ -567,7 +569,7 @@ asyncio
567569
* Added:func:`asyncio.eager_task_factory` and:func:`asyncio.create_eager_task_factory`
568570
functions to allow opting an event loop in to eager task execution,
569571
making some use-cases 2x to 5x faster.
570-
(Contributed by Jacob Bower & ItamarO in:gh:`102853`,:gh:`104140`, and:gh:`104138`)
572+
(Contributed by Jacob Bower & ItamarOren in:gh:`102853`,:gh:`104140`, and:gh:`104138`)
571573

572574
* On Linux,:mod:`asyncio` uses:class:`asyncio.PidfdChildWatcher` by default
573575
if:func:`os.pidfd_open` is available and functional instead of
@@ -594,7 +596,7 @@ asyncio
594596
(Contributed by Kumar Aditya in:gh:`99388`.)
595597

596598
* Add C implementation of:func:`asyncio.current_task` for 4x-6x speedup.
597-
(Contributed by ItamarOstricher and Pranav Thulasiram Bhat in:gh:`100344`.)
599+
(Contributed by ItamarOren and Pranav Thulasiram Bhat in:gh:`100344`.)
598600

599601
*:func:`asyncio.iscoroutine` now returns ``False`` for generators as
600602
:mod:`asyncio` does not support legacy generator-based coroutines.
@@ -985,7 +987,7 @@ Optimizations
985987
(Contributed by Serhiy Storchaka in:gh:`91524`.)
986988

987989
* Speed up:class:`asyncio.Task` creation by deferring expensive string formatting.
988-
(Contributed by ItamarO in:gh:`103793`.)
990+
(Contributed by ItamarOren in:gh:`103793`.)
989991

990992
* The:func:`tokenize.tokenize` and:func:`tokenize.generate_tokens` functions are
991993
up to 64% faster as a side effect of the changes required to cover:pep:`701` in
@@ -1000,9 +1002,9 @@ Optimizations
10001002
CPython bytecode changes
10011003
========================
10021004

1003-
* Remove the:opcode:`LOAD_METHOD` instruction. It has been merged into
1005+
* Remove the:opcode:`!LOAD_METHOD` instruction. It has been merged into
10041006
:opcode:`LOAD_ATTR`.:opcode:`LOAD_ATTR` will now behave like the old
1005-
:opcode:`LOAD_METHOD` instruction if the low bit of its oparg is set.
1007+
:opcode:`!LOAD_METHOD` instruction if the low bit of its oparg is set.
10061008
(Contributed by Ken Jin in:gh:`93429`.)
10071009

10081010
* Remove the:opcode:`!JUMP_IF_FALSE_OR_POP` and:opcode:`!JUMP_IF_TRUE_OR_POP`
@@ -1838,7 +1840,7 @@ New Features
18381840
* Added:c:func:`PyCode_AddWatcher` and:c:func:`PyCode_ClearWatcher`
18391841
APIs to register callbacks to receive notification on creation and
18401842
destruction of code objects.
1841-
(Contributed by ItamarOstricher in:gh:`91054`.)
1843+
(Contributed by ItamarOren in:gh:`91054`.)
18421844

18431845
* Add:c:func:`PyFrame_GetVar` and:c:func:`PyFrame_GetVarString` functions to
18441846
get a frame variable by its name.

‎Lib/asyncio/subprocess.py‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -147,15 +147,17 @@ def kill(self):
147147

148148
asyncdef_feed_stdin(self,input):
149149
debug=self._loop.get_debug()
150-
ifinputisnotNone:
151-
self.stdin.write(input)
152-
ifdebug:
153-
logger.debug(
154-
'%r communicate: feed stdin (%s bytes)',self,len(input))
155150
try:
151+
ifinputisnotNone:
152+
self.stdin.write(input)
153+
ifdebug:
154+
logger.debug(
155+
'%r communicate: feed stdin (%s bytes)',self,len(input))
156+
156157
awaitself.stdin.drain()
157158
except (BrokenPipeError,ConnectionResetError)asexc:
158-
# communicate() ignores BrokenPipeError and ConnectionResetError
159+
# communicate() ignores BrokenPipeError and ConnectionResetError.
160+
# write() and drain() can raise these exceptions.
159161
ifdebug:
160162
logger.debug('%r communicate: stdin got %r',self,exc)
161163

‎Lib/concurrent/futures/process.py‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,11 @@ def __init__(self):
7171
self._reader,self._writer=mp.Pipe(duplex=False)
7272

7373
defclose(self):
74+
# Please note that we do not take the shutdown lock when
75+
# calling clear() (to avoid deadlocking) so this method can
76+
# only be called safely from the same thread as all calls to
77+
# clear() even if you hold the shutdown lock. Otherwise we
78+
# might try to read from the closed pipe.
7479
ifnotself._closed:
7580
self._closed=True
7681
self._writer.close()
@@ -426,8 +431,12 @@ def wait_result_broken_or_wakeup(self):
426431
elifwakeup_readerinready:
427432
is_broken=False
428433

429-
withself.shutdown_lock:
430-
self.thread_wakeup.clear()
434+
# No need to hold the _shutdown_lock here because:
435+
# 1. we're the only thread to use the wakeup reader
436+
# 2. we're also the only thread to call thread_wakeup.close()
437+
# 3. we want to avoid a possible deadlock when both reader and writer
438+
# would block (gh-105829)
439+
self.thread_wakeup.clear()
431440

432441
returnresult_item,is_broken,cause
433442

@@ -717,7 +726,10 @@ def __init__(self, max_workers=None, mp_context=None,
717726
# as it could result in a deadlock if a worker process dies with the
718727
# _result_queue write lock still acquired.
719728
#
720-
# _shutdown_lock must be locked to access _ThreadWakeup.
729+
# _shutdown_lock must be locked to access _ThreadWakeup.close() and
730+
# .wakeup(). Care must also be taken to not call clear or close from
731+
# more than one thread since _ThreadWakeup.clear() is not protected by
732+
# the _shutdown_lock
721733
self._executor_manager_thread_wakeup=_ThreadWakeup()
722734

723735
# Create communication channels for the executor

‎Lib/test/test_asyncio/test_subprocess.py‎

Lines changed: 42 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
importos
22
importsignal
33
importsys
4+
importtextwrap
45
importunittest
56
importwarnings
67
fromunittestimportmock
@@ -12,9 +13,14 @@
1213
fromtestimportsupport
1314
fromtest.supportimportos_helper
1415

15-
ifsys.platform!='win32':
16+
17+
MS_WINDOWS= (sys.platform=='win32')
18+
ifMS_WINDOWS:
19+
importmsvcrt
20+
else:
1621
fromasyncioimportunix_events
1722

23+
1824
ifsupport.check_sanitizer(address=True):
1925
raiseunittest.SkipTest("Exposes ASAN flakiness in GitHub CI")
2026

@@ -270,26 +276,43 @@ async def send_signal(proc):
270276
finally:
271277
signal.signal(signal.SIGHUP,old_handler)
272278

273-
defprepare_broken_pipe_test(self):
279+
deftest_stdin_broken_pipe(self):
274280
# buffer large enough to feed the whole pipe buffer
275281
large_data=b'x'*support.PIPE_MAX_SIZE
276282

283+
rfd,wfd=os.pipe()
284+
self.addCleanup(os.close,rfd)
285+
self.addCleanup(os.close,wfd)
286+
ifMS_WINDOWS:
287+
handle=msvcrt.get_osfhandle(rfd)
288+
os.set_handle_inheritable(handle,True)
289+
code=textwrap.dedent(f'''
290+
import os, msvcrt
291+
handle ={handle}
292+
fd = msvcrt.open_osfhandle(handle, os.O_RDONLY)
293+
os.read(fd, 1)
294+
''')
295+
fromsubprocessimportSTARTUPINFO
296+
startupinfo=STARTUPINFO()
297+
startupinfo.lpAttributeList= {"handle_list": [handle]}
298+
kwargs=dict(startupinfo=startupinfo)
299+
else:
300+
code=f'import os; fd ={rfd}; os.read(fd, 1)'
301+
kwargs=dict(pass_fds=(rfd,))
302+
277303
# the program ends before the stdin can be fed
278304
proc=self.loop.run_until_complete(
279305
asyncio.create_subprocess_exec(
280-
sys.executable,'-c','pass',
306+
sys.executable,'-c',code,
281307
stdin=subprocess.PIPE,
308+
**kwargs
282309
)
283310
)
284311

285-
return (proc,large_data)
286-
287-
deftest_stdin_broken_pipe(self):
288-
proc,large_data=self.prepare_broken_pipe_test()
289-
290312
asyncdefwrite_stdin(proc,data):
291-
awaitasyncio.sleep(0.5)
292313
proc.stdin.write(data)
314+
# Only exit the child process once the write buffer is filled
315+
os.write(wfd,b'go')
293316
awaitproc.stdin.drain()
294317

295318
coro=write_stdin(proc,large_data)
@@ -300,7 +323,16 @@ async def write_stdin(proc, data):
300323
self.loop.run_until_complete(proc.wait())
301324

302325
deftest_communicate_ignore_broken_pipe(self):
303-
proc,large_data=self.prepare_broken_pipe_test()
326+
# buffer large enough to feed the whole pipe buffer
327+
large_data=b'x'*support.PIPE_MAX_SIZE
328+
329+
# the program ends before the stdin can be fed
330+
proc=self.loop.run_until_complete(
331+
asyncio.create_subprocess_exec(
332+
sys.executable,'-c','pass',
333+
stdin=subprocess.PIPE,
334+
)
335+
)
304336

305337
# communicate() must ignore BrokenPipeError when feeding stdin
306338
self.loop.set_exception_handler(lambdaloop,msg:None)

‎Lib/test/test_concurrent_futures/test_deadlock.py‎

Lines changed: 71 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
importcontextlib
2+
importqueue
3+
importsignal
24
importsys
35
importtime
46
importunittest
7+
importunittest.mock
58
frompickleimportPicklingError
69
fromconcurrentimportfutures
7-
fromconcurrent.futures.processimportBrokenProcessPool
10+
fromconcurrent.futures.processimportBrokenProcessPool,_ThreadWakeup
811

912
fromtestimportsupport
1013

@@ -241,6 +244,73 @@ def test_crash_big_data(self):
241244

242245
executor.shutdown(wait=True)
243246

247+
deftest_gh105829_should_not_deadlock_if_wakeup_pipe_full(self):
248+
# Issue #105829: The _ExecutorManagerThread wakeup pipe could
249+
# fill up and block. See: https://github.com/python/cpython/issues/105829
250+
251+
# Lots of cargo culting while writing this test, apologies if
252+
# something is really stupid...
253+
254+
self.executor.shutdown(wait=True)
255+
256+
ifnothasattr(signal,'alarm'):
257+
raiseunittest.SkipTest(
258+
"Tested platform does not support the alarm signal")
259+
260+
deftimeout(_signum,_frame):
261+
importfaulthandler
262+
faulthandler.dump_traceback()
263+
264+
raiseRuntimeError("timed out while submitting jobs?")
265+
266+
thread_run=futures.process._ExecutorManagerThread.run
267+
defmock_run(self):
268+
# Delay thread startup so the wakeup pipe can fill up and block
269+
time.sleep(3)
270+
thread_run(self)
271+
272+
classMockWakeup(_ThreadWakeup):
273+
"""Mock wakeup object to force the wakeup to block"""
274+
def__init__(self):
275+
super().__init__()
276+
self._dummy_queue=queue.Queue(maxsize=1)
277+
278+
defwakeup(self):
279+
self._dummy_queue.put(None,block=True)
280+
super().wakeup()
281+
282+
defclear(self):
283+
try:
284+
whileTrue:
285+
self._dummy_queue.get_nowait()
286+
exceptqueue.Empty:
287+
super().clear()
288+
289+
with (unittest.mock.patch.object(futures.process._ExecutorManagerThread,
290+
'run',mock_run),
291+
unittest.mock.patch('concurrent.futures.process._ThreadWakeup',
292+
MockWakeup)):
293+
withself.executor_type(max_workers=2,
294+
mp_context=self.get_context())asexecutor:
295+
self.executor=executor# Allow clean up in fail_on_deadlock
296+
297+
job_num=100
298+
job_data=range(job_num)
299+
300+
# Need to use sigalarm for timeout detection because
301+
# Executor.submit is not guarded by any timeout (both
302+
# self._work_ids.put(self._queue_count) and
303+
# self._executor_manager_thread_wakeup.wakeup() might
304+
# timeout, maybe more?). In this specific case it was
305+
# the wakeup call that deadlocked on a blocking pipe.
306+
old_handler=signal.signal(signal.SIGALRM,timeout)
307+
try:
308+
signal.alarm(int(self.TIMEOUT))
309+
self.assertEqual(job_num,len(list(executor.map(int,job_data))))
310+
finally:
311+
signal.alarm(0)
312+
signal.signal(signal.SIGALRM,old_handler)
313+
244314

245315
create_executor_tests(globals(),ExecutorDeadlockTest,
246316
executor_mixins=(ProcessPoolForkMixin,

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp