Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit79f7a4c

Browse files
vstinnertiranzwaresobolevnzitterbewegung
authored
[3.11]gh-108822: Backport libregrtest changes from the main branch (#108820)
* Revert "[3.11]gh-101634: regrtest reports decoding error as failed test (#106169) (#106175)"This reverts commitd5418e9.* Revert "[3.11]bpo-46523: fix tests rerun when `setUp[Class|Module]` fails (GH-30895) (GH-103342)"This reverts commitecb09a8.* Revert "gh-95027: Fix regrtest stdout encoding on Windows (GH-98492)"This reverts commitb2aa28e.* Revert "[3.11]gh-94026: Buffer regrtest worker stdout in temporary file (GH-94253) (GH-94408)"This reverts commit0122ab2.* Revert "Run Tools/scripts/reindent.py (GH-94225)"This reverts commitf0f3a42.* Revert "gh-94052: Don't re-run failed tests with --python option (GH-94054)"This reverts commit1347607.* Revert "[3.11]gh-84461: Fix Emscripten umask and permission issues (GH-94002) (GH-94006)"This reverts commit1073184.*gh-93353: regrtest checks for leaked temporary files (#93776)When running tests with -jN, create a temporary directory per processand mark a test as "environment changed" if a test leaks a temporaryfile or directory.(cherry picked from commite566ce5)*gh-93353: Fix regrtest for -jN with N >= 2 (GH-93813)(cherry picked from commit36934a1)*gh-93353: regrtest supports checking tmp files with -j2 (#93909)regrtest now also implements checking for leaked temporary files anddirectories when using -jN for N >= 2. Use tempfile.mkdtemp() tocreate the temporary directory. Skip this check on WASI.(cherry picked from commit4f85cec)*gh-84461: Fix Emscripten umask and permission issues (GH-94002)- Emscripten's default umask is too strict, seeemscripten-core/emscripten#17269- getuid/getgid and geteuid/getegid are stubs that always return 0 (root). Disable effective uid/gid syscalls and fix tests that use chmod() current user.- Cannot drop X bit from directory.(cherry picked from commit2702e40)*gh-94052: Don't re-run failed tests with --python option (#94054)(cherry picked from commit0ff7b99)* Run Tools/scripts/reindent.py (#94225)Reindent files which were not properly formatted (PEP 8: 4 spaces).Remove also some trailing spaces.(cherry picked from commite87ada4)*gh-94026: Buffer regrtest worker stdout in temporary file (GH-94253)Co-authored-by: Victor Stinner <vstinner@python.org>(cherry picked from commit199ba23)*gh-96465: Clear fractions hash lru_cache under refleak testing (GH-96689)Automerge-Triggered-By: GH:zware(cherry picked from commit9c8f379)*gh-95027: Fix regrtest stdout encoding on Windows (#98492)On Windows, when the Python test suite is run with the -jN option,the ANSI code page is now used as the encoding for the stdouttemporary file, rather than using UTF-8 which can lead to decodingerrors.(cherry picked from commitec1f6f5)*gh-98903: Test suite fails with exit code 4 if no tests ran (#98904)The Python test suite now fails wit exit code 4 if no tests ran. Itshould help detecting typos in test names and test methods.* Add "EXITCODE_" constants to Lib/test/libregrtest/main.py.* Fix a typo: "NO TEST RUN" becomes "NO TESTS RAN"(cherry picked from commitc76db37)*gh-100086: Add build info to test.libregrtest (#100093)The Python test runner (libregrtest) now logs Python build information like"debug" vs "release" build, or LTO and PGO optimizations.(cherry picked from commit3c89202)*bpo-46523: fix tests rerun when `setUp[Class|Module]` fails (#30895)Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>Co-authored-by: Łukasz Langa <lukasz@langa.pl>(cherry picked from commit9953860)*gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. (#103927)This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests otherwise run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel.After porting we can see the direct impact on a multicore system.Without this change: Running make test is 5 min 26 secondsWith this change: Running make test takes 3 min 39 secondsThat'll vary based on system and parallelism. On a `-j 4` run similar to what CI and buildbot systems often do, it reduced the overall test suite completion latency by 10%.The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the wall time saved it is worth it, especially in low-core count CI runs as it pulls a long tail. The win for productivity and reserved CI resource usage is significant.Future tests that deserve to be refactored into split up suites to benefit from are test_concurrent_futures and the way the _test_multiprocessing suite gets run for all start methods. As exposed by passing the -o flag to python -m test to get a list of the 10 longest running tests.---------Co-authored-by: Carl Meyer <carl@oddbird.net>Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google, LLC](cherry picked from commit9e011e7)* Display the sanitizer config in the regrtest header. (#105301)Display the sanitizers present in libregrtest.Having this in the CI output for tests with the relevant environmentvariable displayed will help make it easier to do what we need tocreate an equivalent local test run.(cherry picked from commit852348a)*gh-101634: regrtest reports decoding error as failed test (#106169)When running the Python test suite with -jN option, if a worker stdoutcannot be decoded from the locale encoding report a failed testn so theexitcode is non-zero.(cherry picked from commit2ac3eec)*gh-108223: test.pythoninfo and libregrtest log Py_NOGIL (#108238)Enable with --disable-gil --without-pydebug: $ make pythoninfo|grep NOGIL sysconfig[Py_NOGIL]: 1 $ ./python -m test ... == Python build: nogil debug ...(cherry picked from commit5afe0c1)*gh-90791: test.pythoninfo logs ASAN_OPTIONS env var (#108289)* Cleanup libregrtest code logging ASAN_OPTIONS.* Fix a typo on "ASAN_OPTIONS" vs "MSAN_OPTIONS".(cherry picked from commit3a1ac87)*gh-108388: regrtest splits test_asyncio package (#108393)Currently, test_asyncio package is only splitted into sub-tests whenusing command "./python -m test". With this change, it's alsosplitted when passing it on the command line:"./python -m test test_asyncio".Remove the concept of "STDTESTS". Python is now mature enough to nothave to bother with that anymore. Removing STDTESTS simplify thecode.(cherry picked from commit174e9da)* regrtest computes statistics (#108793)test_netrc, test_pep646_syntax and test_xml_etree now return resultsin the test_main() function.Changes:* Rewrite TestResult as a dataclass with a new State class.* Add test.support.TestStats class and Regrtest.stats_dict attribute.* libregrtest.runtest functions now modify a TestResult instance in-place.* libregrtest summary lists the number of run tests and skipped tests, and denied resources.* Add TestResult.has_meaningful_duration() method.* Compute TestResult duration in the upper function.* Use time.perf_counter() instead of time.monotonic().* Regrtest: rename 'resource_denieds' attribute to 'resource_denied'.* Rename CHILD_ERROR to MULTIPROCESSING_ERROR.* Use match/case syntadx to have different code depending on the test state.Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>(cherry picked from commitd4e534c)*gh-108822: Add Changelog entry for regrtest statistics (#108821)---------Co-authored-by: Christian Heimes <christian@python.org>Co-authored-by: Zachary Ware <zach@python.org>Co-authored-by: Nikita Sobolev <mail@sobolevn.me>Co-authored-by: Joshua Herman <zitterbewegung@gmail.com>Co-authored-by: Gregory P. Smith <greg@krypto.org>
1 parentba47d87 commit79f7a4c

18 files changed

+812
-375
lines changed

‎Lib/test/libregrtest/main.py‎

Lines changed: 133 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -11,14 +11,14 @@
1111
importunittest
1212
fromtest.libregrtest.cmdlineimport_parse_args
1313
fromtest.libregrtest.runtestimport (
14-
findtests,runtest,get_abs_module,is_failed,
15-
STDTESTS,NOTTESTS,PROGRESS_MIN_TIME,
16-
Passed,Failed,EnvChanged,Skipped,ResourceDenied,Interrupted,
17-
ChildError,DidNotRun)
14+
findtests,split_test_packages,runtest,get_abs_module,
15+
PROGRESS_MIN_TIME,State)
1816
fromtest.libregrtest.setupimportsetup_tests
1917
fromtest.libregrtest.pgoimportsetup_pgo_tests
20-
fromtest.libregrtest.utilsimportremovepy,count,format_duration,printlist
18+
fromtest.libregrtest.utilsimport (removepy,count,format_duration,
19+
printlist,get_build_info)
2120
fromtestimportsupport
21+
fromtest.supportimportTestStats
2222
fromtest.supportimportos_helper
2323
fromtest.supportimportthreading_helper
2424

@@ -77,13 +77,14 @@ def __init__(self):
7777
self.good= []
7878
self.bad= []
7979
self.skipped= []
80-
self.resource_denieds= []
80+
self.resource_denied= []
8181
self.environment_changed= []
8282
self.run_no_tests= []
8383
self.need_rerun= []
8484
self.rerun= []
8585
self.first_result=None
8686
self.interrupted=False
87+
self.stats_dict:dict[str,TestStats]= {}
8788

8889
# used by --slow
8990
self.test_times= []
@@ -92,7 +93,7 @@ def __init__(self):
9293
self.tracer=None
9394

9495
# used to display the progress bar "[ 3/100]"
95-
self.start_time=time.monotonic()
96+
self.start_time=time.perf_counter()
9697
self.test_count=''
9798
self.test_count_width=1
9899

@@ -110,36 +111,41 @@ def __init__(self):
110111

111112
defget_executed(self):
112113
return (set(self.good)|set(self.bad)|set(self.skipped)
113-
|set(self.resource_denieds)|set(self.environment_changed)
114+
|set(self.resource_denied)|set(self.environment_changed)
114115
|set(self.run_no_tests))
115116

116117
defaccumulate_result(self,result,rerun=False):
117-
test_name=result.name
118-
119-
ifnotisinstance(result, (ChildError,Interrupted))andnotrerun:
120-
self.test_times.append((result.duration_sec,test_name))
121-
122-
ifisinstance(result,Passed):
123-
self.good.append(test_name)
124-
elifisinstance(result,ResourceDenied):
125-
self.skipped.append(test_name)
126-
self.resource_denieds.append(test_name)
127-
elifisinstance(result,Skipped):
128-
self.skipped.append(test_name)
129-
elifisinstance(result,EnvChanged):
130-
self.environment_changed.append(test_name)
131-
elifisinstance(result,Failed):
132-
ifnotrerun:
133-
self.bad.append(test_name)
134-
self.need_rerun.append(result)
135-
elifisinstance(result,DidNotRun):
136-
self.run_no_tests.append(test_name)
137-
elifisinstance(result,Interrupted):
138-
self.interrupted=True
139-
else:
140-
raiseValueError("invalid test result: %r"%result)
118+
test_name=result.test_name
119+
120+
ifresult.has_meaningful_duration()andnotrerun:
121+
self.test_times.append((result.duration,test_name))
141122

142-
ifrerunandnotisinstance(result, (Failed,Interrupted)):
123+
matchresult.state:
124+
caseState.PASSED:
125+
self.good.append(test_name)
126+
caseState.ENV_CHANGED:
127+
self.environment_changed.append(test_name)
128+
caseState.SKIPPED:
129+
self.skipped.append(test_name)
130+
caseState.RESOURCE_DENIED:
131+
self.skipped.append(test_name)
132+
self.resource_denied.append(test_name)
133+
caseState.INTERRUPTED:
134+
self.interrupted=True
135+
caseState.DID_NOT_RUN:
136+
self.run_no_tests.append(test_name)
137+
case _:
138+
ifresult.is_failed(self.ns.fail_env_changed):
139+
ifnotrerun:
140+
self.bad.append(test_name)
141+
self.need_rerun.append(result)
142+
else:
143+
raiseValueError(f"invalid test state:{state!r}")
144+
145+
ifresult.statsisnotNone:
146+
self.stats_dict[result.test_name]=result.stats
147+
148+
ifrerunandnot(result.is_failed(False)orresult.state==State.INTERRUPTED):
143149
self.bad.remove(test_name)
144150

145151
xml_data=result.xml_data
@@ -161,7 +167,7 @@ def log(self, line=''):
161167
line=f"load avg:{load_avg:.2f}{line}"
162168

163169
# add the timestamp prefix: "0:01:05 "
164-
test_time=time.monotonic()-self.start_time
170+
test_time=time.perf_counter()-self.start_time
165171

166172
mins,secs=divmod(int(test_time),60)
167173
hours,mins=divmod(mins,60)
@@ -245,26 +251,23 @@ def find_tests(self, tests):
245251
# add default PGO tests if no tests are specified
246252
setup_pgo_tests(self.ns)
247253

248-
stdtests=STDTESTS[:]
249-
nottests=NOTTESTS.copy()
254+
exclude=set()
250255
ifself.ns.exclude:
251256
forarginself.ns.args:
252-
ifarginstdtests:
253-
stdtests.remove(arg)
254-
nottests.add(arg)
257+
exclude.add(arg)
255258
self.ns.args= []
256259

257-
# if testdir is set, then we are not running the python tests suite, so
258-
# don't add default tests to be executed or skipped (pass empty values)
259-
ifself.ns.testdir:
260-
alltests=findtests(self.ns.testdir,list(),set())
261-
else:
262-
alltests=findtests(self.ns.testdir,stdtests,nottests)
260+
alltests=findtests(testdir=self.ns.testdir,exclude=exclude)
263261

264262
ifnotself.ns.fromfile:
265-
self.selected=self.testsorself.ns.argsoralltests
263+
self.selected=self.testsorself.ns.args
264+
ifself.selected:
265+
self.selected=split_test_packages(self.selected)
266+
else:
267+
self.selected=alltests
266268
else:
267269
self.selected=self.tests
270+
268271
ifself.ns.single:
269272
self.selected=self.selected[:1]
270273
try:
@@ -339,7 +342,7 @@ def rerun_failed_tests(self):
339342
rerun_list=list(self.need_rerun)
340343
self.need_rerun.clear()
341344
forresultinrerun_list:
342-
test_name=result.name
345+
test_name=result.test_name
343346
self.rerun.append(test_name)
344347

345348
errors=result.errorsor []
@@ -366,7 +369,7 @@ def rerun_failed_tests(self):
366369

367370
self.accumulate_result(result,rerun=True)
368371

369-
ifisinstance(result,Interrupted):
372+
ifresult.state==State.INTERRUPTED:
370373
break
371374

372375
ifself.bad:
@@ -463,7 +466,7 @@ def run_tests_sequential(self):
463466

464467
previous_test=None
465468
fortest_index,test_nameinenumerate(self.tests,1):
466-
start_time=time.monotonic()
469+
start_time=time.perf_counter()
467470

468471
text=test_name
469472
ifprevious_test:
@@ -482,14 +485,14 @@ def run_tests_sequential(self):
482485
result=runtest(self.ns,test_name)
483486
self.accumulate_result(result)
484487

485-
ifisinstance(result,Interrupted):
488+
ifresult.state==State.INTERRUPTED:
486489
break
487490

488491
previous_test=str(result)
489-
test_time=time.monotonic()-start_time
492+
test_time=time.perf_counter()-start_time
490493
iftest_time>=PROGRESS_MIN_TIME:
491494
previous_test="%s in %s"% (previous_test,format_duration(test_time))
492-
elifisinstance(result,Passed):
495+
elifresult.state==State.PASSED:
493496
# be quiet: say nothing if the test passed shortly
494497
previous_test=None
495498

@@ -498,7 +501,7 @@ def run_tests_sequential(self):
498501
ifmodulenotinsave_modulesandmodule.startswith("test."):
499502
support.unload(module)
500503

501-
ifself.ns.failfastandis_failed(result,self.ns):
504+
ifself.ns.failfastandresult.is_failed(self.ns.fail_env_changed):
502505
break
503506

504507
ifprevious_test:
@@ -518,22 +521,53 @@ def display_header(self):
518521
print("==",platform.python_implementation(),*sys.version.split())
519522
print("==",platform.platform(aliased=True),
520523
"%s-endian"%sys.byteorder)
524+
print("== Python build:",' '.join(get_build_info()))
521525
print("== cwd:",os.getcwd())
522526
cpu_count=os.cpu_count()
523527
ifcpu_count:
524528
print("== CPU count:",cpu_count)
525529
print("== encodings: locale=%s, FS=%s"
526530
% (locale.getencoding(),sys.getfilesystemencoding()))
531+
self.display_sanitizers()
532+
533+
defdisplay_sanitizers(self):
534+
# This makes it easier to remember what to set in your local
535+
# environment when trying to reproduce a sanitizer failure.
536+
asan=support.check_sanitizer(address=True)
537+
msan=support.check_sanitizer(memory=True)
538+
ubsan=support.check_sanitizer(ub=True)
539+
sanitizers= []
540+
ifasan:
541+
sanitizers.append("address")
542+
ifmsan:
543+
sanitizers.append("memory")
544+
ifubsan:
545+
sanitizers.append("undefined behavior")
546+
ifnotsanitizers:
547+
return
548+
549+
print(f"== sanitizers:{', '.join(sanitizers)}")
550+
forsanitizer,env_varin (
551+
(asan,"ASAN_OPTIONS"),
552+
(msan,"MSAN_OPTIONS"),
553+
(ubsan,"UBSAN_OPTIONS"),
554+
):
555+
options=os.environ.get(env_var)
556+
ifsanitizerandoptionsisnotNone:
557+
print(f"=={env_var}={options!r}")
558+
559+
defno_tests_run(self):
560+
returnnotany((self.good,self.bad,self.skipped,self.interrupted,
561+
self.environment_changed))
527562

528563
defget_tests_result(self):
529564
result= []
530565
ifself.bad:
531566
result.append("FAILURE")
532567
elifself.ns.fail_env_changedandself.environment_changed:
533568
result.append("ENV CHANGED")
534-
elifnotany((self.good,self.bad,self.skipped,self.interrupted,
535-
self.environment_changed)):
536-
result.append("NO TEST RUN")
569+
elifself.no_tests_run():
570+
result.append("NO TESTS RAN")
537571

538572
ifself.interrupted:
539573
result.append("INTERRUPTED")
@@ -609,13 +643,48 @@ def finalize(self):
609643
coverdir=self.ns.coverdir)
610644

611645
print()
612-
duration=time.monotonic()-self.start_time
613-
print("Total duration: %s"%format_duration(duration))
614-
print("Tests result: %s"%self.get_tests_result())
646+
self.display_summary()
615647

616648
ifself.ns.runleaks:
617649
os.system("leaks %d"%os.getpid())
618650

651+
defdisplay_summary(self):
652+
duration=time.perf_counter()-self.start_time
653+
654+
# Total duration
655+
print("Total duration: %s"%format_duration(duration))
656+
657+
# Total tests
658+
total=TestStats()
659+
forstatsinself.stats_dict.values():
660+
total.accumulate(stats)
661+
stats= [f'run={total.tests_run:,}']
662+
iftotal.failures:
663+
stats.append(f'failures={total.failures:,}')
664+
iftotal.skipped:
665+
stats.append(f'skipped={total.skipped:,}')
666+
print(f"Total tests:{' '.join(stats)}")
667+
668+
# Total test files
669+
report= [f'success={len(self.good)}']
670+
ifself.bad:
671+
report.append(f'failed={len(self.bad)}')
672+
ifself.environment_changed:
673+
report.append(f'env_changed={len(self.environment_changed)}')
674+
ifself.skipped:
675+
report.append(f'skipped={len(self.skipped)}')
676+
ifself.resource_denied:
677+
report.append(f'resource_denied={len(self.resource_denied)}')
678+
ifself.rerun:
679+
report.append(f'rerun={len(self.rerun)}')
680+
ifself.run_no_tests:
681+
report.append(f'run_no_tests={len(self.run_no_tests)}')
682+
print(f"Total test files:{' '.join(report)}")
683+
684+
# Result
685+
result=self.get_tests_result()
686+
print(f"Result:{result}")
687+
619688
defsave_xml_result(self):
620689
ifnotself.ns.xmlpathandnotself.testsuite_xml:
621690
return
@@ -782,11 +851,13 @@ def _main(self, tests, kwargs):
782851
self.save_xml_result()
783852

784853
ifself.bad:
785-
sys.exit(2)
854+
sys.exit(EXITCODE_BAD_TEST)
786855
ifself.interrupted:
787-
sys.exit(130)
856+
sys.exit(EXITCODE_INTERRUPTED)
788857
ifself.ns.fail_env_changedandself.environment_changed:
789-
sys.exit(3)
858+
sys.exit(EXITCODE_ENV_CHANGED)
859+
ifself.no_tests_run():
860+
sys.exit(EXITCODE_NO_TESTS_RAN)
790861
sys.exit(0)
791862

792863

‎Lib/test/libregrtest/refleak.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -83,11 +83,12 @@ def get_pooled_int(value):
8383
print(("1234567890"*(repcount//10+1))[:repcount],file=sys.stderr,
8484
flush=True)
8585

86+
results=None
8687
dash_R_cleanup(fs,ps,pic,zdc,abcs)
8788
support.gc_collect()
8889

8990
foriinrep_range:
90-
test_func()
91+
results=test_func()
9192

9293
dash_R_cleanup(fs,ps,pic,zdc,abcs)
9394
support.gc_collect()
@@ -146,7 +147,7 @@ def check_fd_deltas(deltas):
146147
print(msg,file=refrep)
147148
refrep.flush()
148149
failed=True
149-
returnfailed
150+
return(failed,results)
150151

151152

152153
defdash_R_cleanup(fs,ps,pic,zdc,abcs):

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp