Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.7k
Description
Bug report
Bug description:
Background
I've been working to build from source on Alpine. Starting in 3.13 pgo optimizations now error and that led to the test_re tests failing and blocking builds. I didn't want to fully disable pgo optimizations and miss out on all of the related performance wins.
Attempted Fix and Problem Identification
I initially attempted to preclude this by leveragingPROFILE_TASK="-m test --pgo -x test_re" when calling make. This had the impact of making all testsexcept the pgo tests run. After some spelunking I found that this logic fromLib/test/libregrtest/main.py is the culprit.
deffind_tests(self,tests:TestList|None=None)->tuple[TestTuple,TestList|None]:iftestsisNone:tests= []ifself.single_test_run:self.next_single_filename=os.path.join(self.tmp_dir,'pynexttest')try:withopen(self.next_single_filename,'r')asfp:next_test=fp.read().strip()tests= [next_test]exceptOSError:passifself.fromfile:tests= []# regex to match 'test_builtin' in line:# '0:00:00 [ 4/400] test_builtin -- test_dict took 1 sec'regex=re.compile(r'\btest_[a-zA-Z0-9_]+\b')withopen(os.path.join(os_helper.SAVEDCWD,self.fromfile))asfp:forlineinfp:line=line.split('#',1)[0]line=line.strip()match=regex.search(line)ifmatchisnotNone:tests.append(match.group())strip_py_suffix(tests)ifself.pgo:# add default PGO tests if no tests are specifiedsetup_pgo_tests(self.cmdline_args,self.pgo_extended)ifself.tsan:setup_tsan_tests(self.cmdline_args)ifself.tsan_parallel:setup_tsan_parallel_tests(self.cmdline_args)exclude_tests=set()ifself.exclude:forarginself.cmdline_args:exclude_tests.add(arg)self.cmdline_args= []alltests=findtests(testdir=self.test_dir,exclude=exclude_tests)ifnotself.fromfile:selected=testsorself.cmdline_argsifselected:selected=split_test_packages(selected)else:selected=alltestselse:selected=testsifself.single_test_run:selected=selected[:1]try:pos=alltests.index(selected[0])self.next_single_test=alltests[pos+1]exceptIndexError:pass# Remove all the selected tests that precede start if it's set.ifself.starting_test:try:delselected[:selected.index(self.starting_test)]exceptValueError:print(f"Cannot find starting test:{self.starting_test}")sys.exit(1)random.seed(self.random_seed)ifself.randomize:random.shuffle(selected)forpriority_testinreversed(self.prioritize_tests):try:selected.remove(priority_test)exceptValueError:print(f"warning: --prioritize={priority_test} used"f" but test not actually selected")continueelse:selected.insert(0,priority_test)return (tuple(selected),tests)
Due to the order of operations what ends up happening is the pgo test suite gets added tocmdline_args and then they all get excluded leaving everything else to run.
Hacky Workaround
My eventual workaround is... less than great. I ultimately settled on two choices: explicitly maintain my own copy of the PGO test list in my dockerfile or update the list defined in the testing framework. I opted for the later and have added this to my build script.
sed -Ei"s/'test_re',?$//g" Lib/test/libregrtest/pgo.pyThis felt like the less likely option to break over time as we not only build 3.13 but other versions as well.
Suggested Improvement Options
1. Automatic Exclusion for Musl:
Automatically exclude incompatible tests on musl-based systems, which appears consistent with other test behaviors. This minimizes friction for maintainers who otherwise may forgo PGO optimizations.
From my view, this is the preferred option.
2. Improved Test Exclusion Logic:
Adjust--pgo and-x flag logic to allow intuitive exclusion of PGO tests, or otherwise enable this behavior, and combine with clearer documentation for builders.
3. Explicit Error Messaging:
Explicitly error when these flags are used together to reduce confusion and accidental behaviors. This aligns with other checks for mutually exclusive flags.
CPython versions tested on:
3.13
Operating systems tested on:
Other