NotificationsYou must be signed in to change notification settings
Fork34k
Star71.3k

gh-75572: Speed up test_xpickle#144393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Jump to bottom

Merged

serhiy-storchaka merged 2 commits intopython:mainfromserhiy-storchaka:test_xpickle-speedup

Feb 2, 2026

Merged

gh-75572: Speed up test_xpickle#144393

serhiy-storchaka merged 2 commits intopython:mainfromserhiy-storchaka:test_xpickle-speedup

Feb 2, 2026

+95 −37

Conversation

Copy link

Member

serhiy-storchaka commentedFeb 2, 2026•
edited by bedevere-appbot
Loading

Run a long living subprocess which handles multiple requests instead of running a new subprocess for each request.

Issue:Forward-port test_xpickle from 2.7 to 3.x #75572

pythongh-75572: Speed up test_xpickle

c7bd567

Run a long living subprocess which handles multiple requests instead ofrunning a new subprocess for each request.

serhiy-storchaka added tests

Tests in the Lib/test dir

skip news needs backport to 3.13

bugs and security fixes

needs backport to 3.14bugs and security fixes labels

Feb 2, 2026

bedevere-appbot added the awaiting core review label

Feb 2, 2026

bedevere-appbot mentioned this pull request

Feb 2, 2026

Forward-port test_xpickle from 2.7 to 3.x#75572

Closed

vstinner reviewed

Feb 2, 2026

View reviewed changes

Lib/test/test_xpickle.py Outdated

		worker =cls.worker
		ifworkeris None:
		target = os.path.join(os.path.dirname(__file__), 'xpickle_worker.py')
		worker =subprocess.Popen([*python, target],

Copy link

Member

vstinnerFeb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why not calling start_worker() here?

Copy link

MemberAuthor

serhiy-storchakaFeb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Oh, this is a remnant. I was not sure whether it is worth to move this code out into a method.

Lib/test/test_xpickle.py Outdated

		cls.worker = worker

		@classmethod
		def close_worker(cls):

Copy link

Member

vstinnerFeb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

"stop_worker()" name would be better to have start/stop_worker methods.

Lib/test/test_xpickle.py

		stdout=subprocess.PIPE,
		stderr=subprocess.PIPE,
		# For windows bpo-17023.
		shell=is_windows)

Copy link

Member

vstinnerFeb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

You should usetext=True to fix this code below:

_,stderr=worker.communicate()raiseRuntimeError(stderr)

Copy link

MemberAuthor

serhiy-storchakaFeb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

But we do binary IO for stdin/stdout in normal case.

This code is only for troubleshooting. It is not executed when the tests pass.

Cleanup.

1081bff

vstinner reviewed

Feb 2, 2026

View reviewed changes

Copy link

Member

vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

./python -m test test_xpickle -u xpickle,cpu -v command takes:

main branch:24 min 38 sec
this PR:1 min 24 sec

Well, it's much faster with this PR :-)

Note: I measured timings on a debug build (gcc -Og).

vstinner approved these changes

Feb 2, 2026

View reviewed changes

Copy link

Member

vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM.

bedevere-appbot added awaiting merge and removed awaiting core review labels

Feb 2, 2026

Copy link

MemberAuthor

serhiy-storchaka commentedFeb 2, 2026

Note that you can run it only for the selected version, e.g.xpython=3.13. In debug build it is about 50% slower.

serhiy-storchaka merged commit29acc08 intopython:main

Feb 2, 2026

49 checks passed

Copy link

miss-islington-appbot commentedFeb 2, 2026

Thanks@serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

bedevere-appbot removed the awaiting merge label

Feb 2, 2026

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request

Feb 2, 2026

pythongh-75572: Speed up test_xpickle (pythonGH-144393)

d16d65c

Run a long living subprocess which handles multiple requests instead ofrunning a new subprocess for each request.(cherry picked from commit 29acc08c8dad664cd5713cb392e5beba65724c10)Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request

Feb 2, 2026

pythongh-75572: Speed up test_xpickle (pythonGH-144393)

8929009

Run a long living subprocess which handles multiple requests instead ofrunning a new subprocess for each request.(cherry picked from commit29acc08)Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

Copy link

bedevere-appbot commentedFeb 2, 2026

GH-144403 is a backport of this pull request to the3.14 branch.

bedevere-appbot removed the needs backport to 3.14bugs and security fixes label

Feb 2, 2026

serhiy-storchaka deleted the test_xpickle-speedup branch

February 2, 2026 18:00

Copy link

bedevere-appbot commentedFeb 2, 2026

GH-144404 is a backport of this pull request to the3.13 branch.

bedevere-appbot removed the needs backport to 3.13bugs and security fixes label

Feb 2, 2026

serhiy-storchaka added a commit that referenced this pull request

Feb 3, 2026

[3.14]gh-75572: Speed up test_xpickle (GH-144393) (GH-144403)

5af4e68

Run a long living subprocess which handles multiple requests instead ofrunning a new subprocess for each request.(cherry picked from commit29acc08)Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

serhiy-storchaka added a commit that referenced this pull request

Feb 3, 2026

[3.13]gh-75572: Speed up test_xpickle (GH-144393) (GH-144404)

9d76689

Run a long living subprocess which handles multiple requests instead ofrunning a new subprocess for each request.(cherry picked from commit29acc08)Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

Copy link

bedevere-bot commentedFeb 3, 2026

⚠️⚠️⚠️ Buildbot failure⚠️⚠️⚠️

Hi! The buildbotAMD64 Windows Server 2022 NoGIL 3.13 (tier-1) has failed when building commit9d76689.

What do you need to do:

Don't panic.
Checkthe buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1488/builds/1904) and take a look at the build logs.
Check if the failure is related to this commit (9d76689) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1488/builds/1904

Failed tests:

test_launcher

Failed subtests:

test_search_major_2 - test.test_launcher.TestLauncher.test_search_major_2
test_search_major_3 - test.test_launcher.TestLauncher.test_search_major_3

Summary of the results of the build (if available):

Click to see traceback logs

Traceback (most recent call last):  File"C:\Users\Administrator\buildarea\3.13.itamaro-win64-srv-22-aws.x64.nogil\build\Lib\test\test_launcher.py", line445, intest_search_major_3self.assertEqual("PythonCore", data["env.company"])~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^AssertionError:'PythonCore' != 'PythonTestSuite'- PythonCore+ PythonTestSuiteTraceback (most recent call last):  File"C:\Users\Administrator\buildarea\3.13.itamaro-win64-srv-22-aws.x64.nogil\build\Lib\test\test_launcher.py", line465, intest_search_major_2self.assertEqual("PythonCore", data["env.company"])~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^AssertionError:'PythonCore' != 'PythonTestSuite'- PythonCore+ PythonTestSuite

Copy link

bedevere-bot commentedFeb 3, 2026

⚠️⚠️⚠️ Buildbot failure⚠️⚠️⚠️

Hi! The buildbotAMD64 Debian root 3.13 (tier-1) has failed when building commit9d76689.

What do you need to do:

Don't panic.
Checkthe buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1441/builds/1761) and take a look at the build logs.
Check if the failure is related to this commit (9d76689) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1441/builds/1761

Summary of the results of the build (if available):

Click to see traceback logs

Traceback (most recent call last):  File"/root/buildarea/3.13.angelico-debian-amd64/build/Lib/test/support/__init__.py", line856, ingc_collect    gc.collect()ResourceWarning:unclosed file <_io.FileIO name=13 mode='wb' closefd=True>

Labels

skip news tests

Tests in the Lib/test dir

Movatterモバイル変換

Uh oh!

gh-75572: Speed up test_xpickle#144393

gh-75572: Speed up test_xpickle#144393

Conversation

serhiy-storchaka commentedFeb 2, 2026• edited by bedevere-appbotLoading Uh oh!There was an error while loading.Please reload this page.

Uh oh!

Uh oh!

vstinnerFeb 2, 2026

Choose a reason for hiding this comment

Uh oh!

serhiy-storchakaFeb 2, 2026

Choose a reason for hiding this comment

Uh oh!

vstinnerFeb 2, 2026

Choose a reason for hiding this comment

Uh oh!

vstinnerFeb 2, 2026

Choose a reason for hiding this comment

Uh oh!

serhiy-storchakaFeb 2, 2026

Choose a reason for hiding this comment

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka commentedFeb 2, 2026

Uh oh!

Uh oh!

miss-islington-appbot commentedFeb 2, 2026

Uh oh!

bedevere-appbot commentedFeb 2, 2026

Uh oh!

bedevere-appbot commentedFeb 2, 2026

Uh oh!

bedevere-bot commentedFeb 3, 2026

⚠️⚠️⚠️ Buildbot failure⚠️⚠️⚠️

Uh oh!

bedevere-bot commentedFeb 3, 2026

⚠️⚠️⚠️ Buildbot failure⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

serhiy-storchaka commentedFeb 2, 2026•
edited by bedevere-appbot
Loading