See previous discussions:

• issueos.process_cpu_count(): get the number CPUs usable by the current process #109649
• issueSupport python -Xcpu_count=<n> feature for container environment. #109595
• PRgh-109595: Add -Xcpu_count=<n> cmdline for container users #109667
• PRgh-109649: Add affinity parameter to os.cpu_count() #109652

Adding a new process_cpu_count() makes it easier to extend the function later for new use cases, instead of modifying the existing os.cpu_count():

• Maybe read Linux cgroups
• Maybe add apid parameter to get the CPU count of another process.
• -X cpu_count=value option
• Future new exciting stuff :-)

About thepid parameter. First, I would like to implement this function on Windows (get process affinity), and then check if it would be possible to get the CPU affinity of another process on Windows, before considering to add apid parameter.

In psutil, this function is called Process.cpu_affinity():https://psutil.readthedocs.io/en/latest/#psutil.Process.cpu_affinity

I prefer adding a newos.process_cpu_count() function, instead of adding aprocess=True parameter to the existingos.cpu_count() function because:

• I consider adding apid optional argument toos.process_cpu_count()
• IMO the word "process" makes it explicit that the result is specific to the process, wheras cpu_count() issystem-wide

I will take a look at it by tomorrow!

cc@gpshead@indygreg

@gpshead: I updated my PR to reimplement os.process_cpu_count() in Python.

cpu_set=sched_getaffinity(0)num_cpus=cpu_count()returnmin(len(cpu_set),num_cpus)ifcpu_setelsenum_cpus

Why do you want to return cpu_count() if it's smaller than len(sched_getaffinity(0))? It should not happen. If it happens, I would prefer to not workaround bugs, but fix the OS instead. For me, it should not happen.

sched_getaffinity() is basically a "mask" on the list of all available CPUs. It cannot be larger, unless cpu_count() ... counts CPUs differently. In that case, we should fix cpu_count() instead, no?

When I addedtime.monotonic(), I added an assertion in debug mode to make sure that... the clock is monotonic (doesn't go backward). Surprise, surprise. The clock is not. On some virtual machines, I saw it jumping backwards sometimes. Honestly. I just removed the assertion. Python cannot and should not work around OS bugs, but just expose them.Maybe for the worst known OS bugs, we might detect them and issue a warning. But I'm not even sure if it's worth it.

I recall that Python detects broken poll() implementation on macOS. In that case, we go further: weremove the function at runtime! (at Python startup)

I prefer to not make assumptions about hypothetical bugs, but wait until we get real concrete bug reports, and then decide how to deal with them.

Anyway, thanks for thinking about all corner cases, it's a good thing!

Why do you want to return cpu_count() if it's smaller than len(sched_getaffinity(0))? It should not happen. If it happens, I would prefer to not workaround bugs, but fix the OS instead. For me, it should not happen

If we pass the -Xcpu_count=limited_number, it will be a more efficient implementation, we have to overrideos.cpu_count only. (It will automatically overrideos.process_cpu_count)

If we pass the -Xcpu_count=limited_number, it will be a more efficient implementation, we have to override os.cpu_count only. (It will automatically override os.process_cpu_count)

I would prefer that the discussion separately the implementation and the expected behavior.

Also, can we discussion -Xcpu_count later? This PR is about adding process_cpu_count(), nothing more.

@gpshead is right, the number of CPUs is "per thread" in thread.

Example:

importosimportthreadingdefsched_remove_one_cpu():mask=os.sched_getaffinity(0)mask.pop()os.sched_setaffinity(0,mask)defset_affinity():tid=threading.get_native_id()print(f"thread{tid}: remove a second CPU")sched_remove_one_cpu()print(f"thread{tid}:{os.process_cpu_count()=}")tid=threading.get_native_id()print(f"main thread{tid}: remove a CPU")sched_remove_one_cpu()print(f"main thread{tid} before:{os.cpu_count()=}")print(f"main thread{tid} before:{os.process_cpu_count()=}")print()thread=threading.Thread(target=set_affinity)thread.start()thread.join()print()print(f"main thread{tid} after:{os.cpu_count()=}")print(f"main thread{tid} after:{os.process_cpu_count()=}")

Output on Linux:

main thread 2059677: remove a CPUmain thread 2059677 before: os.cpu_count()=12main thread 2059677 before: os.process_cpu_count()=11thread 2059678: remove a second CPUthread 2059678: os.process_cpu_count()=10main thread 2059677 after: os.cpu_count()=12main thread 2059677 after: os.process_cpu_count()=11

You can see that the main thread loses a CPU when sched_remove_one_cpu() is called: os.process_cpu_count() is affected, but os.cpu_count() is not affected.

When a thread removes a second CPU, it affects os.process_cpu_count() in the thread,but the second CPU removal does not affected the main thread.

New thread inherit the mask of the caller thread, but a change in a child thread does not affect the parent thread. Well, that's the expected behavior.

I made a few further changes to clarify differences between cpu_count(), process_cpu_count() and sched_affinity().

Tests / Hypothesis tests on Ubuntu (pull_request) Failing after 6m

FAIL: test_find_periodic_pattern (test.test_userstring.UserStringTest.test_find_periodic_pattern) (p='', text='')

I created issueGH-110160 for this bug.

Unrelated CI failures:

• Windows x86:FAIL: test_kbhit (test.test_msvcrt.TestConsoleIO.test_kbhit):test_msvcrt: test_getwche() failed with timeout (10 min) on GHA Windows x64 #110147 (comment)
• Windows x86:FAIL: test_timeout (test.test_multiprocessing_spawn.test_manager.WithManagerTestBarrier.test_timeout):test_multiprocessing_spawn: test_timeout() failed on GHA Windows x86 #110162
• Hypothesis tests on Ubuntu:FAIL: test_find_periodic_pattern (test.test_userstring.UserStringTest.test_find_periodic_pattern) (p='', text=''):test_userstring: test_find_periodic_pattern() failed on GHA Hypothesis tests on Ubuntu #110160

Merged, thanks for reviews@gpshead and@corona10!

⚠️⚠️⚠️ Buildbot failure⚠️⚠️⚠️

Hi! The buildbotaarch64 RHEL8 3.x has failed when building commitc815210.

What do you need to do:

1. Don't panic.
2. Checkthe buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/529/builds/5071) and take a look at the build logs.
4. Check if the failure is related to this commit (c815210) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/529/builds/5071

Summary of the results of the build (if available):

==

Traceback (most recent call last):  File"/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/threading.py", line1066, in_bootstrap_innerself.run()  File"/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/threading.py", line1003, inrunself._target(*self._args,**self._kwargs)  File"/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/test_interpreters.py", line484, intask    interp= interpreters.create()^^^^^^^^^^^^^^^^^^^^^  File"/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64/build/Lib/test/support/interpreters.py", line25, increateid= _interpreters.create(isolated=isolated)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^RuntimeError:interpreter creation failedkTraceback (most recent call last):  File"<frozen importlib._bootstrap>", line1354, in_find_and_load  File"<frozen importlib._bootstrap>", line1325, in_find_and_load_unlocked  File"<frozen importlib._bootstrap>", line929, in_load_unlocked  File"<frozen importlib._bootstrap_external>", line1004, inexec_module  File"<frozen importlib._bootstrap_external>", line1100, inget_code  File"<frozen importlib._bootstrap_external>", line1199, inget_dataTypeError:descriptor 'close' for '_io.BufferedReader' objects doesn't apply to a '_io.FileIO' object

if you're inclined to do so, the documentation improvements to the existing APIs in this PR would be worthwhile backporting to the 3.12 branch so that they show up on the default /3/ docs on python.org soon.

@gpshead:

if you're inclined to do so, the documentation improvements to the existing APIs in this PR would be worthwhile backporting to the 3.12 branch so that they show up on the default /3/ docs on python.org soon.

It makes sense. I wrote PRgh-110169 for Python 3.12 (and added "backport to 3.11" label).