Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

GH-87804: Fix counter overflow in statvfs on macOS#99570

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged

Conversation

@ronaldoussoren
Copy link
Contributor

@ronaldoussorenronaldoussoren commentedNov 18, 2022
edited by bedevere-bot
Loading

On macOS the statvfs interface returns block counts as 32-bit integers, and that results in bad reporting for larger disks.

Therefore reimplement statvfs in terms of statfs, which does use 64-bit integers for block counts.

Tested using a sparse filesystem image of 100TB.

roachcord3 reacted with thumbs up emoji
On macOS the statvfs interface returns block counts as32-bit integers, and that results in bad reporting forlarger disks.Therefore reimplement statvfs in terms of statfs, whichdoes use 64-bit integers for block counts.Tested using a sparse filesystem image of 100TB.
@ronaldoussoren
Copy link
ContributorAuthor

The conversion from struct statfs to a statvfs value is based on the implementation in Apple's libc, in particular for the reported maximum filename length.

I'm not entirely sure if how far back we should back port this. The patch is clean so could easily be back ported to 3.10, but is conceptually a fairly large change.

Copy link
Contributor

@giampaologiampaolo left a comment
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Patch LGTM. The only thing which I'm doubtful about is thef_flag field. According to the man pages these are the originalstatvfs fields:

ST_MANDLOCK             Mandatory locking is permitted on the filesystem (see fcntl(2)).ST_NOATIME              Do not update access times; see mount(2).ST_NODEV                Disallow access to device special files on this filesystem.ST_NODIRATIME           Do not update directory access times; see mount(2).ST_NOEXEC               Execution of programs is disallowed on this filesystem.ST_NOSUID               The set-user-ID and set-group-ID bits are ignored by exec(3) for executable files on this filesystemST_RDONLY               This filesystem is mounted read-only.ST_RELATIME             Update atime relative to mtime/ctime; see mount(2).ST_SYNCHRONOUS          Writes are synched to the filesystem immediately (see the description of O_SYNC in open(2)).

...and these are macOSfstatfs's counterparts:

MNT_RDONLY             A read-only filesystemMNT_SYNCHRONOUS        File system is written to synchronouslyMNT_NOEXEC             Can't exec from filesystemMNT_NOSUID             Setuid bits are not honored on this filesystemMNT_NODEV              Don't interpret special filesMNT_UNION              Union with underlying filesystenMNT_ASYNC              File system written to asynchronouslyMNT_EXPORTED           File system is exported File system is stored locallyMNT_QUOTA              Quotas are enabled on this file systemMNT_ROOTFS             This file system is the root of the file systemMNT_DOVOLFS            File system supports volfsMNT_DONTBROWSE         File system is not appropriate path to user dataMNT_UNKNOWNPERMISSIONS VFS will ignore ownership information on filesys-tem filesystem tem objectsMNT_AUTOMOUNTED        File system was mounted by automounterMNT_JOURNALED          File system is journaledMNT_DEFWRITE           File system should defer writesMNT_MULTILABEL         MAC support for individual labels

In your PR you rightly mapMNT_RDONLY andMNT_NOSUID toST_RDONLY andST_NOSUID (note: you can also mapMNT_NOEXEC toST_NOEXEC) , but the otherST_* flags are lost. I'm not sure if this is appropriate, but perhaps it would make sense to call bothstatvfs andfstatfs, and use the former to fetch all flags.

}
if (st.f_flags&MNT_NOSUID) {
flags |=ST_NOSUID;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

In addition to these, I think you can also mapMNT_NOEXEC toST_NOEXEC.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

My current implementation matches that of Apple's emulation, but I agree that it is useful to add a mapping for *_NOEXEC as well.

@Safihre
Copy link

@ronaldoussoren Do you think you might revisit this? :)

@ronaldoussoren
Copy link
ContributorAuthor

@ronaldoussoren Do you think you might revisit this? :)

I am revisiting this. Finally...

@ronaldoussoren
Copy link
ContributorAuthor

ronaldoussoren commentedDec 25, 2023
edited
Loading

In your PR you rightly mapMNT_RDONLY andMNT_NOSUID toST_RDONLY andST_NOSUID (note: you can also mapMNT_NOEXEC toST_NOEXEC) , but the otherST_* flags are lost. I'm not sure if this is appropriate, but perhaps it would make sense to call bothstatvfs andfstatfs, and use the former to fetch all flags.

The statvfs implementation on macOS:https://github.com/apple-oss-distributions/Libc/blob/c5a3293354e22262702a3add5b2dfc9bb0b93b85/emulated/statvfs.c#L35

As you can see this implementation does not implement flags other thanST_RDONLY andST_NOSUID.AddingST_NOEXEC is easy enough, as isST_NODEV. But callingstatvfs(3) to get other flags doesn't give us more information.

The current manual page, and SDK headers, only list two flags:

     There are two flags defined for the f_flag member:           ST_RDONLY      The file system is mounted read-only.           ST_NOSUID      The semantics of the S_ISUID and S_ISGID file mode bits are not supported by, or are disabled on, this file system.

That's another reason to not callstatvfs: There are no other flags to expose than the ones we already emulate.

@ronaldoussoren
Copy link
ContributorAuthor

@Yhg1s and@pablogsal : What's your opinion on back porting this to 3.12 and 3.11?

The PR reimplementsos.statvfs on macOS, but should result in the same behaviour other than fixing the problem with large disks because I based it on Apple implementation ofstatvfs(3).

@ronaldoussoren
Copy link
ContributorAuthor

@Yhg1s and@pablogsal : What's your opinion on back porting this to 3.12 and 3.11?

The PR reimplementsos.statvfs on macOS, but should result in the same behaviour other than fixing the problem with large disks because I based it on Apple implementation ofstatvfs(3).

I won't backport to 3.11 and 3.12, the change is too large for that (IMHO)

@ronaldoussorenronaldoussoren merged commit6e222a5 intopython:mainFeb 10, 2024
@ronaldoussorenronaldoussoren deleted the gh87804-statvfs-32bit branchFebruary 10, 2024 10:16
@sobolevn
Copy link
Member

Follow up:#115236

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure⚠️⚠️⚠️

Hi! The buildbotAMD64 Windows10 3.x has failed when building commit6e222a5.

What do you need to do:

  1. Don't panic.
  2. Checkthe buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/146/builds/7668) and take a look at the build logs.
  4. Check if the failure is related to this commit (6e222a5) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/146/builds/7668

Failed tests:

  • test.test_concurrent_futures.test_shutdown

Failed subtests:

  • test_interpreter_shutdown - test.test_concurrent_futures.test_shutdown.ProcessPoolSpawnProcessPoolShutdownTest.test_interpreter_shutdown

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):  File"D:\buildarea\3.x.bolen-windows10\build\Lib\test\test_concurrent_futures\test_shutdown.py", line50, intest_interpreter_shutdownself.assertEqual(out.strip(),b"apple")~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^AssertionError:b'' != b'apple'

fsc-eriker pushed a commit to fsc-eriker/cpython that referenced this pull requestFeb 14, 2024
On macOS the statvfs interface returns block counts as32-bit integers, and that results in bad reporting forlarger disks.Therefore reimplement statvfs in terms of statfs, whichdoes use 64-bit integers for block counts.Tested using a sparse filesystem image of 100TB.
Safihre added a commit to sabnzbd/sabnzbd that referenced this pull requestSep 30, 2025
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@giampaologiampaologiampaolo left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ronaldoussoren@Safihre@sobolevn@bedevere-bot@giampaolo

[8]ページ先頭

©2009-2025 Movatter.jp