Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.3k
GH-87804: Fix counter overflow in statvfs on macOS#99570
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
On macOS the statvfs interface returns block counts as32-bit integers, and that results in bad reporting forlarger disks.Therefore reimplement statvfs in terms of statfs, whichdoes use 64-bit integers for block counts.Tested using a sparse filesystem image of 100TB.
The conversion from struct statfs to a statvfs value is based on the implementation in Apple's libc, in particular for the reported maximum filename length. I'm not entirely sure if how far back we should back port this. The patch is clean so could easily be back ported to 3.10, but is conceptually a fairly large change. |
giampaolo left a comment• edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Patch LGTM. The only thing which I'm doubtful about is thef_flag field. According to the man pages these are the originalstatvfs fields:
ST_MANDLOCK Mandatory locking is permitted on the filesystem (see fcntl(2)).ST_NOATIME Do not update access times; see mount(2).ST_NODEV Disallow access to device special files on this filesystem.ST_NODIRATIME Do not update directory access times; see mount(2).ST_NOEXEC Execution of programs is disallowed on this filesystem.ST_NOSUID The set-user-ID and set-group-ID bits are ignored by exec(3) for executable files on this filesystemST_RDONLY This filesystem is mounted read-only.ST_RELATIME Update atime relative to mtime/ctime; see mount(2).ST_SYNCHRONOUS Writes are synched to the filesystem immediately (see the description of O_SYNC in open(2))....and these are macOSfstatfs's counterparts:
MNT_RDONLY A read-only filesystemMNT_SYNCHRONOUS File system is written to synchronouslyMNT_NOEXEC Can't exec from filesystemMNT_NOSUID Setuid bits are not honored on this filesystemMNT_NODEV Don't interpret special filesMNT_UNION Union with underlying filesystenMNT_ASYNC File system written to asynchronouslyMNT_EXPORTED File system is exported File system is stored locallyMNT_QUOTA Quotas are enabled on this file systemMNT_ROOTFS This file system is the root of the file systemMNT_DOVOLFS File system supports volfsMNT_DONTBROWSE File system is not appropriate path to user dataMNT_UNKNOWNPERMISSIONS VFS will ignore ownership information on filesys-tem filesystem tem objectsMNT_AUTOMOUNTED File system was mounted by automounterMNT_JOURNALED File system is journaledMNT_DEFWRITE File system should defer writesMNT_MULTILABEL MAC support for individual labelsIn your PR you rightly mapMNT_RDONLY andMNT_NOSUID toST_RDONLY andST_NOSUID (note: you can also mapMNT_NOEXEC toST_NOEXEC) , but the otherST_* flags are lost. I'm not sure if this is appropriate, but perhaps it would make sense to call bothstatvfs andfstatfs, and use the former to fetch all flags.
| } | ||
| if (st.f_flags&MNT_NOSUID) { | ||
| flags |=ST_NOSUID; | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
In addition to these, I think you can also mapMNT_NOEXEC toST_NOEXEC.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
My current implementation matches that of Apple's emulation, but I agree that it is useful to add a mapping for *_NOEXEC as well.
Safihre commentedDec 1, 2023
@ronaldoussoren Do you think you might revisit this? :) |
I am revisiting this. Finally... |
ronaldoussoren commentedDec 25, 2023 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
The statvfs implementation on macOS:https://github.com/apple-oss-distributions/Libc/blob/c5a3293354e22262702a3add5b2dfc9bb0b93b85/emulated/statvfs.c#L35 As you can see this implementation does not implement flags other than The current manual page, and SDK headers, only list two flags: That's another reason to not call |
@Yhg1s and@pablogsal : What's your opinion on back porting this to 3.12 and 3.11? The PR reimplements |
I won't backport to 3.11 and 3.12, the change is too large for that (IMHO) |
Follow up:#115236 |
bedevere-bot commentedFeb 10, 2024
|
On macOS the statvfs interface returns block counts as32-bit integers, and that results in bad reporting forlarger disks.Therefore reimplement statvfs in terms of statfs, whichdoes use 64-bit integers for block counts.Tested using a sparse filesystem image of 100TB.
Part of Python 3.13 and above.python/cpython#99570
Uh oh!
There was an error while loading.Please reload this page.
On macOS the statvfs interface returns block counts as 32-bit integers, and that results in bad reporting for larger disks.
Therefore reimplement statvfs in terms of statfs, which does use 64-bit integers for block counts.
Tested using a sparse filesystem image of 100TB.