Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork32.1k
gh-117755: Fix mimalloc for huge allocation on s390x#117809
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
cc@brandtbucher@colesbury@corona10 See#117755 (comment) for the rationale for this fix. The currenthttps://github.com/microsoft/mimalloc upstream doesn't seem to have the concept of "slices" in |
They are still there. The branches are a bit confusing. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
We should report and fix theuint32_t overflow, but I don't think we should be adding more tests that try to allocate huge amounts of memory. Especially with Linux's OOM killer, we are only setting ourselves up for more problems
I filed an issue upstream:microsoft/mimalloc#876 |
Fix mimalloc allocator for huge memory allocation (around8,589,934,592 GiB) on s390x.* Abort allocation early in mimalloc if the number of slices doesn't fit into uint32_t, to prevent a integer overflow (cast 64-bit size_t to uint32_t).* Add test_large_alloc() to test_bigaddrspace (test skipped on 32-bit platforms).* Reenable test_maxcontext_exact_arith() of test_decimal on s390x.* Reenable test_constructor() tests of test_io on s390x.
For now, I skipped the 3 test_io tests on s390x impacted by the bug:#117801 (merged). I rebased this PR on top of it to reenable these 3 test_io tests. |
I proposed a fix upstream:microsoft/mimalloc#877 |
@colesbury: Would it be acceptable to have this downstream change for now? I suppose the mimalloc upstream cares less about Linux s390x than Python (which has Linux s390x CIs: buildbots), and it may take a few weeks to get the fix merged upstream. At least, I already proposed the fix upstream. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The mimalloc changes look good to me.
I'm not thrilled with thetest_large_alloc test. If mimalloc fixes the bug by making the slice count 64-bits, then we'll run into problems. It strikes me as another problematic test that makes too many assumptions and won't provide enough value in terms of catching actual bugs.
I feel the same way about thetest_constructor tests. We are skipping these tests on sanitizers, 32-bit builds, and previously on s390x. That's a sign that they are unreliable tests. We should get rid of them! Unreliable tests are worse than no test.
vstinner commentedApr 16, 2024 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Let me try with uint64_t... :-) In short, the Linux s390x kernel doesn't kill the process allocating an insane amount of memory. I didn't check if the process is killed later with Out Of Memory (OOM) since I tested on a shared machine. The kernel is fine with a process allocating 822368422110644 kB of data. The strange part is "VmPTE: 26876 kB": I expected way more. I don't know which page size is used for the large allocation. Command: gdb: Process: The quantity of Swap is growing slowly... I stopped the process before most Swap was used... |
Remove unreliable tests on huge memory allocations:* Remove test_maxcontext_exact_arith() of test_decimal. Stefan Krah, test author, agreed on removing the test:python#114331 (comment)* Remove test_constructor() tests of test_io. Sam Gross suggests remove them:python#117809 (review)On Linux, depending how overcommit is configured, especially on Linuxs390x, a huge memory allocation (half or more of the full addressspace) can succeed, but then the process will eat the full systemswap and make the system slower and slower until the whole systembecomes unusable.Moreover, these tests had to be skipped when Python is built withsanitizers.
I'm fine with removing these tests: I created PRgh-117938 for that. |
I modified this PR to restrict it to the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Great! I've also added the PR to my tracking issue in#113141
Remove unreliable tests on huge memory allocations:* Remove test_maxcontext_exact_arith() of test_decimal. Stefan Krah, test author, agreed on removing the test:#114331 (comment)* Remove test_constructor() tests of test_io. Sam Gross suggests remove them:#117809 (review)On Linux, depending how overcommit is configured, especially on Linuxs390x, a huge memory allocation (half or more of the full addressspace) can succeed, but then the process will eat the full systemswap and make the system slower and slower until the whole systembecomes unusable.Moreover, these tests had to be skipped when Python is built withsanitizers.
Remove unreliable tests on huge memory allocations:* Remove test_maxcontext_exact_arith() of test_decimal. Stefan Krah, test author, agreed on removing the test:python#114331 (comment)* Remove test_constructor() tests of test_io. Sam Gross suggests remove them:python#117809 (review)On Linux, depending how overcommit is configured, especially on Linuxs390x, a huge memory allocation (half or more of the full addressspace) can succeed, but then the process will eat the full systemswap and make the system slower and slower until the whole systembecomes unusable.Moreover, these tests had to be skipped when Python is built withsanitizers.
…7809)Fix mimalloc allocator for huge memory allocation (around8,589,934,592 GiB) on s390x.Abort allocation early in mimalloc if the number of slices doesn'tfit into uint32_t, to prevent a integer overflow (cast 64-bitsize_t to uint32_t).
Uh oh!
There was an error while loading.Please reload this page.
Fix mimalloc allocator for huge memory allocation (around 8,589,934,592 GiB) on s390x.
test.test_io.CBufferedReaderTest.test_constructorcrash withFloating point exception#117755