Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.7k
gh-90815: Add mimalloc memory allocator#109914
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
8b3ec52 to06e86d7Comparee8c4f01 to2dd8675Compareericsnowcurrently commentedSep 27, 2023
DinoV commentedSep 27, 2023
@ericsnowcurrently Yep, and much of the autoconf stuff is taken from there... I just finally filled in the description with a little more background on where things are coming from :) |
ericsnowcurrently commentedSep 27, 2023
I expect that most of this PR is a relatively vanilla use of mimalloc in CPython, like@tiran's PR is. Would it be much trouble to keep the whole PR vanilla and have a follow-up PR that applies the customizations to mimalloc (e.g. per-thread allocator state)? That distinction would help, possibly a lot, when reviewing. |
7bda1e4 to35d1ebdCompareDinoV commentedOct 23, 2023
Okay, the default is now pymalloc, for now |
thesamesam commentedOct 30, 2023 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Any thoughts? I think this got missed earlier in the review flurry. I've not seen any substantive discussion about why bundling is a good idea, which feels like it's worth proper consideration, given it's short-term easier but long-term expensive. I understand upstreaming it initially is going to take some time, but it is going to be worthwhile for cpython upstream maintenance long-term too, as you can avoid:
This also, of course, has the advantage of forcing one to document what the changes are which I think should be done either way. |
Uh oh!
There was an error while loading.Please reload this page.
vstinner left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
See also issue#111499: PYTHONMALLOCSTATS=1 fails with fatal error at Python exit. Using mimalloc, Python is not affected :-)
vstinner commentedOct 30, 2023
@DinoV: I pushed a few cleanup changes. I fixed the outdated doc which still said that mimalloc is the default. |
vstinner commentedOct 30, 2023
ericsnowcurrently commentedOct 30, 2023
It's worth further discussion. CC@daanx |
vstinner commentedOct 30, 2023
Congrats@DinoV, I merged your PR :-) Do you want to propose a follow-up PR to enable it by default when |
ericsnowcurrently commentedOct 30, 2023
Looks like this broke the WASM buildbot:https://buildbot.python.org/all/#/builders/1046/builds/3371. |
brettcannon commentedOct 30, 2023
Yep, looks like there's an implicit function definition: |
vstinner commentedOct 30, 2023
I wrote PR#111524 to fix WASI build. |
vstinner commentedOct 31, 2023
I wrote 4 follow-up PRs to fix different issues:
While mimalloc C code is built on Windows, Python cannot currently use mimalloc on Windows: see PR#111528. |
* Add mimalloc v2.12Modified src/alloc.c to remove include of alloc-override.c and notcompile new handler.Did not include the following files: - include/mimalloc-new-delete.h - include/mimalloc-override.h - src/alloc-override-osx.c - src/alloc-override.c - src/static.c - src/region.cmimalloc is thread safe and shares a single heap across all runtimes,therefore finalization and getting global allocated blocks across allruntimes is different.* mimalloc: minimal changes for use in Python: - remove debug spam for freeing large allocations - use same bytes (0xDD) for freed allocations in CPython and mimalloc This is important for the test_capi debug memory tests* Don't export mimalloc symbol in libpython.* Enable mimalloc as Python allocator option.* Add mimalloc MIT license.* Log mimalloc in Lib/test/pythoninfo.py.* Document new mimalloc support.* Use macro defs for exports as done in:python#31164Co-authored-by: Sam Gross <colesbury@gmail.com>Co-authored-by: Christian Heimes <christian@python.org>Co-authored-by: Victor Stinner <vstinner@python.org>
* Add mimalloc v2.12Modified src/alloc.c to remove include of alloc-override.c and notcompile new handler.Did not include the following files: - include/mimalloc-new-delete.h - include/mimalloc-override.h - src/alloc-override-osx.c - src/alloc-override.c - src/static.c - src/region.cmimalloc is thread safe and shares a single heap across all runtimes,therefore finalization and getting global allocated blocks across allruntimes is different.* mimalloc: minimal changes for use in Python: - remove debug spam for freeing large allocations - use same bytes (0xDD) for freed allocations in CPython and mimalloc This is important for the test_capi debug memory tests* Don't export mimalloc symbol in libpython.* Enable mimalloc as Python allocator option.* Add mimalloc MIT license.* Log mimalloc in Lib/test/pythoninfo.py.* Document new mimalloc support.* Use macro defs for exports as done in:python#31164Co-authored-by: Sam Gross <colesbury@gmail.com>Co-authored-by: Christian Heimes <christian@python.org>Co-authored-by: Victor Stinner <vstinner@python.org>
* Add mimalloc v2.12Modified src/alloc.c to remove include of alloc-override.c and notcompile new handler.Did not include the following files: - include/mimalloc-new-delete.h - include/mimalloc-override.h - src/alloc-override-osx.c - src/alloc-override.c - src/static.c - src/region.cmimalloc is thread safe and shares a single heap across all runtimes,therefore finalization and getting global allocated blocks across allruntimes is different.* mimalloc: minimal changes for use in Python: - remove debug spam for freeing large allocations - use same bytes (0xDD) for freed allocations in CPython and mimalloc This is important for the test_capi debug memory tests* Don't export mimalloc symbol in libpython.* Enable mimalloc as Python allocator option.* Add mimalloc MIT license.* Log mimalloc in Lib/test/pythoninfo.py.* Document new mimalloc support.* Use macro defs for exports as done in:python#31164Co-authored-by: Sam Gross <colesbury@gmail.com>Co-authored-by: Christian Heimes <christian@python.org>Co-authored-by: Victor Stinner <vstinner@python.org>
Uh oh!
There was an error while loading.Please reload this page.
This adds mimalloc as an optional (but preferred when available) allocator to CPython. This is a bit of a mashup of the work from#109914 and the work of@colesbury to use mimalloc for no-gil and various updates to bring it up to current CPython.
The configuration logic added by@tiran is re-used and we keep pymalloc support unlike in the version from@colesbury. mimalloc is updated to 2.12 and along with a few changes@colesbury made to it.
This has run into some issues with subinterpreter support in that the allocator's are now stored in thread state and are per-thread. Sub interpreters in some scenarios will create a thread state on one thread and run that on another thread. Most of these are documented in the code base as being known issues. I've modified these so that we will find the right thread based upon the current thread ID and switch to it rather than getting the head thread. This seems pretty reasonable but looks pretty weird when we need to do it at interpreter shutdown.