Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

gh-144319: Fix huge page safety in pymalloc arenas#144331

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
pablogsal merged 2 commits intopython:mainfrompablogsal:gh-144319-2
Jan 30, 2026
Merged
Show file tree
Hide file tree
Changes fromall commits
Commits
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletionsDoc/using/cmdline.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1087,6 +1087,27 @@ conflict.
It now has no effect if set to an empty string.


..envvar::PYTHON_PYMALLOC_HUGEPAGES

If set to a non-zero integer, enable huge page support for
:ref:`pymalloc<pymalloc>` arenas. Set to ``0`` or unset to disable.
Python must be compiled with:option:`--with-pymalloc-hugepages` for this
variable to have any effect.

When enabled, arena allocation uses ``MAP_HUGETLB`` (Linux) or
``MEM_LARGE_PAGES`` (Windows) with automatic fallback to regular pages if
huge pages are not available.

..warning::

On Linux, if the huge-page pool is exhausted, page faults — including
copy-on-write faults triggered by:func:`os.fork` — deliver ``SIGBUS``
and kill the process. Only enable this in environments where the
huge-page pool is properly sized and fork-safety is not a concern.

..versionadded::next


..envvar::PYTHONLEGACYWINDOWSFSENCODING

If set to a non-empty string, the default:term:`filesystem encoding and
Expand Down
6 changes: 6 additions & 0 deletionsDoc/using/configure.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -790,6 +790,12 @@ also be used to improve performance.
2 MiB and arena allocation uses ``MAP_HUGETLB`` (Linux) or
``MEM_LARGE_PAGES`` (Windows) with automatic fallback to regular pages.

Even when compiled with this option, huge pages are **not** used at runtime
unless the:envvar:`PYTHON_PYMALLOC_HUGEPAGES` environment variable is set
to ``1``. This opt-in is required because huge pages carry risks on Linux:
if the huge-page pool is exhausted, page faults (including copy-on-write
faults after:func:`os.fork`) deliver ``SIGBUS`` and kill the process.

The configure script checks that the platform supports ``MAP_HUGETLB``
and emits a warning if it is not available.

Expand Down
2 changes: 2 additions & 0 deletionsDoc/whatsnew/3.15.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1482,6 +1482,8 @@ Build changes
increases to 2 MiB and allocation uses ``MAP_HUGETLB`` (Linux) or
``MEM_LARGE_PAGES`` (Windows) with automatic fallback to regular pages.
On Windows, use ``build.bat --pymalloc-hugepages``.
At runtime, huge pages must be explicitly enabled by setting the
:envvar:`PYTHON_PYMALLOC_HUGEPAGES` environment variable to ``1``.

* Annotating anonymous mmap usage is now supported if Linux kernel supports
:manpage:`PR_SET_VMA_ANON_NAME <PR_SET_VMA(2const)>` (Linux 5.17 or newer).
Expand Down
1 change: 1 addition & 0 deletionsInclude/cpython/initconfig.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -149,6 +149,7 @@ typedef struct PyConfig {
intdump_refs;
wchar_t*dump_refs_file;
intmalloc_stats;
intpymalloc_hugepages;
wchar_t*filesystem_encoding;
wchar_t*filesystem_errors;
wchar_t*pycache_prefix;
Expand Down
1 change: 1 addition & 0 deletionsInclude/internal/pycore_runtime_structs.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -31,6 +31,7 @@ struct _pymem_allocators {
debug_alloc_api_tobj;
}debug;
intis_debug_enabled;
intuse_hugepages;
PyObjectArenaAllocatorobj_arena;
};

Expand Down
1 change: 1 addition & 0 deletionsLib/test/test_capi/test_config.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -63,6 +63,7 @@ def test_config_get(self):
("interactive",bool,None),
("isolated",bool,None),
("malloc_stats",bool,None),
("pymalloc_hugepages",bool,None),
("module_search_paths",list[str],"path"),
("optimization_level",int,None),
("orig_argv",list[str],"orig_argv"),
Expand Down
4 changes: 4 additions & 0 deletionsLib/test/test_embed.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -642,6 +642,7 @@ class InitConfigTests(EmbeddingTestsMixin, unittest.TestCase):
'dump_refs':False,
'dump_refs_file':None,
'malloc_stats':False,
'pymalloc_hugepages':False,

'filesystem_encoding':GET_DEFAULT_CONFIG,
'filesystem_errors':GET_DEFAULT_CONFIG,
Expand DownExpand Up@@ -1044,6 +1045,7 @@ def test_init_from_config(self):
'code_debug_ranges':False,
'show_ref_count':True,
'malloc_stats':True,
'pymalloc_hugepages':True,

'stdio_encoding':'iso8859-1',
'stdio_errors':'replace',
Expand DownExpand Up@@ -1109,6 +1111,7 @@ def test_init_compat_env(self):
'import_time':1,
'code_debug_ranges':False,
'malloc_stats':True,
'pymalloc_hugepages':True,
'inspect':True,
'optimization_level':2,
'pythonpath_env':'/my/path',
Expand DownExpand Up@@ -1145,6 +1148,7 @@ def test_init_python_env(self):
'import_time':1,
'code_debug_ranges':False,
'malloc_stats':True,
'pymalloc_hugepages':True,
'inspect':True,
'optimization_level':2,
'pythonpath_env':'/my/path',
Expand Down
75 changes: 64 additions & 11 deletionsObjects/obmalloc.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -13,6 +13,7 @@

#include <stdlib.h> // malloc()
#include <stdbool.h>
#include <stdio.h> // fopen(), fgets(), sscanf()
#ifdef WITH_MIMALLOC
// Forward declarations of functions used in our mimalloc modifications
static void _PyMem_mi_page_clear_qsbr(mi_page_t *page);
Expand DownExpand Up@@ -492,16 +493,57 @@ _PyMem_DefaultRawWcsdup(const wchar_t *str)
# endif
#endif

/* Return the system's default huge page size in bytes, or 0 if it
* cannot be determined. The result is cached after the first call.
*
* This is Linux-only (/proc/meminfo). On other systems that define
* MAP_HUGETLB the caller should skip huge pages gracefully. */
#if defined(PYMALLOC_USE_HUGEPAGES) && defined(ARENAS_USE_MMAP) && defined(MAP_HUGETLB)
static size_t
_pymalloc_system_hugepage_size(void)
{
static size_t hp_size = 0;
static int initialized = 0;

if (initialized) {
return hp_size;
}

#ifdef __linux__
FILE *f = fopen("/proc/meminfo", "r");
if (f != NULL) {
char line[256];
while (fgets(line, sizeof(line), f)) {
unsigned long size_kb;
if (sscanf(line, "Hugepagesize: %lu kB", &size_kb) == 1) {
hp_size = (size_t)size_kb * 1024;
break;
}
}
fclose(f);
}
#endif

initialized = 1;
return hp_size;
}
#endif

void *
_PyMem_ArenaAlloc(void *Py_UNUSED(ctx), size_t size)
{
#ifdef MS_WINDOWS
# ifdef PYMALLOC_USE_HUGEPAGES
void *ptr = VirtualAlloc(NULL, size,
MEM_COMMIT | MEM_RESERVE | MEM_LARGE_PAGES,
PAGE_READWRITE);
if (ptr != NULL)
return ptr;
if (_PyRuntime.allocators.use_hugepages) {
SIZE_T lp_size = GetLargePageMinimum();
if (lp_size > 0 && size % lp_size == 0) {
void *ptr = VirtualAlloc(NULL, size,
MEM_COMMIT | MEM_RESERVE | MEM_LARGE_PAGES,
PAGE_READWRITE);
if (ptr != NULL)
return ptr;
}
}
/* Fall back to regular pages */
# endif
return VirtualAlloc(NULL, size,
Expand All@@ -510,12 +552,23 @@ _PyMem_ArenaAlloc(void *Py_UNUSED(ctx), size_t size)
void *ptr;
# ifdef PYMALLOC_USE_HUGEPAGES
# ifdef MAP_HUGETLB
ptr = mmap(NULL, size, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB, -1, 0);
if (ptr != MAP_FAILED) {
assert(ptr != NULL);
(void)_PyAnnotateMemoryMap(ptr, size, "cpython:pymalloc:hugepage");
return ptr;
if (_PyRuntime.allocators.use_hugepages) {
size_t hp_size = _pymalloc_system_hugepage_size();
/* Only use huge pages if the arena size is a multiple of the
* system's default huge page size. When the arena is smaller
* than the huge page, mmap still succeeds but silently
* allocates an entire huge page; the subsequent munmap with
* the smaller arena size then fails with EINVAL, leaking
* all of that memory. */
if (hp_size > 0 && size % hp_size == 0) {
ptr = mmap(NULL, size, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB, -1, 0);
if (ptr != MAP_FAILED) {
assert(ptr != NULL);
(void)_PyAnnotateMemoryMap(ptr, size, "cpython:pymalloc:hugepage");
return ptr;
}
}
}
/* Fall back to regular pages */
# endif
Expand Down
2 changes: 2 additions & 0 deletionsPrograms/_testembed.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -639,6 +639,7 @@ static int test_init_from_config(void)

putenv("PYTHONMALLOCSTATS=0");
config.malloc_stats=1;
config.pymalloc_hugepages=1;

putenv("PYTHONPYCACHEPREFIX=env_pycache_prefix");
config_set_string(&config,&config.pycache_prefix,L"conf_pycache_prefix");
Expand DownExpand Up@@ -795,6 +796,7 @@ static void set_most_env_vars(void)
putenv("PYTHONPROFILEIMPORTTIME=1");
putenv("PYTHONNODEBUGRANGES=1");
putenv("PYTHONMALLOCSTATS=1");
putenv("PYTHON_PYMALLOC_HUGEPAGES=1");
putenv("PYTHONUTF8=1");
putenv("PYTHONVERBOSE=1");
putenv("PYTHONINSPECT=1");
Expand Down
18 changes: 18 additions & 0 deletionsPython/initconfig.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -160,6 +160,7 @@ static const PyConfigSpec PYCONFIG_SPEC[] = {
SPEC(legacy_windows_stdio, BOOL, READ_ONLY, NO_SYS),
#endif
SPEC(malloc_stats, BOOL, READ_ONLY, NO_SYS),
SPEC(pymalloc_hugepages, BOOL, READ_ONLY, NO_SYS),
SPEC(orig_argv, WSTR_LIST, READ_ONLY, SYS_ATTR("orig_argv")),
SPEC(parse_argv, BOOL, READ_ONLY, NO_SYS),
SPEC(pathconfig_warnings, BOOL, READ_ONLY, NO_SYS),
Expand DownExpand Up@@ -900,6 +901,7 @@ config_check_consistency(const PyConfig *config)
assert(config->show_ref_count >= 0);
assert(config->dump_refs >= 0);
assert(config->malloc_stats >= 0);
assert(config->pymalloc_hugepages >= 0);
assert(config->site_import >= 0);
assert(config->bytes_warning >= 0);
assert(config->warn_default_encoding >= 0);
Expand DownExpand Up@@ -1879,6 +1881,18 @@ config_read_env_vars(PyConfig *config)
if (config_get_env(config, "PYTHONMALLOCSTATS")) {
config->malloc_stats = 1;
}
{
const char *env = _Py_GetEnv(use_env, "PYTHON_PYMALLOC_HUGEPAGES");
if (env) {
int value;
if (_Py_str_to_int(env, &value) < 0 || value < 0) {
/* PYTHON_PYMALLOC_HUGEPAGES=text or negative
behaves as PYTHON_PYMALLOC_HUGEPAGES=1 */
value = 1;
}
config->pymalloc_hugepages = (value > 0);
}
}

if (config->dump_refs_file == NULL) {
status = CONFIG_GET_ENV_DUP(config, &config->dump_refs_file,
Expand DownExpand Up@@ -2812,6 +2826,10 @@ _PyConfig_Write(const PyConfig *config, _PyRuntimeState *runtime)
return _PyStatus_NO_MEMORY();
}

#ifdef PYMALLOC_USE_HUGEPAGES
runtime->allocators.use_hugepages = config->pymalloc_hugepages;
#endif

return _PyStatus_OK();
}

Expand Down
Loading
[8]ページ先頭
©2009-2026 Movatter.jp