Aug 15, 2024 · Dec 31, 2023 · Dec 31, 2023 · Jan 15, 2024 · Jan 15, 2024 · Jan 15, 2024
diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
   Undo all initializations made by :c:func:`Py_Initialize` and subsequent use of
   Python/C API functions, and destroy all sub-interpreters (see
   :c:func:`Py_NewInterpreter` below) that were created and not yet destroyed since
   the last call to :c:func:`Py_Initialize`.  Ideally, this frees all memory
   allocated by the Python interpreter.  This is a no-op when called for a second
   the last call to :c:func:`Py_Initialize`. This is a no-op when called for a second
   time (without calling :c:func:`Py_Initialize` again first).  Normally the
   return value is ``0``.  If there were errors during finalization
   (flushing buffered data), ``-1`` is returned.

   Note that Python will do a best effort at freeing all memory allocated by the Python
   interpreter.  Therefore, any C-Extension should make sure to correctly clean up all
   of the preveiously allocated PyObjects before using them in subsequent calls to
   :c:func:`Py_Initialize`.  Otherwise it could introduce vulnerabilities and incorrect
   behavior.

   This function is provided for a number of reasons.  An embedding application
   might want to restart Python without having to restart the application itself.
   An application that has loaded the Python interpreter from a dynamically
   loaded extension modules loaded by Python are not unloaded.  Small amounts of
   memory allocated by the Python interpreter may not be freed (if you find a leak,
   please report it).  Memory tied up in circular references between objects is not
   freed.  Some memory allocated by extension modules may not be freed.  Some
   extensions may not work properly if their initialization routine is called more
   than once; this can happen if an application calls :c:func:`Py_Initialize` and
   :c:func:`Py_FinalizeEx` more than once.
   freed.  Interned strings will all be deallocated regarldess of their reference count.
   Some memory allocated by extension modules may not be freed.  Some extensions may not
   work properly if their initialization routine is called more than once; this can
   happen if an application calls :c:func:`Py_Initialize` and :c:func:`Py_FinalizeEx`
   more than once.

   .. audit-event:: cpython._PySys_ClearAuditHooks "" c.Py_FinalizeEx

diff --git a/Doc/whatsnew/3.13.rst b/Doc/whatsnew/3.13.rst
 * Add :c:func:`Py_HashPointer` function to hash a pointer.
  (Contributed by Victor Stinner in :gh:`111545`.)

 * Modified the ``_PyUnicode_ClearInterned`` function to always delete all
  interned strings during a call to :c:func:`Py_Finalize`. This makes all
  is backwards incompatible to any C-Extension that holds onto an interned
  string after a call to :c:func:`Py_Finalize` and is then reused after a
  call to :c:func:`Py_Initialize`.  Any issues arising from this behavior will
  normally result in crashes during the exectuion of the subsequent call to
  :c:func:`Py_Initialize` from accessing uninitialized memory. To fix, use
  an address sanitizer (i.e. ASAN) to identify any use-after-free coming from
  an interned string and deallocate it during module shutdown.
  (Contribued by Eddie Elizondo in :gh:`113601`.)


 Porting to Python 3.13
 ----------------------
diff --git a/Lib/test/_test_embed_structseq.py b/Lib/test/_test_embed_structseq.py
 import sys
 import types
 import unittest

 # Note: This test file can't import `unittest` since the runtime can't
 # currently guarantee that it will not leak memory. Doing so will mark
 # the test as passing but with reference leaks. This can safely import
 # the `unittest` library once there's a strict guarantee of no leaks
 # during runtime shutdown.

 # bpo-46417: Test that structseq types used by the sys module are still
 # valid when Py_Finalize()/Py_Initialize() are called multiple times.
 class TestStructSeq:
 class TestStructSeq(unittest.TestCase):
    # test PyTypeObject members
    def_check_structseq(self, obj_type):
    defcheck_structseq(self, obj_type):
        # ob_refcnt
 assertsys.getrefcount(obj_type) > 1
 self.assertGreaterEqual(sys.getrefcount(obj_type), 1)
        # tp_base
 assertissubclass(obj_type, tuple)
 self.assertTrue(issubclass(obj_type, tuple))
        # tp_bases
 assertobj_type.__bases__ ==(tuple,)
 self.assertEqual(obj_type.__bases__,(tuple,))
        # tp_dict
 assert isinstance(obj_type.__dict__, types.MappingProxyType)
 self.assertIsInstance(obj_type.__dict__, types.MappingProxyType)
        # tp_mro
 assertobj_type.__mro__ ==(obj_type, tuple, object)
 self.assertEqual(obj_type.__mro__,(obj_type, tuple, object))
        # tp_name
 assert isinstance(type.__name__, str)
 self.assertIsInstance(type.__name__, str)
        # tp_subclasses
 assertobj_type.__subclasses__() == []
 self.assertEqual(obj_type.__subclasses__(), [])

    def test_sys_attrs(self):
        for attr_name in (
            'thread_info',    # ThreadInfoType
            'version_info',   # VersionInfoType
        ):
            attr = getattr(sys, attr_name)
            self._check_structseq(type(attr))
            with self.subTest(attr=attr_name):
                attr = getattr(sys, attr_name)
                self.check_structseq(type(attr))

    def test_sys_funcs(self):
        func_names = ['get_asyncgen_hooks']  # AsyncGenHooksType
        if hasattr(sys, 'getwindowsversion'):
            func_names.append('getwindowsversion')  # WindowsVersionType
        for func_name in func_names:
            func = getattr(sys, func_name)
            obj = func()
            self._check_structseq(type(obj))
            with self.subTest(func=func_name):
                func = getattr(sys, func_name)
                obj = func()
                self.check_structseq(type(obj))


 try:
    tests = TestStructSeq()
    tests.test_sys_attrs()
    tests.test_sys_funcs()
    unittest.main()
 except SystemExit as exc:
    if exc.args[0] != 0:
        raise
diff --git a/Misc/NEWS.d/next/Core and Builtins/2024-01-15-18-11-48.gh-issue-113190.OwQX64.rst b/Misc/NEWS.d/next/Core and Builtins/2024-01-15-18-11-48.gh-issue-113190.OwQX64.rst
 This updates the interned string deallocation function
 ``_PyUnicode_ClearInterned`` to delete all interned strings during runtime
 finalization when calling ``Py_Finalize``, regardless of their reference
 count.

 Worth noting that if an extension accidentally holds onto a interned string,
 event after calling Py_Finalize, it will result in use-after-free error,
 leaving the user to a potential vulnerabilities. That said, the history of
 how these interned strings are handled have been changing during throughout
 different versions.

 For example, in Python 3.9 and older, interned strings were never deleted.
 Only special builds for Valgrind and Purity cleared them at exit. In Python
 3.10 and 3.11, interned strings are always deleted at exit. In Python 3.12,
 interned strings are deleted only if Python is built in debug mode: they
 are not deleted in release mode. In Python 3.13, interned strings will
 now be all deleted.

 Given how the guarantees changed throughout different versions, it is not
 expected that users actively rely on this behavior for their extensions.
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
    }
    assert(PyDict_CheckExact(interned));

    /* TODO:
     * Currently, the runtime is not able to guarantee that it can exit without
     * allocations that carry over to a future initialization of Python within
     * the same process. i.e:
     *   ./python -X showrefcount -c 'import itertools'
     *   [237 refs, 237 blocks]
     *
     * Therefore, this should remain disabled for until there is a strict guarantee
     * that no memory will be left after `Py_Finalize`.
     */
 #ifdef Py_DEBUG
    /* For all non-singleton interned strings, restore the two valid references
       to that instance from within the intern string dictionary and let the
       normal reference counting process clean up these instances. */
    for (Py_ssize_t i=0; i < ids->size; i++) {
        Py_XINCREF(ids->array[i]);
    }
 #endif /* Py_DEBUG */
    clear_interned_dict(interp);
 }
Original file line number	Diff line number	Diff line change
Expand Up		@@ -386,12 +386,17 @@ Initializing and finalizing the interpreter
		Undo all initializations made by :c:func:`Py_Initialize` and subsequent use of
		Python/C API functions, and destroy all sub-interpreters (see
		:c:func:`Py_NewInterpreter` below) that were created and not yet destroyed since
		the last call to :c:func:`Py_Initialize`. Ideally, this frees all memory
		allocated by the Python interpreter. This is a no-op when called for a second
		the last call to :c:func:`Py_Initialize`. This is a no-op when called for a second
		time (without calling :c:func:`Py_Initialize` again first). Normally the
		return value is ``0``. If there were errors during finalization
		(flushing buffered data), ``-1`` is returned.

		Note that Python will do a best effort at freeing all memory allocated by the Python
		interpreter. Therefore, any C-Extension should make sure to correctly clean up all
		of the preveiously allocated PyObjects before using them in subsequent calls to
		:c:func:`Py_Initialize`. Otherwise it could introduce vulnerabilities and incorrect
		behavior.

		This function is provided for a number of reasons. An embedding application
		might want to restart Python without having to restart the application itself.
		An application that has loaded the Python interpreter from a dynamically
Expand All		@@ -406,10 +411,11 @@ Initializing and finalizing the interpreter
		loaded extension modules loaded by Python are not unloaded. Small amounts of
		memory allocated by the Python interpreter may not be freed (if you find a leak,
		please report it). Memory tied up in circular references between objects is not
		freed. Some memory allocated by extension modules may not be freed. Some
		extensions may not work properly if their initialization routine is called more
		than once; this can happen if an application calls :c:func:`Py_Initialize` and
		:c:func:`Py_FinalizeEx` more than once.
		freed. Interned strings will all be deallocated regarldess of their reference count.
		Some memory allocated by extension modules may not be freed. Some extensions may not
		work properly if their initialization routine is called more than once; this can
		happen if an application calls :c:func:`Py_Initialize` and :c:func:`Py_FinalizeEx`
		more than once.

		.. audit-event:: cpython._PySys_ClearAuditHooks "" c.Py_FinalizeEx

Expand Down
Original file line number	Diff line number	Diff line change
Expand Up		@@ -1363,6 +1363,17 @@ New Features
		* Add :c:func:`Py_HashPointer` function to hash a pointer.
		(Contributed by Victor Stinner in :gh:`111545`.)

		* Modified the ``_PyUnicode_ClearInterned`` function to always delete all
		interned strings during a call to :c:func:`Py_Finalize`. This makes all
		is backwards incompatible to any C-Extension that holds onto an interned
		string after a call to :c:func:`Py_Finalize` and is then reused after a
		call to :c:func:`Py_Initialize`. Any issues arising from this behavior will
		normally result in crashes during the exectuion of the subsequent call to
		:c:func:`Py_Initialize` from accessing uninitialized memory. To fix, use
		an address sanitizer (i.e. ASAN) to identify any use-after-free coming from
		an interned string and deallocate it during module shutdown.
		(Contribued by Eddie Elizondo in :gh:`113601`.)


		Porting to Python 3.13
		----------------------
Expand Down
Original file line number	Diff line number	Diff line change
		@@ -1,31 +1,27 @@
		import sys
		import types
		import unittest

		# Note: This test file can't import `unittest` since the runtime can't
		# currently guarantee that it will not leak memory. Doing so will mark
		# the test as passing but with reference leaks. This can safely import
		# the `unittest` library once there's a strict guarantee of no leaks
		# during runtime shutdown.

		# bpo-46417: Test that structseq types used by the sys module are still
		# valid when Py_Finalize()/Py_Initialize() are called multiple times.
		class TestStructSeq:
		class TestStructSeq(unittest.TestCase):
		# test PyTypeObject members
		def_check_structseq(self, obj_type):
		defcheck_structseq(self, obj_type):
		# ob_refcnt
		assertsys.getrefcount(obj_type) > 1
		self.assertGreaterEqual(sys.getrefcount(obj_type), 1)
		# tp_base
		assertissubclass(obj_type, tuple)
		self.assertTrue(issubclass(obj_type, tuple))
		# tp_bases
		assertobj_type.__bases__ ==(tuple,)
		self.assertEqual(obj_type.__bases__,(tuple,))
		# tp_dict
		assert isinstance(obj_type.__dict__, types.MappingProxyType)
		self.assertIsInstance(obj_type.__dict__, types.MappingProxyType)
		# tp_mro
		assertobj_type.__mro__ ==(obj_type, tuple, object)
		self.assertEqual(obj_type.__mro__,(obj_type, tuple, object))
		# tp_name
		assert isinstance(type.__name__, str)
		self.assertIsInstance(type.__name__, str)
		# tp_subclasses
		assertobj_type.__subclasses__() == []
		self.assertEqual(obj_type.__subclasses__(), [])

		def test_sys_attrs(self):
		for attr_name in (
Expand All		@@ -36,23 +32,23 @@ def test_sys_attrs(self):
		'thread_info', # ThreadInfoType
		'version_info', # VersionInfoType
		):
		attr = getattr(sys, attr_name)
		self._check_structseq(type(attr))
		with self.subTest(attr=attr_name):
		attr = getattr(sys, attr_name)
		self.check_structseq(type(attr))

		def test_sys_funcs(self):
		func_names = ['get_asyncgen_hooks'] # AsyncGenHooksType
		if hasattr(sys, 'getwindowsversion'):
		func_names.append('getwindowsversion') # WindowsVersionType
		for func_name in func_names:
		func = getattr(sys, func_name)
		obj = func()
		self._check_structseq(type(obj))
		with self.subTest(func=func_name):
		func = getattr(sys, func_name)
		obj = func()
		self.check_structseq(type(obj))


		try:
		tests = TestStructSeq()
		tests.test_sys_attrs()
		tests.test_sys_funcs()
		unittest.main()
		except SystemExit as exc:
		if exc.args[0] != 0:
		raise
Expand Down
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,20 @@
		This updates the interned string deallocation function
		``_PyUnicode_ClearInterned`` to delete all interned strings during runtime
		finalization when calling ``Py_Finalize``, regardless of their reference
		count.

		Worth noting that if an extension accidentally holds onto a interned string,
		event after calling Py_Finalize, it will result in use-after-free error,
eduardo-elizondo marked this conversation as resolved. OutdatedShow resolvedHide resolved
		leaving the user to a potential vulnerabilities. That said, the history of
		how these interned strings are handled have been changing during throughout
		different versions.

		For example, in Python 3.9 and older, interned strings were never deleted.
		Only special builds for Valgrind and Purity cleared them at exit. In Python
		3.10 and 3.11, interned strings are always deleted at exit. In Python 3.12,
		interned strings are deleted only if Python is built in debug mode: they
		are not deleted in release mode. In Python 3.13, interned strings will
		now be all deleted.

		Given how the guarantees changed throughout different versions, it is not
		expected that users actively rely on this behavior for their extensions.
Original file line number	Diff line number	Diff line change
Expand Up		@@ -14961,17 +14961,6 @@ _PyUnicode_ClearInterned(PyInterpreterState *interp)
		}
		assert(PyDict_CheckExact(interned));

		/* TODO:
		* Currently, the runtime is not able to guarantee that it can exit without
		* allocations that carry over to a future initialization of Python within
		* the same process. i.e:
		* ./python -X showrefcount -c 'import itertools'
		* [237 refs, 237 blocks]
		*
		* Therefore, this should remain disabled for until there is a strict guarantee
		* that no memory will be left after `Py_Finalize`.
		*/
		#ifdef Py_DEBUG
		/* For all non-singleton interned strings, restore the two valid references
		to that instance from within the intern string dictionary and let the
		normal reference counting process clean up these instances. */
Expand DownExpand Up		@@ -15028,7 +15017,6 @@ _PyUnicode_ClearInterned(PyInterpreterState *interp)
		for (Py_ssize_t i=0; i < ids->size; i++) {
		Py_XINCREF(ids->array[i]);
		}
		#endif /* Py_DEBUG */
		clear_interned_dict(interp);
		}

Expand Down