
This issue trackerhas been migrated toGitHub, and is currentlyread-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.
Created on2016-09-01 15:34 byOkko.Willeboordse, last changed2022-04-11 14:58 byadmin. This issue is nowclosed.
| Files | ||||
|---|---|---|---|---|
| File name | Uploaded | Description | Edit | |
| leaked_objects.txt | Okko.Willeboordse,2016-09-01 15:34 | List of leaked objects | ||
| Pull Requests | |||
|---|---|---|---|
| URL | Status | Linked | Edit |
| PR 552 | closed | dstufft,2017-03-31 16:36 | |
| Messages (13) | |||
|---|---|---|---|
| msg274144 -(view) | Author: Okko Willeboordse (Okko.Willeboordse) | Date: 2016-09-01 15:34 | |
Running;Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)] on win32platform.win32_ver() returns;('10', '10.0.10586', '', u'Multiprocessor Free') | |||
| msg274145 -(view) | Author: STINNER Victor (vstinner)*![]() | Date: 2016-09-01 15:35 | |
Hum, can you please elaborate the issue? What is leaked_objects.txt? | |||
| msg274146 -(view) | Author: Okko Willeboordse (Okko.Willeboordse) | Date: 2016-09-01 15:38 | |
I did a gc.get_objects() before and after platform.win32_ver() and printedobjects that were not present before calling platform.win32_ver().If I run platform.win32_ver() in a loop I see the memory increasing.On 1 September 2016 at 17:35, STINNER Victor <report@bugs.python.org> wrote:>> STINNER Victor added the comment:>> Hum, can you please elaborate the issue? What is leaked_objects.txt?>> ----------> nosy: +haypo>> _______________________________________> Python tracker <report@bugs.python.org>> <http://bugs.python.org/issue27932>> _______________________________________> | |||
| msg275520 -(view) | Author: Roundup Robot (python-dev)![]() | Date: 2016-09-10 01:05 | |
New changeset384c178cf823 by Steve Dower in branch '3.5':Issue#27932: Fixes memory leak in platform.win32_ver()https://hg.python.org/cpython/rev/384c178cf823New changeset31b7eaff5588 by Steve Dower in branch 'default':Issue#27932: Fixes memory leak in platform.win32_ver()https://hg.python.org/cpython/rev/31b7eaff5588 | |||
| msg275521 -(view) | Author: Steve Dower (steve.dower)*![]() | Date: 2016-09-10 01:06 | |
I fixed in 3.5 and 3.6, but I'm not set up for 2.7 so someone will have to backport it. | |||
| msg275672 -(view) | Author: Roundup Robot (python-dev)![]() | Date: 2016-09-10 18:54 | |
New changesetdbfcb3b9c3c1 by Steve Dower in branch '3.5':Issue#27932: Backs out changehttps://hg.python.org/cpython/rev/dbfcb3b9c3c1 | |||
| msg275673 -(view) | Author: Steve Dower (steve.dower)*![]() | Date: 2016-09-10 18:54 | |
Backed out because ofissue28059. I'll rewrite this to properly use native code. | |||
| msg275715 -(view) | Author: Eryk Sun (eryksun)*![]() | Date: 2016-09-10 23:44 | |
The leak is due the pointer-type cache that ctypes.POINTER uses. The type the pointer refers to is used as the key. In this case, VS_FIXEDFILEINFO is created each time win32_ver is called, so the pointer-type cache grows without bound. Example leak: >>> import psutil, platform >>> proc = psutil.Process() >>> proc.memory_info().rss 14704640 >>> for i in range(10000): ... v = platform.win32_ver() ... >>> proc.memory_info().rss 92704768 >>> for i in range(10000): ... v = platform.win32_ver() ... >>> proc.memory_info().rss 168861696Clearing the cache followed by a collect() reclaims the leaked memory for the most part: >>> import gc, ctypes >>> gc.collect() 333 >>> proc.memory_info().rss 168849408 >>> ctypes._pointer_type_cache.clear() >>> gc.collect() 740000 >>> proc.memory_info().rss 20303872It's a moot point, since Steve plans to re-implement this check in C, but the minimal change to fix this leak is to bypass the pointer-type cache by manually subclassing ctypes._Pointer: class PVS_FIXEDFILEINFO(_Pointer): _type_ = VS_FIXEDFILEINFO pvi = PVS_FIXEDFILEINFO()There's no more leak after this change: >>> import psutil, platform >>> proc = psutil.Process() >>> proc.memory_info().rss 15450112 >>> for i in range(10000): ... v = platform.win32_ver() ... >>> proc.memory_info().rss 16592896 >>> for i in range(10000): ... v = platform.win32_ver() ... >>> proc.memory_info().rss 16601088 | |||
| msg275744 -(view) | Author: STINNER Victor (vstinner)*![]() | Date: 2016-09-11 08:15 | |
> pointer-type cache grows without boundMaybe we should also fix ctypes to use an LRU cache like@functools.lru_cache? A cache without limit doesn't seem like a goodidea. | |||
| msg275753 -(view) | Author: Eryk Sun (eryksun)*![]() | Date: 2016-09-11 09:19 | |
Limiting the pointer-type cache could be a problem. It's expected that POINTER() returns a reference to an existing pointer type. ctypes uses Python types to ensure compatible C data types. For example: LP_c_int = ctypes.POINTER(ctypes.c_int) class LP_c_int_uncached(ctypes._Pointer): _type_ = ctypes.c_int >>> arr = (LP_c_int * 2)() >>> ctypes.POINTER(ctypes.c_int) is LP_c_int True >>> arr[0] = ctypes.POINTER(ctypes.c_int)() >>> arr[1] = LP_c_int_uncached() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: incompatible types, LP_c_int_uncached instance instead of LP_c_int instanceThe docs could warn users that the cache used by ctypes.POINTER(), and implicitly by ctypes.pointer(), will leak memory if it's called with a type that's created at function scope instead of at module or class-body scope. | |||
| msg276859 -(view) | Author: Roundup Robot (python-dev)![]() | Date: 2016-09-17 23:43 | |
New changeset9b0d661a16de by Steve Dower in branch '2.7':Issue#27932: Prevent memory leak in win32_ver().https://hg.python.org/cpython/rev/9b0d661a16de | |||
| msg276861 -(view) | Author: Roundup Robot (python-dev)![]() | Date: 2016-09-18 00:30 | |
New changesetdb4b254f5df4 by Steve Dower in branch '3.5':Issue#27932: Prevent memory leak in win32_ver().https://hg.python.org/cpython/rev/db4b254f5df4New changeset14fca2c8eb36 by Steve Dower in branch '3.6':Issue#27932: Prevent memory leak in win32_ver().https://hg.python.org/cpython/rev/14fca2c8eb36New changeset19ceb5034fe5 by Steve Dower in branch 'default':Issue#27932: Prevent memory leak in win32_ver().https://hg.python.org/cpython/rev/19ceb5034fe5 | |||
| msg276862 -(view) | Author: Steve Dower (steve.dower)*![]() | Date: 2016-09-18 00:32 | |
Phew. Doing that in a reasonable way for all the active versions was a pain, but I think I managed it.In short:* 2.7 gets the fix proposed by Eryk Sun* 3.5 gets a private sys.getwindowsversion()._platform_version member* 3.6+ gets a public/documented sys.getwindowsversion().platform_version memberAFAICT the implementation is solid, but please shout if you spot anything weird/wrong. | |||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:58:35 | admin | set | github: 72119 |
| 2017-03-31 16:36:14 | dstufft | set | pull_requests: +pull_request898 |
| 2016-09-21 16:08:56 | steve.dower | set | status: open -> closed resolution: fixed stage: commit review -> resolved |
| 2016-09-18 00:32:51 | steve.dower | set | stage: commit review messages: +msg276862 versions: + Python 3.7 |
| 2016-09-18 00:30:26 | python-dev | set | messages: +msg276861 |
| 2016-09-17 23:43:53 | python-dev | set | messages: +msg276859 |
| 2016-09-11 09:19:11 | eryksun | set | messages: +msg275753 |
| 2016-09-11 08:15:52 | vstinner | set | messages: +msg275744 |
| 2016-09-10 23:44:30 | eryksun | set | nosy: +eryksun messages: +msg275715 |
| 2016-09-10 18:54:59 | steve.dower | set | assignee:steve.dower messages: +msg275673 |
| 2016-09-10 18:54:03 | python-dev | set | messages: +msg275672 |
| 2016-09-10 01:06:01 | steve.dower | set | messages: +msg275521 versions: + Python 3.5, Python 3.6 |
| 2016-09-10 01:05:25 | python-dev | set | nosy: +python-dev messages: +msg275520 |
| 2016-09-01 15:38:35 | Okko.Willeboordse | set | messages: +msg274146 |
| 2016-09-01 15:35:39 | vstinner | set | nosy: +vstinner messages: +msg274145 |
| 2016-09-01 15:34:31 | Okko.Willeboordse | create | |