Movatterモバイル変換


[0]ホーム

URL:


homepage

Issue38379

This issue trackerhas been migrated toGitHub, and is currentlyread-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title:finalizer resurrection in gc
Type:behaviorStage:resolved
Components:Interpreter CoreVersions:Python 3.9, Python 3.8, Python 3.7
process
Status:closedResolution:fixed
Dependencies:Superseder:
Assigned To: terry.reedyNosy List:
Priority:normalKeywords:patch

Created on2019-10-05 17:31 bytim.peters, last changed2022-04-11 14:59 byadmin. This issue is nowclosed.

Files
File nameUploadedDescriptionEdit
zleak.pytim.peters,2019-10-05 17:31
Pull Requests
URLStatusLinkedEdit
PR 16658mergedtim.peters,2019-10-08 14:27
PR 16683mergedpablogsal,2019-10-09 21:07
PR 16685mergedpablogsal,2019-10-09 21:11
PR 16687mergedpablogsal,2019-10-10 00:34
Messages (13)
msg354020 -(view)Author: Tim Peters (tim.peters)*(Python committer)Date: 2019-10-05 17:31
While people are thinking about gc, zleak.py shows a small bug, and a possible opportunity for improvement, in the way gc treats finalizers that resurrect objects.The bug:  the stats keep claiming gc is collecting an enormous number of objects, but in fact it's not collecting any.  Objects in the unreachable set shouldn't add to the "collected" count unless they _are_ collected.Output:resurrectingcollect 2000002gen 2 stats {'collections': 2, 'collected': 2000002, 'uncollectable': 0}resurrectingcollect 4000004gen 2 stats {'collections': 3, 'collected': 6000006, 'uncollectable': 0}resurrectingcollect 6000006gen 2 stats {'collections': 4, 'collected': 12000012, 'uncollectable': 0}resurrectingcollect 8000008gen 2 stats {'collections': 5, 'collected': 20000020, 'uncollectable': 0}resurrectingcollect 10000010gen 2 stats {'collections': 6, 'collected': 30000030, 'uncollectable': 0}...Memory use grows without bound, and collections take ever longer.The opportunity:  if any finalizer resurrects anything, gc gives up.  But the process of computing whether anything was resurrected also determines which initially-trash objects are reachable from the risen dead.  Offhand I don't see why we couldn't proceed collecting what remains trash.  Then zleak.py would reclaim everything instead of nothing.Sketch:  rather than just set a flag, check_garbage() could move now-reachable objects to the old generation (and, for each one moved, decrement the count of collected objects).  Then delete_garbage() could proceed on what remains in the unreachable list.
msg354041 -(view)Author: Tim Peters (tim.peters)*(Python committer)Date: 2019-10-06 18:33
Just noting that check_garbage() currently only determines which trash objects are now directly reachable from outside.  To be usable for the intended purpose, it would need to go on to compute which trash objects are reachable from those too.Maybe a new functionstatic voiddeduce_unreachable(PyGC_Head *base, PyGC_Head *unreachable)that packaged the steps through move_unreachable().  The main function would use that on `young`, and check_garbage() on `unreachable` to find the _still_ unreachable objects.I don't care about the expense.  Outside of zleak.py, the number of unreachable objects is usually small, so it's usually cheap to make another pass over just them.
msg354213 -(view)Author: Tim Peters (tim.peters)*(Python committer)Date: 2019-10-08 14:28
PR 16658 aims to repair the stats reported.
msg354290 -(view)Author: Tim Peters (tim.peters)*(Python committer)Date: 2019-10-09 17:37
New changesetecbf35f9335b0420cb8adfda6f299d6747a16515 by Tim Peters in branch 'master':bpo-38379:  don't claim objects are collected when they aren't (#16658)https://github.com/python/cpython/commit/ecbf35f9335b0420cb8adfda6f299d6747a16515
msg354291 -(view)Author: Tim Peters (tim.peters)*(Python committer)Date: 2019-10-09 18:08
I checked the stat fix into master, but GH failed to backport to 3.7 or 3.8 and I'm clueless.  More info in the PR.  Does someone else here know how to get a backport done?
msg354296 -(view)Author: Pablo Galindo Salgado (pablogsal)*(Python committer)Date: 2019-10-09 21:14
Tim, I have created backports for 3.8 and 3.7 (PR 16683,PR 16685). In my case cherry_pickerecbf35f9335b0420cb8adfda6f299d6747a16515 3.7 and cherry_pickerecbf35f9335b0420cb8adfda6f299d6747a16515 3.8 works after fixing merge conflicts. Maybe there is something going on in your git repo (maybe you didn't fetch from master first and that is why git log fails?).
msg354297 -(view)Author: miss-islington (miss-islington)Date: 2019-10-09 21:25
New changeset0bd9fac7a866ec886ae8f93f9c24dcda6d436929 by Miss Islington (bot) (Pablo Galindo) in branch '3.8':[3.8]bpo-38379:  don't claim objects are collected when they aren't  (GH-16658) (GH-16683)https://github.com/python/cpython/commit/0bd9fac7a866ec886ae8f93f9c24dcda6d436929
msg354299 -(view)Author: miss-islington (miss-islington)Date: 2019-10-09 21:42
New changeseta081e931505f190b5ccdff9a781e59b3f13fcc31 by Miss Islington (bot) (Pablo Galindo) in branch '3.7':[3.7]bpo-38379:  don't claim objects are collected when they aren't (GH-16658) (GH-16685)https://github.com/python/cpython/commit/a081e931505f190b5ccdff9a781e59b3f13fcc31
msg354589 -(view)Author: Pablo Galindo Salgado (pablogsal)*(Python committer)Date: 2019-10-13 15:49
New changeset466326dcdf038b948d94302c315be407c73e60d1 by Pablo Galindo in branch 'master':bpo-38379: Don't block collection of unreachable objects when some objects resurrect (GH-16687)https://github.com/python/cpython/commit/466326dcdf038b948d94302c315be407c73e60d1
msg354605 -(view)Author: Tim Peters (tim.peters)*(Python committer)Date: 2019-10-13 22:17
Everything here has been addressed, so closing this.  zleak.py can apparently run forever now without leaking a byte :-)
msg375518 -(view)Author: Lewis Gaul (LewisGaul)*Date: 2020-08-16 16:49
I noticed this bug is mentioned in the 3.9 release notes with a note similar to the title of the 4th PR: "garbage collection does not block on resurrected objects".I can't see any mention of a blocking issue here on the issue:> The bug:  the stats keep claiming gc is collecting an enormous number of objects, but in fact it's not collecting any.  Objects in the unreachable set shouldn't add to the "collected" count unless they _are_ collected.Would someone be able to elaborate on the blocking issue that was fixed as part of this BPO?
msg375520 -(view)Author: Tim Peters (tim.peters)*(Python committer)Date: 2020-08-16 19:56
I suspect you're reading some specific technical meaning into the word "block" that the PR and release note didn't intend by their informal use of the word. But I'm unclear on what technical meaning you have in mind.Before the change, gc "just gave up" after seeing a resurrection, ending the then-current cyclic gc run. It that sense, yes, resurrection "blocked" gc from making progress. It did not, e.g., "block" the interpreter in the sense of deadlock, or of waiting for some lock to be released, or of waiting for a network request to respond, or ...
msg375521 -(view)Author: Lewis Gaul (LewisGaul)*Date: 2020-08-16 20:02
You're right that's how I had interpreted it, thanks for clarifying.I was wondering if this could be related to an issue I've hit with gc.collect() getting slower and slower in a test suite, but that now seems unlikely, so I won't go into that here.
History
DateUserActionArgs
2022-04-11 14:59:21adminsetgithub: 82560
2021-11-04 14:28:28eryksunsetmessages: -msg405696
2021-11-04 14:28:16eryksunsetnosy: -terry.reedy,ahmedsayeed1982

components: + Interpreter Core, - IDLE
versions: + Python 3.7, Python 3.8, Python 3.9, - Python 3.6
2021-11-04 12:10:01ahmedsayeed1982setversions: + Python 3.6, - Python 3.7, Python 3.8, Python 3.9
nosy: +ahmedsayeed1982,terry.reedy, -tim.peters,nascheme,pitrou,pablogsal,miss-islington,nanjekyejoannah,LewisGaul

messages: +msg405696

assignee:terry.reedy
components: + IDLE, - Interpreter Core
2020-08-16 20:02:39LewisGaulsetmessages: +msg375521
2020-08-16 19:56:34tim.peterssetmessages: +msg375520
2020-08-16 16:49:27LewisGaulsetnosy: +LewisGaul
messages: +msg375518
2019-10-13 22:17:33tim.peterssetstatus: open -> closed
resolution: fixed
messages: +msg354605

stage: patch review -> resolved
2019-10-13 15:49:02pablogsalsetmessages: +msg354589
2019-10-10 00:34:42pablogsalsetpull_requests: +pull_request16273
2019-10-09 21:42:57miss-islingtonsetmessages: +msg354299
2019-10-09 21:25:09miss-islingtonsetnosy: +miss-islington
messages: +msg354297
2019-10-09 21:14:53pablogsalsetmessages: +msg354296
2019-10-09 21:11:56pablogsalsetpull_requests: +pull_request16271
2019-10-09 21:07:10pablogsalsetstage: backport needed -> patch review
pull_requests: +pull_request16269
2019-10-09 18:08:58tim.peterssetstage: patch review -> backport needed
messages: +msg354291
versions: + Python 3.7, Python 3.8
2019-10-09 17:37:41tim.peterssetmessages: +msg354290
2019-10-08 23:05:28pablogsalsetnosy: +pablogsal
2019-10-08 14:28:52tim.peterssetmessages: +msg354213
2019-10-08 14:27:02tim.peterssetkeywords: +patch
stage: needs patch -> patch review
pull_requests: +pull_request16241
2019-10-07 17:58:48nanjekyejoannahsetnosy: +nanjekyejoannah
2019-10-06 18:33:41tim.peterssetmessages: +msg354041
2019-10-05 17:31:26tim.peterscreate
Supported byThe Python Software Foundation,
Powered byRoundup
Copyright © 1990-2022,Python Software Foundation
Legal Statements

[8]ページ先頭

©2009-2026 Movatter.jp