Even after adding the current patch inhttp://bugs.python.org/issue3651there are many reference leaks.  This bug can be a placeholder for allthe reference leaks returned from:    ./python ./Lib/test/regrtest.py -R 3:2 -uall,-bsddb The current list is:test_unittest leaked [124, 124] references, sum=248test_array leaked [110, 110] references, sum=220test_audioop leaked [75, 75] references, sum=150test_binascii leaked [4, 4] references, sum=8test_binhex leaked [4, 4] references, sum=8test_codecs leaked [3, 3] references, sum=6test_ctypes leaked [9, 9] references, sum=18test_dbm leaked [194, 194] references, sum=388test_dbm_gnu leaked [2, 2] references, sum=4test_fcntl leaked [2, 2] references, sum=4test_file leaked [8, 8] references, sum=16test_fileio leaked [1, 1] references, sum=2test_memoryio leaked [3, 3] references, sum=6test_minidom leaked [5, 5] references, sum=10test_mmap leaked [307, 307] references, sum=614test_ossaudiodev leaked [2, 2] references, sum=4test_pickle leaked [130, 130] references, sum=260test_pickletools leaked [503, 503] references, sum=1006test_pyexpat leaked [1, 1] references, sum=2test_re leaked [4, 4] references, sum=8test_site leaked [88, 88] references, sum=176test_socket leaked [13, 13] references, sum=26test_sqlite leaked [17, 17] references, sum=34test_ssl leaked [82, 82] references, sum=164test_struct leaked [5, 5] references, sum=10test_unicode leaked [2, 2] references, sum=4test_urllib2_localnet leaked [3, 3] references, sum=6test_xmlrpc leaked [18, 18] references, sum=36test_xmlrpc_net leaked [1, 1] references, sum=2test_zlib leaked [10, 10] references, sum=20

As ofr66047, I get the following results (without "-uall", though):test_unittest leaked [124, 124] references, sum=248test_binascii leaked [1, 1] references, sum=2test_distutils leaked [141, 142] references, sum=283test_logging leaked [219, 147] references, sum=366test_multiprocessing leaked [0, 1] references, sum=1test_pickle leaked [1, 1] references, sum=2test_pickletools leaked [1, 1] references, sum=2test_popen leaked [37, 0] references, sum=37test_site leaked [88, 88] references, sum=176test_sqlite leaked [17, 17] references, sum=34test_unicode leaked [2, 2] references, sum=4test_urllib2_localnet leaked [3, 3] references, sum=6test_xmlrpc leaked [-84, 85] references, sum=124 tests skipped:    test_bsddb3 test_cProfile test_codecmaps_cn test_codecmaps_hk    test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses    test_dbm_gnu test_kqueue test_nis test_normalization    test_ossaudiodev test_pep277 test_socketserver test_startfile    test_tcl test_timeout test_urllib2net test_urllibnet test_winreg    test_winsound test_xmlrpc_net test_zipfile64

- the "test_site leaked [88, 88]" is the same as problem asissue3667.- test_unicodes leaks in PyUnicode_AsEncodedString (attached patch), andalso with:  str(memoryview(b'character buffers are decoded to unicode'), 'utf-8')I tried another patch, but I'm not sure: I get lost between all thesebuffers... a Py_DECREF(self->view.obj) in memory_releasebuf() seems to work.

>   str(memoryview(b'character buffers are decoded to unicode'), 'utf-8')> I tried another patch, but I'm not sure: I get lost between all these> buffers... a Py_DECREF(self->view.obj) in memory_releasebuf() seems to work.Could you open a separate issue for the latter? Adding a DECREF inmemory_releasebuf() isn't the right thing to do, because PyBuffer_Release()already does such a DECREF. I think the problem is rather in memory_getbuf(), ittries to take some strange shortcuts.Oh, and I realize that memoryobject doesn't have tp_traverse and tp_clear, whichlooks quite wrong...

issue3712 tracks the memoryview issues.

the leaks in test_pickle and test_pickletools are corrected by theattached patch.

Amaury, I believe the first part of encode-leak.patch is wrong, youshould Py_DECREF the bytearray after it has been converted to bytes, notbefore. Here is an alternate patch.

Oops, you are right of course.I remove my patch so we don't get confused.

With the two patches applied, we are now at:test_unittest leaked [124, 124] references, sum=248test_distutils leaked [141, 142] references, sum=283test_docxmlrpc leaked [85, -85] references, sum=0test_logging leaked [366, -366] references, sum=0test_os leaked [37, 0] references, sum=37test_site leaked [88, 88] references, sum=176test_smtplib leaked [0, 87] references, sum=87test_sqlite leaked [17, 17] references, sum=34test_telnetlib leaked [151, -151] references, sum=0test_unicode leaked [1, 1] references, sum=2test_urllib2_localnet leaked [3, 3] references, sum=6test_xmlrpc leaked [-85, 0] references, sum=-8524 tests skipped:    test_bsddb3 test_cProfile test_codecmaps_cn test_codecmaps_hk    test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses    test_dbm_gnu test_kqueue test_nis test_normalization    test_ossaudiodev test_pep277 test_socketserver test_startfile    test_tcl test_timeout test_urllib2net test_urllibnet test_winreg    test_winsound test_xmlrpc_net test_zipfile642 skips unexpected on linux2:

Did you look at the patch forissue3667 ? it should at least correcttest_site.

Ok, after the two patches plus the patch in#3667, I get the following:test_asyncore leaked [84, -84] references, sum=0test_distutils leaked [141, 142] references, sum=283test_docxmlrpc leaked [-85, 0] references, sum=-85test_logging leaked [219, -219] references, sum=0test_sqlite leaked [17, 17] references, sum=34test_unicode leaked [1, 1] references, sum=2test_urllib2_localnet leaked [3, 3] references, sum=6test_xmlrpc leaked [-6, -79] references, sum=-85

The patch for _pickle has been committed inr66227.

Numbers for the current py3k branch (without encode-leak2.patch):test_distutils leaked [141, 142] references, sum=283test_docxmlrpc leaked [-7, -85] references, sum=-92test_logging leaked [0, 219] references, sum=219test_poplib leaked [0, 84] references, sum=84test_sys leaked [0, 34] references, sum=34test_unicode leaked [1, 1] references, sum=2test_urllib2_localnet leaked [3, 3] references, sum=6test_xmlrpc leaked [192, -190] references, sum=2

The only one that is probably an issue based on Antoine's info is:test_unicode leaked [1, 1] references, sum=2I've seen test_urllib2_localnet leak 3 before.  I don't know that it'sa real leak.  I'm pretty sure it is not a regression though.

FWIW, applying encode-leak2.patch removes the leak in test_unicode.

Antoine, it seem that with encode-leak2.patch, the error path afterPyErr_WarnEx() leaks the value of "v".I rewrote the whole paragraph to make it more straightforward:- the normal case is tested first- all paths end with a "return", and no goto.- no need to test that PyBytes_FromStringAndSize returns a PyBytes...I find the code much easier to check in this form, but of course this isa subjective POV.

Le vendredi 05 septembre 2008 à 13:08 +0000, Amaury Forgeot d'Arc aécrit :> Antoine, it seem that with encode-leak2.patch, the error path after> PyErr_WarnEx() leaks the value of "v".Hmm, you are right.> I rewrote the whole paragraph to make it more straightforward:> - the normal case is tested first> - all paths end with a "return", and no goto.> - no need to test that PyBytes_FromStringAndSize returns a PyBytes...I'll take a look!

Amaury, your patch is much clearer indeed and it fixes the leak.

encode-leak3.patch applied inr66234.

Current status:test_distutils leaked [141, 142] references, sum=283test_logging leaked [0, -219] references, sum=-219test_smtplib leaked [0, 87] references, sum=87The distutils leak should be investigated, but the overall situation israther good now. The other, transient, leaks might be due to some threadbeing cleaned up too late or something.

test_distutils is also leaking the the trunk:test_distutils leaked [144, 144, 144, 144] references, sum=576

test_distutils will be difficult; the leak is around the "import xx" inLib/distutils/tests/test_build_ext.py.AndPython/import.c says:/* To prevent initializing an extension module more than once, we keep astatic dictionary 'extensions' keyed [...] by filename (for dynamicallyloaded modules). A copy of the module's dictionary is stored [...] */This dictionary keeps growing with random filenames in the tempdirectory. I can't see a way to clean it.

> test_distutils will be difficult; the leak is around the "import xx" in>Lib/distutils/tests/test_build_ext.py.> > AndPython/import.c says:> /* To prevent initializing an extension module more than once, we keep a> static dictionary 'extensions' keyed [...] by filename (for dynamically> loaded modules). A copy of the module's dictionary is stored [...] > */> > This dictionary keeps growing with random filenames in the temp> directory. I can't see a way to clean it.If it's just that (one leaked string per each extension module import),I think we can live with it.

It's not only the name, but a copy of the whole module dict just afterimport (that's why reload(sys) takes you back thesys.setdefaultencoding() function).Actually I found a (hackish) way to clean the 'extension' dict, but thisis not enough: the static variables in xxmodule.c cannot be cleared.Shall we exclude this test from the leak hunter?

> It's not only the name, but a copy of the whole module dict just after> import (that's why reload(sys) takes you back the> sys.setdefaultencoding() function).Ow.> Actually I found a (hackish) way to clean the 'extension' dict, but this> is not enough: the static variables in xxmodule.c cannot be cleared.> Shall we exclude this test from the leak hunter?I'd prefer not. If we hide this leak, we'll end up forgetting about itsexistence.

With the new module structure in 3.0, it should be possible to add acleanup function. It would be a good exercise; I don't know of anymodule defining such a function.

This issue has been fixed.