I have a regression from Python 2.5 to Python SVN (would-be-2.6).  Ibelieve this because of class attribute caching.  The problem shownbelow happens because AbstractGCProtector is an extension class.  So,Python interpreter doesn't have a chance to notice set_default() methodbelow changes a class attribute.>>> from notify.all import *>>> original_protector = AbstractGCProtector.default>>> new_protector = FastGCProtector ()>>> AbstractGCProtector.default is new_protectorFalse>>> AbstractGCProtector.default is original_protectorTruePlease note that this a regression.  This code works as expected in 2.3and 2.5.

Eh, disregard that, I missed one line with set_default() call.  Still,the unit test fails...

OK, here it is:>>> from notify.all import *>>> original_protector = AbstractGCProtector.default>>> new_protector = FastGCProtector ()>>> AbstractGCProtector.set_default (new_protector)>>> AbstractGCProtector.default is new_protectorFalse>>> AbstractGCProtector.default is original_protectorTrueIt seems that this behaviour is somewhat random.  Sometimes theFalse/True lines are reversed, as expected.  I.e. it seems thatattribute cache is sometimes recomputed...

It would be very interesting to know what set_default() actually does.IOW, without the source code of the extension module we can't doanything about this.

set_default() is a static method to set 'default'.  Because of this:>>> AbstractGCProtector.default = 42Traceback (most recent call last):  File "<stdin>", line 1, in <module>TypeError: can't set attributes of built-in/extension type'notify.gc.AbstractGCProtector'About source code: go tohttp://download.gna.org/py-notify/  download,unpack and do './run-tests.py' at top level.  One test fails with PythonSVN precisely because of this problem.

I'm sorry, but I can't get this to run. With a clean 0.1.14 tarball, I getBuilding extension...running build_extbuilding 'notify.gc' extensioncreating buildcreating build/temp.linux-i686-2.5creating build/temp.linux-i686-2.5/notifyi686-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC-I/usr/include/python2.5 -c notify/gc.c -obuild/temp.linux-i686-2.5/notify/gc.ocreating build/lib.linux-i686-2.5creating build/lib.linux-i686-2.5/notifyi686-pc-linux-gnu-gcc -pthread -sharedbuild/temp.linux-i686-2.5/notify/gc.o -L/usr/lib -lpython2.5 -obuild/lib.linux-i686-2.5/notify/gc.so[1]    28189 segmentation fault  ~/devel/python/python run-tests.pywhen running with a trunk python (note the "2.5" in the paths...)When I build the extension manually and comment out the building commandin run-tests.py, I getNote that most of the time is spent in gc.collect() calls, not in thispackage..............................................................FatalPython error:Objects/classobject.c:2311 object at 0x82dd2bc hasnegative ref count -606348326[1]    28540 abort      ~/devel/python/python run-tests.py

Weird.  Does it even run with a stable Python (not trunk)?

Yes, runs fine with 2.5.

Can you run the pasted script (from the third comment) manually then? The crash might be related to the bug in question.

I've now built my trunk python without debugging enabled, and canreproduce your problem.Armin: the extension module directly modifies an extension type'stp_dict -- what should it do instead to make the cache happy?

Even if there is an easy workaround for the extension (or even a fix, ifmodifying 'tp_dict' is not legal), I don't think it would be acceptableto make a backward-incompatible change in 2.6.  I mean, sure Py-notifyis by no means a widely-used library, but can you guarantee that noother extension will get broken?

The issue seems similar to the one we had in ctypes when the methodattribute cache was implemented. Ctypes was corrected inr59943. Maybe similar changes are needed forthis extension.For example,  PyDict_SetItemString (AbstractGCProtector_Type.tp_dict, "default",new_protector)should be modified like this:  PyObject_SetAttr(AbstractGCProtector_Type, "default", new_protector)

It doesn't help:ERROR: test_default_property (test._gc.AbstractGCProtectorTestCase)----------------------------------------------------------------------Traceback (most recent call last):  File "/home/paul/notify/test/_gc.py", line 59, in test_default_property    AbstractGCProtector.set_default (original_protector)TypeError: can't set attributes of built-in/extension type'notify.gc.AbstractGCProtector'With this code:  if (PyObject_SetAttrString ((PyObject *) &AbstractGCProtector_Type,"default", new_protector)      == -1)    return NULL;

I don't see in general how the patch can be kept compatible withextension modules that change the tp_dict of arbitrary types.  All I canthink of is a way to be safe against extension modules that only changethe tp_dict of their own non-heap types (i.e. types defined in C).  Themethod cache is disabled for types that don't have thePy_TPFLAGS_HAVE_VERSION_TAG flag; so if we would leave this flag out ofPy_TPFLAGS_DEFAULT, non-heap types from extension modules would bydefault not use the cache.I'm not sure about the API for this.  Would we then need to putPy_TPFLAGS_HAVE_VERSION_TAG explicitly on all core types?  And about howto change tp_dict in a way that makes the cache happy - do we need toturn type_modified() into a public API?All in all, the lack of abstraction of the C API might kill the idea ofthis patch for CPython.

We can of course add something like in#1229239, which allows typeattributes to be set with PyObject_SetAttr, thereby updating the cache.

I personally think that this bug is a showstopper for the caching patchin 2.6.  Well, the problem can be deemed insignificant, but it is sure abreak of backward compatibility and, worse yet, it results in _very_obscure fails.  Even if type dictionary changes are not all that common,I'm sure there are extensions out there that do it.For Py3k things can be different.  I'm not sure what would be the bestway, but at least Py3k is not required to be compatible with 2.x in allaspects.

Ping

Guido, what say you, live with it, revert it, or apply Py_TPFLAGS_HAVE_VERSION_TAG to all core types?

Maybe there is a better solution along the following line: conditionallydefine Py_TPFLAGS_DEFAULT so that when compiling the Python core itincludes the Py_TPFLAGS_HAVE_VERSION_TAG, but when compiling extensionmodules it does not.  This should ensure compatibility with manyexisting extension modules.  (It would still break if they play trickslike modifying the tp_dict of types other than their own.)In addition, to support the method cache in newer C extension modules:* C types defined by C extension modules can include thePy_TPFLAGS_HAVE_VERSION_TAG explicitly to enable the cache;* new C API functions should be introduced to give an official way toaccess the tp_dict of a type, e.g. PyType_{Get,Set,Del}DictItemString().

I like Armin's latest proposal: have Py_TPFLAGS_DEFAULT includePy_TPFLAGS_HAVE_VERSION_TAG when compiling the core only.  ISTR there'sa way to do this, but I can't find it right now.

Please review the patch here:http://codereview.appspot.com/3005

Submitted asr65874.I will block this for 3.0; 3.0 extensions that want to mess with tp_dictmust explicitly disable this flag.

Do we want a test?

Sure, go right ahead.