Hi, I sometimes see the following exceptions when shutting down my app (using Python 2.5.1):  Unhandled exception in thread started by  Error in sys.excepthook:    Original exception was:  Exception in thread Thread-3 (most likely raised during interpreter shutdown):  Traceback (most recent call last):    File "/usr/local/lib/python2.5/threading.py", line 460, in __bootstrap    File "/usr/local/lib/python2.5/threading.py", line 440, in run    File "/home/yang/local/armed/lib/python2.5/site-packages/afx/threads.py", line 71, in worker    File "/usr/local/lib/python2.5/Queue.py", line 176, in get    File "/usr/local/lib/python2.5/threading.py", line 248, in notify  <type 'exceptions.TypeError'>: exceptions must be classes, instances, or strings (deprecated), not NoneType  Unhandled exception in thread started by  Error in sys.excepthook:    Original exception was:  Exception in thread Thread-6 (most likely raised during interpreter shutdown):  Traceback (most recent call last):    File "/usr/local/lib/python2.5/threading.py", line 460, in __bootstrap    File "/usr/local/lib/python2.5/threading.py", line 440, in run    File "/home/yang/local/armed/lib/python2.5/site-packages/afx/threads.py", line 71, in worker    File "/usr/local/lib/python2.5/Queue.py", line 176, in get    File "/usr/local/lib/python2.5/threading.py", line 248, in notify  <type 'exceptions.TypeError'>: exceptions must be classes, instances, or strings (deprecated), not NoneType  Unhandled exception in thread started by  Error in sys.excepthook:    Original exception was:Here is the code from my application:        def worker():            debug( 'starting worker' )            while True:                msg = i.get() # <-- THIS IS LINE 71                if msg is stop_msg: break                resultbuf, func, args, kwargs = msg                result, exc = None, None                try:                    result = func( *args, **kwargs )                except:                    t, v, tb = exc_info()                    exc = t, v, tb.tb_next                o.put( ( resultbuf, result, exc ) )                s.send( 'x' ) # assuming socket.send is thread-safe            debug( 'stopping worker' )Here is the origin of the exception (in threading.py):    def notify(self, n=1):        assert self._is_owned(), "notify() of un-acquire()d lock" # <-- THIS IS LINE 248        __waiters = self.__waiters        waiters = __waiters[:n]        if not waiters:            if __debug__:                self._note("%s.notify(): no waiters", self)            return        self._note("%s.notify(): notifying %d waiter%s", self, n,                   n!=1 and "s" or "")        for waiter in waiters:            waiter.release()            try:                __waiters.remove(waiter)            except ValueError:                passI'm not sure why this is happening. The threads are not daemon threads; I terminate them cleanly. When I get a SIGINT (I usu. shut down my app with ctrl-C), I enqueue n stop_msg's to the 'i' Queue so that the n workers can all exit.Note I usually launch 5 workers, so I'm not consistently getting an exception per worker. Also, I've been unable to reproduce this at will.

Do you join() the worker threads, waiting until they finish, before exiting the main thread?

No, as they are not daemon threads.

I'm getting the same kind of errors.I'm using a lot of threads, and one of them throws this thread nearly everytime I close my program (by gtk.main_quit)It seems that python sets every variable to None, and then wakeup sleeping threads, which then crash, as they try to work with the None variablesException in thread Thread-3 (most likely raised during interpreter shutdown):Traceback (most recent call last):  File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap  File "/home/thomas/Programmering/python/skak/0.7/lib/pychess/System/ThreadPool.py", line 49, in run  File "/usr/lib/python2.4/Queue.py", line 89, in put  File "/usr/lib/python2.4/threading.py", line 237, in notifyexceptions.TypeError: exceptions must be classes, instances, or strings (deprecated), not NoneTypeUnhandled exception in thread started by Error in sys.excepthook

I see the same problem.   I'm not sure my code is clean,but I'll be darned if I can find the problem.I've traced my code with print statements,and I see some threads reach their return statement,but the process hangs when you try to join() that thread.It seems that there is something in the clean-up codein threading.py (presumably) that can hang under someobscure circumstances.As a result, these threads don't terminate on time,and various exceptions happen as python is beingdismantled and all the variables de-allocated.

Though these exceptions while shutting down are mostly harmless, theyare very noisy and must be squelched in a production application.  Hereis the patch which we at Solido Design (www.solidodesign.com) are usingto hide the exceptions.  Note that this doesn't fix the problem, but itdoes hide the exception for the purposes of releasing an application tothe public.

I think the general idea of the problem has been stated, but I figured Iwould state the official issue.  When Python begins to shutdown it takeseach module and sets each variable in the global namespace to None. If athread has not terminated before the interpreter terminates then thethread tries to use a global variable which has been set to None.This is not about to change since this occurs because of coding"errors". You must make sure that either your thread is as safe as a__del__ method (which means no global namespace access) or you can't letthe app exit until you are positive all of your threads have terminated,not just asked them to shutdown since this is all asynchronous.

Py_Main calls WaitForThreadShutdown before calling Py_Finalize, whichshould wait for all these threads to finish shutting down before itstarts wiping their globals.However, if SystemExit is raised (such as via sys.exit()), Py_Exit iscalled, and it directly calls Py_Finalize, bypassing theWaitForThreadShutdown.Can someone who's experienced this bug check if they're usingSystemExit/sys.exit?

To put it another way: SystemExit turns non-daemon threads into daemonthreads.  This is clearly wrong.  Brent, could you reopen the bug?

Hold on, why is that wrong? What if the threads block forever,preventing shutdown? sys.exit() is not exactly some namby-pamby functionbut a forced shutdown of the interpreter that should guarantee that theinterpreter quits. Changing its semantics now would take that away.

I disagree.  sys.exit() attempts to gracefully shutdown the interpreter,invoking try/finally blocks and the like.  If you want to truly forceshutdown you should use os.abort() or os._exit().Note that, as python doesn't call a main function, you have to usesys.exit() to have an exit status.

OK, I will re-open to see if some other core developer wants to takethis on, but personally I am passing.

> which means no global namespace accessDoes that mean that you cannot use len and range in a Thread?

> > which means no global namespace access> Does that mean that you cannot use len and range in a Thread?No, it means you have to be careful if you do. Shutting down properlywill take care of things. Otherwise you need to save a referencelocally (either on an object or as a local variable) and use thatreference instead of relying on the one defined in the globalnamespace.

This bug was introduced byr53249, which was fixing bug#1566280.Fixed by moving the WaitForThreadShutdown call into Py_Finalize, so allshutdown paths use it.  I also tweaked the name to follow local helperfunction conventions.Martin, since you did the previous fix, can you review this one?

Oh, and the patch includes a testcase.  The current test_threading.pydoesn't work with older versions, but a freestanding version of thistestcase passes in 2.1 to 2.4, fails in 2.5 and trunk, and passes withthe patch.

The patch looks good to me.And since Py_Finalize() claims to "destroy all sub-interpreters" and"free all memory allocated by the Python interpreter", I guess yourapproach makes sense. Can you commit?

Nope, no access.

Ok, I'll do it then!

Patch was committed in trunk, py3k and 3.1. Waiting for 2.6 to beunfrozen before I commit it there too.

Backported to 2.6 inr75749.

Any idea if there is a nightly build for Python 2.6? The latest release was 2.6.4 and was 2 days before submitting the patch. Or the only alternative is to build it myself? Any ideas on when we could see 2.6.5? - I tried to look for a release timeline but I wasn't able to locate one.

I have seen somewhere (ask google), that python 2.6.5 would be released mid-march.But except for a few platforms, python.org does not provide compiled binaries.

According to Barry's latest email on the subject, the dates are:2009-03-01 Python 2.6.5 rc 12009-03-15 Python 2.6.5 finalAnd no, there are no nightly builds, you have to build it yourself, I'm afraid.

Looks like this influenced mod_wsgi as well:http://groups.google.com/group/modwsgi/browse_thread/thread/ba82b2643564d2dd