PyErr_Warn() does an implicit import.  Thus, ifPyErr_Warn() is called on a thread while the mainthread holds the import lock, and the main thread thensubsequently waits for the child thread, Python willdeadlock.  The builtin 'apply' now calls PyErr_Warn(),so simply calling 'apply' on another thread may causethe deadlock.Attaching a sample case.  Executing 'python -c "importhang_apply"' will cause Python to deadlock.  Commentingout the call to "apply" in that file will prevent thedeadlock (even though the apply() in question iseffectively a no-op)The example is a little contrived, but is extractedfrom real code of mine that saw this hang.  The codepath is:* Main thread acquires import lock to import the module.* Module import spawns a new thread.  This new threadcalls apply()* builtin_apply() calls PyErr_Warn* PyErr_Warn does an import of the warnings module, andattempts to acquire the import lock.* Main thread still waiting for child thread tocomplete, but still holds import lock.Note that removing the call to PyErr_Warn() inbuiltin_apply also solves this particular hang -however, it seems like this is a potential time bomb. A potential solution would be to prevent PyErr_Warnfrom doing an import - this would mean importing'warnings' at startup, and keeping a static referencein errors.c.  Other thoughts welcome.

Logged In: YES user_id=33168I can't think of any other/better solution.  Any idea ifthere are there other ticking bombs like this?

Logged In: YES user_id=14198This simple strategy doesn't work - avoiding the import of'warnings' works fine, until 'warnings' imports 'linecache'!I'll look at how we can simply throw the warning away if adeadlock would occur.

Logged In: YES user_id=14198The best I can come up with here is exposing the import lockto the C API, with a "wait" flag.This will allow a C function to reliably determine if an"import" work block, or acquire the lock if not.  It canthen complete its import before releasing the lock.  If theimport would block, then it can take whatever action isnecessary - in the case of PyErr_Warn, it dumps the warningto stdout.Attaching a patch.  It exposes (to the core, but not inheaders) two functions:extern int PyImport_LockImport(int wait);extern void PyImport_UnlockImport(void);PyErr_Warn then uses these.If we do go this route, it makes sense to make thesefunctions truly public (ie, add them to the headers), andcleanup import.c appropriately.  I didn't do this in theinterests of keeping the patch small so it can be easilydigested.Comments?  Other ideas?

Logged In: YES user_id=6380No time to review everything here, but maybe PyErr_Warnshould not do an import? The import could be done at startuptime (when site.py is also imported) and saved in theinterpreter state.

Logged In: YES user_id=6380Oh, and the import of linecache by warnings can be put atthe top of warnings, if that's an issue.

Logged In: YES user_id=14198Yes, the linecache import was the problem.  I abandondedthat approach mainly to avoid yet another import at startup. Should 'warnings' get new features, all required imports inthe future will also need to be imported at process startup. It also struck me as a time bomb.I will make an alternative patch that moves the imports.

Logged In: YES user_id=14198Checking in a patch that avoids the import, thereby avoidingthe hang.  As suggested, the "linecache" import in warningshad to be moved.Even though I checked it in, I attach the patch here, andhave assigned it to Guido for review.  If happy, just markit as closed./cvsroot/python/python/dist/src/Python/errors.c,v  <--  errors.cnew revision: 2.76; previous revision: 2.75/cvsroot/python/python/dist/src/Python/pythonrun.c,v  <-- pythonrun.cnew revision: 2.178; previous revision: 2.177/cvsroot/python/python/dist/src/Lib/warnings.py,v  <-- warnings.pynew revision: 1.19; previous revision: 1.18

Logged In: YES user_id=6380I'm not happy with this solution, since it breaks formultiple interpreters. So keeping this open. Will thinkabout what to do instead after 2.3a2 is released.

Logged In: YES user_id=92689Here's an issue with the patch I just ran into: it seems this patch causes the first Python module to be imported to be warnings.py instead of site.py. This means site.py doesn't have full control of the import mechanism anymore, as all modules loaded by warnings.py will already be loaded. Can't the PyModule_WarningsModule global be set _after_ site.py has run?Use case: I'm building standalone applications in which the import bootstrapping happens in a custom site.py, assuming it will be the first Python module that will be imported. It sets up sys.path appropriately. Now it turns out that when my site.py gets run (eg.) os.py is already loaded from my system-wide installation, instead of from the zip file embedded in the application. This means I can't reliably test whether my built app really works when there's no Python installation available.

Logged In: YES user_id=6380Good point. But loading site.py first means that warningsissued as a result of importing site.py are not handledproperly (they will be dumped to stderr, which is betterthan nothing). Well, I have to revisit this anyway; I'llmake sure to take this issue into account as well.

Logged In: YES user_id=92689Additional info regarding the current situation (with the patch in place):if warnings.py can't be imported before site.py is run, _all_ warnings issued from C will be printed to stderr, even if warnings.py _can_ be imported later.I think it's fair to assume that any warning issued during the import of site.py signifies that something is pretty broken, so having them dumped to stderr isn't all that bad.If you and Mark agree, I'll move the import of warnings.py to after initsite().

Logged In: YES user_id=6380OK, as a stopgap measure that sounds like a good idea.(I wonder if -S should also suppress loading warnings? Or isthat too drastic?)

Logged In: YES user_id=92689Only if PyErr_Warn() would still also attempt to do the import if PyModule_WarningsModule is NULL. But that means that deadlocks can still occur when -S is used. But I _do_ like it that no file system imports are done by Python itself with -S. And I actually don't like it at all that warnings.py always gets imported, even if it's never needed.

Logged In: YES user_id=6380OK, implement that.

Logged In: YES user_id=92689Well, since you said the issue needs to be revisited anyway, I'll just leave it at moving the warnings.py import after initsite().What I don't like is that warnings.py is imported preemptively to begin with, not just with -S. Whether that's resolvable I don't know, but if it is, it would be better than not importing warnings.py with -S.

Logged In: YES user_id=31392Did this get resolved for 2.3 or delayed to 2.4?

Logged In: YES user_id=92689This is from the CVS log of pythonrun.c:revision 2.179date: 2003/02/25 20:25:12;  author: jvr;  state: Exp;  lines: +2 -2Addendum to#683658:import warnings.py _after_ site.py has run. This ensures that site.py is again the first .py to be imported, giving it back full control over sys.path.I'm quite confident this is good enough for 2.3, but the issue should be revisited at some point, so I'm not sure we should close this bug. I'll lower the priority.

Is this still an issue for 2.5? If Python 2.5 may still dead lock thensomebody should back port my PyImport_ImportModuleNoBlock() function.

No, backporting that to 2.5 isn't an option; it introduces a new featureand breaks binary backwards compatibility by removing thePyImport_ImportModuleEx entry point from the executable (turning it intoa macro).

Let's close this unless there's a response from one of the originalstakeholders within a week (i.e. by Jan 11 2008).