Worker processes with multiprocessing.Pool live for the duration of thePool.  If the tasks they run happen to leak memory (from a C extensionmodule, or from creating cycles of unreachable objects, etc) or openfiles or other resources, there's no easy way to clean them up.Similarly, if one task passed to the pool allocates a large amount ofmemory, but further tasks are small, that additional memory isn'treturned to the system because the process involved hasn't exited.A common approach to this problem (as used by Apache, mod_wsgi, andvarious other software) is to allow worker processes to exit (and bereplaced with fresh processes) after completing a specified amount ofwork.  The attached patch (against Python 2.6.2, but applies to variousother versions with some fuzz) implements this as optional new behaviourin multiprocessing.Pool().  An additional optional argument is specifiedfor the maximum number of tasks a worker process performs before itexits and is replaced with a fresh worker process.

Hi Charles; I don't see a doc update for this (see multiprocessing.rst) or unit tests. I'm not against it, but before I can commit this, I'll need those things

Alright, I'll add those.  Thanks for looking at it; first pass wasmostly to ensure it wouldn't be wasted work.

Updated patch attached; handles some of the Pool methods that weren'thandled before.  Now includes documentation and unit test additions as well.

Jesse: this is ready for your review now.  Thanks,Charles

Can someone review this patch?  I believe it's sufficient for inclusionnow, as it includes docs and unit tests, but if anything about itrequires further attention I'd be happy to listen to change requests. We'd like get this into mainline where everyone can benefit rather thancontinuing to maintain it out-of-tree.

I plan on reviewing the patch once my work with PyCon is completed, which should be within the next few weeks. The next release this would show up in (2.7/3.2) is not for some time.

Okay, thanks, Jesse.  Didn't realize the Con was on.

Well, it's in feb, but I'm involved in some intense planning right now.

Hi Jesse -- Any chance you'll be able to review this in time for it tomake it into trunk for the 2.7 alpha release?Charles

On Sat, Nov 14, 2009 at 11:43 AM, Charles Cazabon<report@bugs.python.org> wrote:>> Charles Cazabon <charlesc-python@pyropus.ca> added the comment:>> Hi Jesse -- Any chance you'll be able to review this in time for it to> make it into trunk for the 2.7 alpha release?2.7 isn't slated until next year some time, so yes.

Ping... two alphas into 2.7.  Have you had a chance to review this functionality?Thanks.

I'm fine with the functionality; I'm going to test it out and look at committing it by mid-week next week. I apologize, I've been pretty maxed out.

No problem, Jesse, I realize you're busy.  I've updated the patch very slightly to handle an issue I had where the worker maintainer hung once.

I'm working on this now; I'm going to need to port the patch to trunk before moving forward with it. Shouldn't take me long.

Attached is the ported patch for Python 2.7/trunk. Please review it to make sure I didn't completely flub anything. I noticed you had forgotten the maxtasksperchild argument in the unit test, so I added that. I also expanded the docs a little but, paraphrasing your original post w.r.t to the justification.Let me know what you think, or if you have issues. Once you give the go ahead, I will commit to trunk and then merge to Py3k. Once again, thanks for the patch

Thanks, Jesse -- it looks good.  If there are bugs remaining in the patch, they're mine and not yours.

Committed to trunk inr77794Merged to Py3k inr77795