Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork939
Closed
Labels
Milestone
Description
Sorry, but I believe I found another issue. It appears (maybe I'm missing something here, because this seems very strange to me) that file descriptors are left open after the Repo object is unused or destroyed.
(foo)jantman@jarvis:pts/8:~/tmp/foo$ cat test.py#!/usr/bin/env pythonimport gitimport commandsimport osimport timepid = os.getpid()def show_lsof(): global pid cmd = "lsof -p {p} | grep -v /usr/lib".format(p=pid) print("=============================") print("running {c}:".format(c=cmd)) print(commands.getoutput(cmd)) print("=============================")for rname in ['testrepo', 'testrepo2', 'testrepo3']: print("=> Creating repo object for {rname}:".format(rname=rname)) r = git.Repo(rname) show_lsof() print("=> iterating commits") for c in r.iter_commits(): pass show_lsof() time.sleep(2) print("=> del r") del r time.sleep(2) show_lsof()(foo)jantman@jarvis:pts/8:~/tmp/foo$ ./test.py => Creating repo object for testrepo:=============================running lsof -p 5325 | grep -v /usr/lib:COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEpython 5325 jantman cwd DIR 0,30 200 73513 /tmp/jantman/foopython 5325 jantman rtd DIR 8,5 4096 2 /python 5325 jantman txt REG 0,30 6240 73547 /tmp/jantman/foo/bin/python2python 5325 jantman mem REG 0,30 33020 73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.sopython 5325 jantman 0u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 1u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 2u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 3r FIFO 0,8 0t0 88651 pipe==============================> iterating commits=============================running lsof -p 5325 | grep -v /usr/lib:COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEpython 5325 jantman cwd DIR 0,30 200 73513 /tmp/jantman/foopython 5325 jantman rtd DIR 8,5 4096 2 /python 5325 jantman txt REG 0,30 6240 73547 /tmp/jantman/foo/bin/python2python 5325 jantman mem REG 0,30 33020 73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.sopython 5325 jantman mem REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 0u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 1u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 2u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 3r FIFO 0,8 0t0 91431 pipepython 5325 jantman 4r REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 5r REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx==============================> del r=============================running lsof -p 5325 | grep -v /usr/lib:COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEpython 5325 jantman cwd DIR 0,30 200 73513 /tmp/jantman/foopython 5325 jantman rtd DIR 8,5 4096 2 /python 5325 jantman txt REG 0,30 6240 73547 /tmp/jantman/foo/bin/python2python 5325 jantman mem REG 0,30 33020 73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.sopython 5325 jantman mem REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 0u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 1u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 2u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 3r FIFO 0,8 0t0 91434 pipepython 5325 jantman 4r REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 5r REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx==============================> Creating repo object for testrepo2:=============================running lsof -p 5325 | grep -v /usr/lib:COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEpython 5325 jantman cwd DIR 0,30 200 73513 /tmp/jantman/foopython 5325 jantman rtd DIR 8,5 4096 2 /python 5325 jantman txt REG 0,30 6240 73547 /tmp/jantman/foo/bin/python2python 5325 jantman mem REG 0,30 33020 73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.sopython 5325 jantman mem REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 0u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 1u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 2u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 3r FIFO 0,8 0t0 91435 pipepython 5325 jantman 4r REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 5r REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx==============================> iterating commits=============================running lsof -p 5325 | grep -v /usr/lib:COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEpython 5325 jantman cwd DIR 0,30 200 73513 /tmp/jantman/foopython 5325 jantman rtd DIR 8,5 4096 2 /python 5325 jantman txt REG 0,30 6240 73547 /tmp/jantman/foo/bin/python2python 5325 jantman mem REG 0,30 33020 73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.sopython 5325 jantman mem REG 0,30 3032 89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman mem REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 0u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 1u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 2u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 3r FIFO 0,8 0t0 87601 pipepython 5325 jantman 4r REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 5r REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman 6r REG 0,30 42220 89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 7r REG 0,30 3032 89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx==============================> del r=============================running lsof -p 5325 | grep -v /usr/lib:COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEpython 5325 jantman cwd DIR 0,30 200 73513 /tmp/jantman/foopython 5325 jantman rtd DIR 8,5 4096 2 /python 5325 jantman txt REG 0,30 6240 73547 /tmp/jantman/foo/bin/python2python 5325 jantman mem REG 0,30 33020 73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.sopython 5325 jantman mem REG 0,30 3032 89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman mem REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 0u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 1u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 2u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 3r FIFO 0,8 0t0 90605 pipepython 5325 jantman 4r REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 5r REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman 6r REG 0,30 42220 89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 7r REG 0,30 3032 89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx==============================> Creating repo object for testrepo3:=============================running lsof -p 5325 | grep -v /usr/lib:COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEpython 5325 jantman cwd DIR 0,30 200 73513 /tmp/jantman/foopython 5325 jantman rtd DIR 8,5 4096 2 /python 5325 jantman txt REG 0,30 6240 73547 /tmp/jantman/foo/bin/python2python 5325 jantman mem REG 0,30 33020 73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.sopython 5325 jantman mem REG 0,30 3032 89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman mem REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 0u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 1u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 2u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 3r FIFO 0,8 0t0 87612 pipepython 5325 jantman 4r REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 5r REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman 6r REG 0,30 42220 89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 7r REG 0,30 3032 89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx==============================> iterating commits=============================running lsof -p 5325 | grep -v /usr/lib:COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEpython 5325 jantman cwd DIR 0,30 200 73513 /tmp/jantman/foopython 5325 jantman rtd DIR 8,5 4096 2 /python 5325 jantman txt REG 0,30 6240 73547 /tmp/jantman/foo/bin/python2python 5325 jantman mem REG 0,30 33020 73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.sopython 5325 jantman mem REG 0,30 3032 79855 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 79854 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman mem REG 0,30 3032 89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman mem REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 0u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 1u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 2u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 3r FIFO 0,8 0t0 87616 pipepython 5325 jantman 4r REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 5r REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman 6r REG 0,30 42220 89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 7r REG 0,30 3032 89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman 8r REG 0,30 42220 79854 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 9r REG 0,30 3032 79855 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx==============================> del r=============================running lsof -p 5325 | grep -v /usr/lib:COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEpython 5325 jantman cwd DIR 0,30 200 73513 /tmp/jantman/foopython 5325 jantman rtd DIR 8,5 4096 2 /python 5325 jantman txt REG 0,30 6240 73547 /tmp/jantman/foo/bin/python2python 5325 jantman mem REG 0,30 33020 73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.sopython 5325 jantman mem REG 0,30 3032 79855 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 79854 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman mem REG 0,30 3032 89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman mem REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman mem REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 0u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 1u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 2u CHR 136,8 0t0 11 /dev/pts/8python 5325 jantman 3r FIFO 0,8 0t0 88654 pipepython 5325 jantman 4r REG 0,30 42220 83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 5r REG 0,30 3032 83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman 6r REG 0,30 42220 89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 7r REG 0,30 3032 89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idxpython 5325 jantman 8r REG 0,30 42220 79854 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.packpython 5325 jantman 9r REG 0,30 3032 79855 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx=============================The same thing happens if I explicitlyr = None. This is a real problem for me, as I'm trying to write an analysis script that looks at commits in about 450 repositories. On Linux with a default open file handle limit of 1024, the script ends up crashing asevery repository I examine results in open file handles for the pack and index files, that appear to never be cleaned up.
I feel like Ihave to be doing something wrong here, as I don't understand why these aren't cleaned up even after I'm done using the Repo object, and I set it toNone ordel it.