Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork938
Keep temp files out of project dir and improve cleanup#1825
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Uh oh!
There was an error while loading.Please reload this page.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
This fixes a test bug in TestTree.test_tree_modifier_orderingwhere:- A `tmp` subdirectory of the directory the tests are run from (almost always the root of the GitPython source tree) was created, instead of creating it in /tmp or elsewhere configured.- That directory was never deleted afterward, creating new untracked files in GitPython's own working tree, and also making it so running the tests twice would always fail the second time, unless a manual intervening deletion was performed. (This had not broken CI, since CI checks clone the repository anew each time.)- The directory was changed into to set up the test, but not deterministically changed back out of. It would typically be exited, but this was not guaranteed to occur if some subprocess commands were to fail unexpectedly.In addition to fixing all three aspects of that bug, this also:- Remains in the temporary directory only as long as necessary to execute the subprocesses in it that produce the expected value for the test (including not entering it until running them).- Deletes the temporary directory immediately after exiting it, since it is only used to produce a file list to compare to.- Avoids interleaving file creation and git subprocess commands, since doing so made it harder to understand what was being set up and since the difference is not relevant to the test.- Does the work in that temporary directory before performing any operations with the code under test, because it is conceptually an "arrange" step, and because doing so makes its limited purpose clearer on reading the tests.- Some other refactoring to support the fixes and accompanying changes, including extracting the temporary directory logic to a helper method.This deliberately does not change the order in which any files arecreated. It also keeps the approach of using subprocess functionsdirectly to operate on the temporary git repository (and changingdirectory while doing so, keeping each of the commands the same),since there might be good reasons to do this, such as to make veryclear that the file order being obtained to compare to is reallycoming from git itself. Even if this is to be changed, it seemsoutside the scope of this bugfix.The test still fails if the fix to git/objects/tree.py from365d44f(gitpython-developers#1799) is absent. This was verified by running: git revert --no-commit365d44f pytest --no-cov -vv test/test_tree.py git revert --abort
In Python 3.7, tempfile.TemporaryDirectory doesn't delete files oncleanup whose permissions have to be changed to be deleted.This uses `@with_rw_directory` instead, though that may be overkillhere.
This changes from `@with_rw_directory` back to TemporaryDirectory,but calls git.util.rmtree on the repsitory's .git directory wheresome read-only files otherwise cause TemporaryDirectory's cleanupto raise PermissionError on Windows in Python 3.7.This avoids the gc.collect that is known not to be necessary inthis speciifc situation, as well as the problem that, if operatingin the temporary directory did fail, then name of the helper wouldbe logged as the name of the test where the failure occurred. Butthis has the disadvantage of making the helper more complex andharder to understand. So this may not be the best approach either.
The situation with test_tree_modifier_ordering's helper, where itmakes sense to wrap the helper itself with `@with_rw_directory`while still deleting the temporary directory as soon as the helperitself finishes, in order to make clear that the test itself doesnot use the temporary directory that the helper used, only occursin one place in GitPython's tests as far as I know.In particular, all other places where `@with_rw_directory` wasactually in use are test methods, not helper methods, and the waythe decorator would have its wrapper log on failure documented thedecorated method as a test. Mainly for that reason, I had backedaway from using `@with_rw_directory` here.But the test code seems much easier to understand when it is usedcompared to other approaches. While it also has the disadvantage ofdoing a gc.collect that is unnecessary here, this is not the onlyplace what that is done.This commit brings back the test_tree_modifier_ordering helperimplementation that uses `@with_rw_directory`, while also modifyingthe logging logic in `@with_rw_directory` slightly, so that whenthe decorated method is not named as a test, it is referred to as a"Helper" rather than a "Test" in the failure log message.
Thanks a lot for your help, and apologies for not catching this. Detecting that the temporary directory isn't cleaned up should have been possible. But like you said, maybe it was done that way in the first place because cleaning up a |
lettuce-botbot referenced this pull request in lettuce-financial/github-bot-signed-commitFeb 15, 2024
[](https://renovatebot.com)This PR contains the following updates:| Package | Change | Age | Adoption | Passing | Confidence ||---|---|---|---|---|---|| [GitPython](https://togithub.com/gitpython-developers/GitPython) |`==3.1.41` -> `==3.1.42` |[](https://docs.renovatebot.com/merge-confidence/)|[](https://docs.renovatebot.com/merge-confidence/)|[](https://docs.renovatebot.com/merge-confidence/)|[](https://docs.renovatebot.com/merge-confidence/)|---### Release Notes<details><summary>gitpython-developers/GitPython (GitPython)</summary>###[`v3.1.42`](https://togithub.com/gitpython-developers/GitPython/releases/tag/3.1.42)[CompareSource](https://togithub.com/gitpython-developers/GitPython/compare/3.1.41...3.1.42)#### What's Changed- Fix release link in changelog by[@​PeterJCLaw](https://togithub.com/PeterJCLaw) in[https://github.com/gitpython-developers/GitPython/pull/1795](https://togithub.com/gitpython-developers/GitPython/pull/1795)- Remove test dependency on sumtypes library by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1798](https://togithub.com/gitpython-developers/GitPython/pull/1798)- Pin Sphinx plugins to compatible versions by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1803](https://togithub.com/gitpython-developers/GitPython/pull/1803)- fix: treeNotSorted issue by[@​et-repositories](https://togithub.com/et-repositories) in[https://github.com/gitpython-developers/GitPython/pull/1799](https://togithub.com/gitpython-developers/GitPython/pull/1799)- Remove git.util.NullHandler by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1807](https://togithub.com/gitpython-developers/GitPython/pull/1807)- Clarify why GIT_PYTHON_GIT_EXECUTABLE may be set on failure by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1810](https://togithub.com/gitpython-developers/GitPython/pull/1810)- Report actual attempted Git command when Git.refresh fails by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1812](https://togithub.com/gitpython-developers/GitPython/pull/1812)- Don't suppress messages when logging is not configured by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1813](https://togithub.com/gitpython-developers/GitPython/pull/1813)- Pin Python 3.9.16 on Cygwin CI by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1814](https://togithub.com/gitpython-developers/GitPython/pull/1814)- Have initial refresh use a logger to warn by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1815](https://togithub.com/gitpython-developers/GitPython/pull/1815)- Omit warning prefix in "Bad git executable" message by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1816](https://togithub.com/gitpython-developers/GitPython/pull/1816)- Test with M1 macOS CI runner by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1817](https://togithub.com/gitpython-developers/GitPython/pull/1817)- Bump pre-commit/action from 3.0.0 to 3.0.1 by[@​dependabot](https://togithub.com/dependabot) in[https://github.com/gitpython-developers/GitPython/pull/1818](https://togithub.com/gitpython-developers/GitPython/pull/1818)- Bump Vampire/setup-wsl from 2.0.2 to 3.0.0 by[@​dependabot](https://togithub.com/dependabot) in[https://github.com/gitpython-developers/GitPython/pull/1819](https://togithub.com/gitpython-developers/GitPython/pull/1819)- Remove deprecated section in README.md by[@​marcm-ml](https://togithub.com/marcm-ml) in[https://github.com/gitpython-developers/GitPython/pull/1823](https://togithub.com/gitpython-developers/GitPython/pull/1823)- Keep temp files out of project dir and improve cleanup by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1825](https://togithub.com/gitpython-developers/GitPython/pull/1825)#### New Contributors- [@​PeterJCLaw](https://togithub.com/PeterJCLaw) made their firstcontribution in[https://github.com/gitpython-developers/GitPython/pull/1795](https://togithub.com/gitpython-developers/GitPython/pull/1795)- [@​et-repositories](https://togithub.com/et-repositories) madetheir first contribution in[https://github.com/gitpython-developers/GitPython/pull/1799](https://togithub.com/gitpython-developers/GitPython/pull/1799)- [@​marcm-ml](https://togithub.com/marcm-ml) made their firstcontribution in[https://github.com/gitpython-developers/GitPython/pull/1823](https://togithub.com/gitpython-developers/GitPython/pull/1823)**Full Changelog**:gitpython-developers/GitPython@3.1.41...3.1.42</details>---### Configuration📅 **Schedule**: Branch creation - At any time (no schedule defined),Automerge - At any time (no schedule defined).🚦 **Automerge**: Disabled by config. Please merge this manually once youare satisfied.♻ **Rebasing**: Whenever PR becomes conflicted, or you tick therebase/retry checkbox.🔕 **Ignore**: Close this PR and you won't be reminded about theseupdates again.---- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, checkthis box---This PR has been generated by [MendRenovate](https://www.mend.io/free-developer-tools/renovate/). Viewrepository job log[here](https://developer.mend.io/github/lettuce-financial/github-bot-signed-commit).<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xNzMuMCIsInVwZGF0ZWRJblZlciI6IjM3LjE3My4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->
renovatebot referenced this pull request in allenporter/flux-localFeb 16, 2024
[](https://renovatebot.com)This PR contains the following updates:| Package | Change | Age | Adoption | Passing | Confidence ||---|---|---|---|---|---|| [GitPython](https://togithub.com/gitpython-developers/GitPython) |`==3.1.41` -> `==3.1.42` |[](https://docs.renovatebot.com/merge-confidence/)|[](https://docs.renovatebot.com/merge-confidence/)|[](https://docs.renovatebot.com/merge-confidence/)|[](https://docs.renovatebot.com/merge-confidence/)|---### Release Notes<details><summary>gitpython-developers/GitPython (GitPython)</summary>###[`v3.1.42`](https://togithub.com/gitpython-developers/GitPython/releases/tag/3.1.42)[CompareSource](https://togithub.com/gitpython-developers/GitPython/compare/3.1.41...3.1.42)#### What's Changed- Fix release link in changelog by[@​PeterJCLaw](https://togithub.com/PeterJCLaw) in[https://github.com/gitpython-developers/GitPython/pull/1795](https://togithub.com/gitpython-developers/GitPython/pull/1795)- Remove test dependency on sumtypes library by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1798](https://togithub.com/gitpython-developers/GitPython/pull/1798)- Pin Sphinx plugins to compatible versions by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1803](https://togithub.com/gitpython-developers/GitPython/pull/1803)- fix: treeNotSorted issue by[@​et-repositories](https://togithub.com/et-repositories) in[https://github.com/gitpython-developers/GitPython/pull/1799](https://togithub.com/gitpython-developers/GitPython/pull/1799)- Remove git.util.NullHandler by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1807](https://togithub.com/gitpython-developers/GitPython/pull/1807)- Clarify why GIT_PYTHON_GIT_EXECUTABLE may be set on failure by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1810](https://togithub.com/gitpython-developers/GitPython/pull/1810)- Report actual attempted Git command when Git.refresh fails by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1812](https://togithub.com/gitpython-developers/GitPython/pull/1812)- Don't suppress messages when logging is not configured by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1813](https://togithub.com/gitpython-developers/GitPython/pull/1813)- Pin Python 3.9.16 on Cygwin CI by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1814](https://togithub.com/gitpython-developers/GitPython/pull/1814)- Have initial refresh use a logger to warn by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1815](https://togithub.com/gitpython-developers/GitPython/pull/1815)- Omit warning prefix in "Bad git executable" message by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1816](https://togithub.com/gitpython-developers/GitPython/pull/1816)- Test with M1 macOS CI runner by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1817](https://togithub.com/gitpython-developers/GitPython/pull/1817)- Bump pre-commit/action from 3.0.0 to 3.0.1 by[@​dependabot](https://togithub.com/dependabot) in[https://github.com/gitpython-developers/GitPython/pull/1818](https://togithub.com/gitpython-developers/GitPython/pull/1818)- Bump Vampire/setup-wsl from 2.0.2 to 3.0.0 by[@​dependabot](https://togithub.com/dependabot) in[https://github.com/gitpython-developers/GitPython/pull/1819](https://togithub.com/gitpython-developers/GitPython/pull/1819)- Remove deprecated section in README.md by[@​marcm-ml](https://togithub.com/marcm-ml) in[https://github.com/gitpython-developers/GitPython/pull/1823](https://togithub.com/gitpython-developers/GitPython/pull/1823)- Keep temp files out of project dir and improve cleanup by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1825](https://togithub.com/gitpython-developers/GitPython/pull/1825)#### New Contributors- [@​PeterJCLaw](https://togithub.com/PeterJCLaw) made their firstcontribution in[https://github.com/gitpython-developers/GitPython/pull/1795](https://togithub.com/gitpython-developers/GitPython/pull/1795)- [@​et-repositories](https://togithub.com/et-repositories) madetheir first contribution in[https://github.com/gitpython-developers/GitPython/pull/1799](https://togithub.com/gitpython-developers/GitPython/pull/1799)- [@​marcm-ml](https://togithub.com/marcm-ml) made their firstcontribution in[https://github.com/gitpython-developers/GitPython/pull/1823](https://togithub.com/gitpython-developers/GitPython/pull/1823)**Full Changelog**:gitpython-developers/GitPython@3.1.41...3.1.42</details>---### Configuration📅 **Schedule**: Branch creation - At any time (no schedule defined),Automerge - At any time (no schedule defined).🚦 **Automerge**: Enabled.♻ **Rebasing**: Whenever PR becomes conflicted, or you tick therebase/retry checkbox.🔕 **Ignore**: Close this PR and you won't be reminded about this updateagain.---- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, checkthis box---This PR has been generated by [MendRenovate](https://www.mend.io/free-developer-tools/renovate/). Viewrepository job log[here](https://developer.mend.io/github/allenporter/flux-local).<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xNzMuMCIsInVwZGF0ZWRJblZlciI6IjM3LjE3My4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading.Please reload this page.
Fixes#1824
This fixes temporary directory creation and cleanup in
test_tree_modifier_ordering
, extracts its helper to a separate method, slightly extends@with_rw_directory
so it can be used directly on helper methods while still logging accurate descriptions, and uses it to ensure cleanup of the temporary directory that is used by the helper to generate the expected value that the test will compare its results to.Because that decorator was not previously used on any test helper methods, only on test methods, it is not obvious that my use of it on the helper, and my modification to it to properly support such use, is justified. The reason I think it is justified is that we really do want to complete cleanup, including deleting the temporary directory, before beginning to use the code under test, in order to make clear (when reading the code or when debugging tests or inspecting output) that the helper really is just arranging an expected value for the test.
There's a little more information in the commit messages, but it's mostly covered by the details in#1824 and this PR description.
One important question is whether the new test really still works as a regression test. The answer is yes, as shown by a temporary non-committed revert of365d44f (the fix in#1799). This is from before the change in this PR:
And this is from after the changes in this PR:
Note that they fail the same way, as desired, when the original bug is temporarily brought back--in particular, see the file list order diff under "Full diff" in each run--and pass when the bugfix is applied.