Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork938
Clarify Git.execute and Popen arguments#1688
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
- Reorder the items in the git.cmd.Git.execute docstring that document its parameters, to be in the same order the parameters are actually defined in.- Use consistent spacing, by having a blank line between successive items that document parameters. Before, most of them were separated this way, but some of them were not.- Reorder the elements of execute_kwargs (which list all those parameters except the first parameter, command) to be listed in that order as well. These were mostly in order, but a couple were out of order. This is just about the order they appear in the definition, since sets in Python (unlike dicts) have no key order guarantees.
The top line of the Git.execute docstring said that it used ashell, which is not necessarily the case (and is not usually thecase, since the default is not to use one). This removes thatclaim while keeping the top-line wording otherwise the same.It also rephrases the description of the command parameter, in away that does not change its meaning but reflects the more commonpractice of passing a sequence of arguments (since portable callsthat do not use a shell must do that).
These are some small clarity and consistency revisions to thedocstring of git.cmd.Git.execute that didn't specifically fit inthe narrow topics of either of the preceding two commits.
(Not specific to git.cmd.Git.execute.)
The kill_process local function defined in the Git.execute methodis a local function and not a method, but it was easy to misread asa method for two reasons:- Its docstring described it as a method.- It was named with a leading underscore, as though it were a nonpublic method. But this name is a local variable, and local variables are always nonpublic (except when they are function parameters, in which case they are in a sense public). A leading underscore in a local variable name usually means the variable is unused in the function.This fixes the docstring and drops the leading underscore from thename. If this is ever extracted from the Git.execute method andplaced at class or module scope, then the name can be changed back.
Instead of swallowing GitCommandError exceptions in the helper usedby test_it_uses_shell_or_not_as_specified andtest_it_logs_if_it_uses_a_shell, this modifies the helper so itprevents Git.execute from raising the exception in the first place.
This extracts the logic of searching log messages, and assertingthat (at least) one matches a pattern for the report of a Popencall with a given argument, from test_it_logs_if_it_uses_a_shellinto a new nonpublic test helper method _assert_logged_for_popen.The extracted version is modified to make it slightly more general,and slightly more robust. This is still not extremely robust: thenotation used to log Popen calls is informal, so it wouldn't makesense to really parse it as code. But this no longer assumes thatthe representation of a value ends at a word boundary, nor that thevalue is free of regular expression metacharacters.
This changes how the Popen call debug logging line shows theinformal summary of what kind of thing is being passed as the stdinargument to Popen, showing it with stdin= rather than istream=.The new test, with "istream" in place of "stdin", passed before thecode change in the git.cmd module, failed once "istream" waschanged to "stdin" in the test, and then, as expected, passed againonce "istream=" was changed to "stdin=" in the log.debug call ingit.cmd.Git.execute.
This is still not including all or even most of the arguments, norare all the logged arguments literal (nor should either of thosethings likely be changed). It is just to facilitate easiercomparison of what is logged to the Popen call in the code.
In Git.execute, the stdin argument to Popen is the only one where acompound expression (rather than a single term) is currentlypassed. So having that be the same in the log message makes iteasier to understand what is going on, as well as to see how theinformation shown in the log corresponds to what Popen receives.
def test_execute_kwargs_set_agrees_with_method(self): | ||
parameter_names = inspect.signature(cmd.Git.execute).parameters.keys() | ||
self_param, command_param, *most_params, extra_kwargs_param = parameter_names | ||
self.assertEqual(self_param, "self") | ||
self.assertEqual(command_param, "command") | ||
self.assertEqual(set(most_params), cmd.execute_kwargs) # Most important. | ||
self.assertEqual(extra_kwargs_param, "subprocess_kwargs") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Orgit.cmd.execute_kwargs
could be generated in a manner similar to this, and this test, the need to manually update it, and the note in theGit.execute
docstring about that need, could all be done away with. Maybe something like this:
execute_kwargs=inspect.signature(Git.execute).parameters.keys()- {"self","command","subprocess_kwargs"}
Right now it is defined very high up in the module, even higher than__all__
, and this suggests that it may be intended to be available for static inspection. But I don't think any tools will statically inspect aset of strings like that (plus, static analysis toolscan examine the parameters ofGit.execute
... though the incomplete lists of parameters in the@overload
-decorated stubs that precede it confuse the situation somewhat).
git.cmd.__all__
contains only"Git"
and I hope that means code that uses GitPython should not be usinggit.cmd.execute_kwargs
or relying on its presence. If possible, perhaps it could be made more explicitly private (_execute_kwargs
, or if it needs to remain statically defined, maybe_EXECUTE_KWARGS
) or, even better, removed altogether if theGit.execute
method's arguments can be inspected efficiently enough without it whereexecute_kwargs
is currently used. On the other hand, people may have come to depend on it even though the presence of__all__
that omits it means no oneshould have depended on it.
Anyway, I do not think any of the changes I suggest in this comment need to be made in this pull request. But I wanted to mention the issue just in case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Removingexecute_kwargs
seems like a reduction of complexity, which would always be a valuable reduction of maintenance costs should changes need to be made.
Asexecute_kwargs
was never advertised in__all__
I'd think that it's fair to say that those who depend on it nonetheless new the risk. I think the same argument is valid knowing that nothing is ever truly private, everything can be introspected if one truly wants to, yet it's something one simply has to ignore in order to be able to make any changes to python software once released.
"stdout_as_string", | ||
"output_stream", | ||
"with_stdout", | ||
"stdout_as_string", | ||
"kill_after_timeout", | ||
"with_stdout", |
EliahKaganOct 3, 2023 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This changes only the order in which they are given, not the contents, and the set is equal to the same value as before.
However, I am wondering ifcommand
should actually be added here. Although the@overload
-decorated stubs forGit.execute
define some parameters as keyword-only, in the actual definition no parameter is keyword-only and there is definitional symmetry betweencommand
and the others. Should I worry about what happens if someone has agit-command
script that they can usually run asgit command
and tries to rung.command()
whereg
is agit.cmd.Git
instance? Or related situations?
Another ramification of the parameters not being keyword-only is that, because they can be passed positionally, it is a breaking change to add a new one toGit.execute
elsewhere than the end. Still, regarding#1686 (comment), if you think astdin
parameter should be added as a synonym ofistream
, this could still be done even withstdin
at the end, with the docstring items that document the parameters referring to each other to overcome any confusion. I am inclined to say the added complexity is not worthwhile (for example, the function would have to handle the case where they are both passed and with inconsistent values). But a possible argument against this is that thetext
synonym ofuniversal_newlines
could also be added.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Should I worry about what happens if someone has a
git-command
script that they can usually run asgit command
and tries to rung.command()
whereg
is agit.cmd.Git
instance? Or related situations?
In these situations it would be great to know whycommand
was put there in the first place, and I for one do not remember. I know there never was an issue related tocommand
specifically, which seems to indicate it's a limitation worth ignoring or fixing in a non-breaking fashion (which seems possible).
Regardingistream
, I'd think maintaining stability would let me sleep better at night especially since you seem to agree that it's used consistently here and ingitdb
. Probably along with the documentation improvements, all is well and better than it was before, which even in the past didn't seem to have caused enough confusion to cause an issue to be opened.
EliahKaganOct 3, 2023 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
In these situations it would be great to know why
command
was put there in the first place, and I for one do not remember.
Do you mean why it wasomitted from theexecute_kwargs
set? If so, my guess is that it was intended to be used positionally while the others were intended to be keyword-only. This is judging by how they are given as keyword-only arguments in the@overload
s, where they are preceded by a*,
item in the list of formal parameters, which is absent in the actual function definition (thus causing them to accept both positional and keyword arguments).
But it may be that I am misunderstanding what you mean here. It is also possible that their keyword-only status in the@overload
s was intentionally different from the ultimate definition and intended to provide guidance. (I'm not sure. The@overload
s are missing most of the arguments, which confuses tooling and seems unintentional.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I am happy to adopt your conclusion in that this is not intentional and since there are negative side-effects, like tooling not working correctly, it's certainly something that could be improved.
If against all odds such an action does create downstream breakage, it could also be undone with a patch release.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I don't think we should make the parameters in the real definition actually keyword-only, since that is a breaking change. Or, at least, I would not rush to that. It seems likely to me that, at least for the first few, people are passing them positionally.
I believe it is merely the absence of many parameters from the@overload
-decorated stubs (which precede the real definition) that is causing VS Code not to show or autocomplete some arguments. Adding the missing parameters to the@overload
-decorated stubs should be sufficient to fix that, and it shouldn't be a breaking change because(a) they are just type stubs, so it doesn't break runtime behavior,(b) the actual change seems compatible even relative to what the type stubs said before, and(c) GitPython doesn't have static typing stability currently anyway, because althoughpy.typed
is included in the package,mypy
checks don't pass (nor, per#1659, dopyright
checks pass).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Thanks a lot for this second round of improvements!
It's at the core of GitPython and any improvements is most certainly valued downstream as long as it's not subtly breaking, which you made sure of.
With that said, I remember that this module is also used to start two long-running python processes lazily when objects are accessed, along with various calls togit
for doing pretty much anything else.
Also thanks to your work the idea of usinggitoxide
(or rather its still to be created python-bindings) has been forming in my head, which could lead togit
invocations to be reduced to zero while probably performing as well or better. Of course, anything like it is still years away, but it's something I look forward to.
def test_execute_kwargs_set_agrees_with_method(self): | ||
parameter_names = inspect.signature(cmd.Git.execute).parameters.keys() | ||
self_param, command_param, *most_params, extra_kwargs_param = parameter_names | ||
self.assertEqual(self_param, "self") | ||
self.assertEqual(command_param, "command") | ||
self.assertEqual(set(most_params), cmd.execute_kwargs) # Most important. | ||
self.assertEqual(extra_kwargs_param, "subprocess_kwargs") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Removingexecute_kwargs
seems like a reduction of complexity, which would always be a valuable reduction of maintenance costs should changes need to be made.
Asexecute_kwargs
was never advertised in__all__
I'd think that it's fair to say that those who depend on it nonetheless new the risk. I think the same argument is valid knowing that nothing is ever truly private, everything can be introspected if one truly wants to, yet it's something one simply has to ignore in order to be able to make any changes to python software once released.
[](https://renovatebot.com)This PR contains the following updates:| Package | Change | Age | Adoption | Passing | Confidence ||---|---|---|---|---|---|| [GitPython](https://togithub.com/gitpython-developers/GitPython) |`==3.1.37` -> `==3.1.40` |[](https://docs.renovatebot.com/merge-confidence/)|[](https://docs.renovatebot.com/merge-confidence/)|[](https://docs.renovatebot.com/merge-confidence/)|[](https://docs.renovatebot.com/merge-confidence/)|---### Release Notes<details><summary>gitpython-developers/GitPython (GitPython)</summary>###[`v3.1.40`](https://togithub.com/gitpython-developers/GitPython/compare/3.1.38...3.1.40)[CompareSource](https://togithub.com/gitpython-developers/GitPython/compare/3.1.38...3.1.40)###[`v3.1.38`](https://togithub.com/gitpython-developers/GitPython/releases/tag/3.1.38)[CompareSource](https://togithub.com/gitpython-developers/GitPython/compare/3.1.37...3.1.38)#### What's Changed- Add missing assert keywords by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1678](https://togithub.com/gitpython-developers/GitPython/pull/1678)- Make clear every test's status in every CI run by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1679](https://togithub.com/gitpython-developers/GitPython/pull/1679)- Fix new link to license in readme by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1680](https://togithub.com/gitpython-developers/GitPython/pull/1680)- Drop unneeded flake8 suppressions by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1681](https://togithub.com/gitpython-developers/GitPython/pull/1681)- Update instructions and test helpers for git-daemon by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1684](https://togithub.com/gitpython-developers/GitPython/pull/1684)- Fix Git.execute shell use and reporting bugs by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1687](https://togithub.com/gitpython-developers/GitPython/pull/1687)- No longer allow CI to select a prerelease for 3.12 by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1689](https://togithub.com/gitpython-developers/GitPython/pull/1689)- Clarify Git.execute and Popen arguments by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1688](https://togithub.com/gitpython-developers/GitPython/pull/1688)- Ask git where its daemon is and use that by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1697](https://togithub.com/gitpython-developers/GitPython/pull/1697)- Fix bugs affecting exception wrapping in rmtree callback by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1700](https://togithub.com/gitpython-developers/GitPython/pull/1700)- Fix dynamically-set **all** variable by[@​DeflateAwning](https://togithub.com/DeflateAwning) in[https://github.com/gitpython-developers/GitPython/pull/1659](https://togithub.com/gitpython-developers/GitPython/pull/1659)- Fix small[#​1662](https://togithub.com/gitpython-developers/GitPython/issues/1662)regression due to[#​1659](https://togithub.com/gitpython-developers/GitPython/issues/1659)by [@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1701](https://togithub.com/gitpython-developers/GitPython/pull/1701)- Drop obsolete info on yanking from security policy by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1703](https://togithub.com/gitpython-developers/GitPython/pull/1703)- Have Dependabot offer submodule updates by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1702](https://togithub.com/gitpython-developers/GitPython/pull/1702)- Bump git/ext/gitdb from `49c3178` to `8ec2390` by[@​dependabot](https://togithub.com/dependabot) in[https://github.com/gitpython-developers/GitPython/pull/1704](https://togithub.com/gitpython-developers/GitPython/pull/1704)- Bump git/ext/gitdb from `8ec2390` to `6a22706` by[@​dependabot](https://togithub.com/dependabot) in[https://github.com/gitpython-developers/GitPython/pull/1705](https://togithub.com/gitpython-developers/GitPython/pull/1705)- Update readme for milestone-less releasing by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1707](https://togithub.com/gitpython-developers/GitPython/pull/1707)- Run Cygwin CI workflow commands in login shells by[@​EliahKagan](https://togithub.com/EliahKagan) in[https://github.com/gitpython-developers/GitPython/pull/1709](https://togithub.com/gitpython-developers/GitPython/pull/1709)#### New Contributors- [@​DeflateAwning](https://togithub.com/DeflateAwning) made theirfirst contribution in[https://github.com/gitpython-developers/GitPython/pull/1659](https://togithub.com/gitpython-developers/GitPython/pull/1659)**Full Changelog**:gitpython-developers/GitPython@3.1.37...3.1.38</details>---### Configuration📅 **Schedule**: Branch creation - At any time (no schedule defined),Automerge - At any time (no schedule defined).🚦 **Automerge**: Enabled.♻ **Rebasing**: Whenever PR becomes conflicted, or you tick therebase/retry checkbox.🔕 **Ignore**: Close this PR and you won't be reminded about this updateagain.---- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, checkthis box---This PR has been generated by [MendRenovate](https://www.mend.io/free-developer-tools/renovate/). Viewrepository job log[here](https://developer.mend.io/github/allenporter/flux-local).<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xOS4yIiwidXBkYXRlZEluVmVyIjoiMzcuMTkuMiIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Uh oh!
There was an error while loading.Please reload this page.
This is a sequel to#1687, improving how parameters to
Git.execute
are documented, and improving debug logging and tests of how they affect the arguments passed toPopen
.In the
Git.execute
docstring, the method is no longer described as using a shell, since this is typically (and by default) not the case; the items documenting each parameter are listed in the order of the parameters; and some copyediting is done, for clarity, consistency, spacing, and in a few cases other formatting. I copyedited some other docstrings in the module as well.The order of names in the
execute_kwargs
set is tweaked to also match the order in which they appear as parameters toGit.execute
. (This is just for the purpose of code clarity, asset
objects guarantee no particular iteration order.) Since not all unintentional mismatches between this set and the defined method parameters would cause existing tests to fail, and the failures that would occur would not always immediately show the cause of the problem, I also added a test thatexecute_kwargs
has the exact expected relationship to the parameters ofGit.execute
. (Though an alternative might be to generateexecute_kwargs
programmatically fromGit.execute
using a similar technique.)One part of the test logic in#1687 was unnecessarily complicated, due to swallowing an exception produced by running
git
with no arguments. This changes that by passingwith_exceptions=False
so that exception is never generated.Another part of the the test logic in#1687 combined claims about the code under test with custom assertion logic in a way that made it hard to see what claims were being made by reading the test code. This fixes that by generalizing, and extracting out, an
_assert_logged_for_popen
test helper method. I also took this opportunity to remove its unstated incompatibility with value representations containing regular expression metacharacters. (Please note that the new code is still only robust enough for the special purpose for which it is intended; it does not actually parse the debug message rigorously.)As requested in#1686 (comment), I changed
istream=
tostdin=
in the debug logging message documenting thePopen
call. I reused the test helper in writing a new test,test_it_logs_istream_summary_for_stdin
,which checks both that it has the new name and that it has the expected simplified value representation. I did not change any parameters or variables called
istream
tostdin
or any other name; rather, this changes the message to better reflect thePopen
call it exists to document.If I would only have changed the displayed parameter name, I would likely not have written a test, but I also wanted to change two other things, in which the test helped verify that correctness was maintained: I reordered the displayed
name=value
representations in the log message to match the relative order in which they are passed toPopen
. And I eliminated theistream_ok
variable (which was named like a boolean flag but was not one), because having the logic for producing the string in the logging call makes it better resemble the nonidentical but corresponding logic in thePopen
call, so they can be compared.Although this is peripheral to the core concept of the clarity of function arguments, I also renamed and fixed the docstring of a local function that appeared (including by claiming) to be a method.