Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Improve scripts and tool configurations#1693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
Byron merged 32 commits intogitpython-developers:mainfromEliahKagan:sh
Oct 4, 2023

Conversation

EliahKagan
Copy link
Member

@EliahKaganEliahKagan commentedOct 4, 2023
edited
Loading

Fixes#1690
Fixes#1691
Fixes#1692

This combines the solutions I recommended in those three issues on a single branch. Although I feel this does compose into a coherent whole, with each of those issues' practical ramifications having some tendrils into each other, and opening separate PRs would have resulted in nontrivial merge conflicts... nonetheless this PR is broader than I would usually like. But I have done it this way because it was by working on them together that I was able to get clear on the distinct ideas that I presented as those issues, as well as whether the solutions would work reasonably when all combined.

This PR is what remains after pieces that seemed reasonable to sever into their on PRs, such as#1684, have been removed, and excessively enterprising ideas, such as writing a batch-file version ofinit-tests-after-clone.sh (discussed in#1691) or switchinglint.yml to usepre-commit.ci, abandoned or deferred. However, I would be pleased to make further changes as requested. I think the changes here are feasible to review together, but I am willing to rework this into multiple more narrowly scoped PRs if necessary.

The problems and their solutions in general are presented in the three linked issues that this would close. (Details are given in commit messages.)

Specific considerations that I suspect may be important when reviewing this PR follow.

Shell script portability

I believe the shell scripts that pertain to building do not need to be any more portable than they are now: they run onbash version 3 or later, which most systems have or can get. I did make modifications to them, some of which were inspired and emboldened byshellcheck, but not for portability to other shells.

I take#1661 (comment) (specifically:d18d90a,1e0b3f9) as an indication of a general preference ofecho overprintf. Thebash shell provides anecho builtin that has good design decisions and, more importantly, behaves the same inbash on any system, because it is a builtin. (I have actually slightly increased the use ofecho inbash in this PR.)

However, for POSIX shell scripts that assume only what the standard insists on (or that callecho through another command, such as withfind -exec orxargs, thereby using an externalecho executable), the situation isconsiderably messier. To write a portable script, I have avoided the use ofecho in the init script that, for portability, I have changed frombash tosh.

What fallback version-tag fetching looks like

Fallback fetching of version tags is the "back-up" strategy that is part of the fix for#1690. It works both locally and on CI. Here's what it looks like on CI (I temporarily deleted all the version tags from my fork and reran the tip of this PR's branch):

(In case for any reason you want to see the version from an earlier commit and prior to multiple rebases that I had shown before in an earlier draft of this PR description,that's here.)

Fallback version-tag fetching would fail on oldgit

At least as currently written, it uses a feature that was introduced in Git 2.6.0.

I haveposted a review comment on the specific code this affects, with full details.

Security implications of whereinit-tests-after-clone.sh makesmaster

Edit: Based on#1615, I think you might prefergit checkout -B master not be used, for reasons of stability rather than security. Maybegit checkout -B master ea43defd777a9c0751fc44a9c6a622fc2dbd18a0 should even be used, to get the same old commit even in forks that don't have themaster branch, and to add the same commits to the top of the reflog history even when__testing_point__ is deleted and the script rerun.

Currently, onmain,init-tests-after-clone.sh creates or switches to amaster branch like this:

git checkout master|| git checkout -b master

One of the changes in this PR is to ensure thatmaster cannot be interpreted as a path--it has to be taken to be a branch, or at least a ref--which is mainly for better output to stderr when the branch doesn't exist, but also to safeguard against rare situations where a file or directory namedmaster exists:

git checkout master --|| git checkout -b master

However, there is a simpler, more elegant, and more robust way to get a suitablemaster branch for tests, that I think is more likely to work well most of the time[edit: though the answer to#1615 suggests a reason it might not]:

git checkout -B master

master is already getting reset back to__testing_point__, so the main disadvantage of-B, that one can lose one's branch if it is not pushed, applies already.

I would like to do it that way (and I havea branch for it, ready to go). But it has security implications,specifically for people who review pull requests. As detailed in#1690, wherevermaster gets checked out, editors and IDEs may be fooled into running unreviewed untrusted code from there. Using the-B way I want to use would be a mitigation for the situation described in#1690 (though that's not the main reason I want to do it). But it would be anexacerbation of it for a reviewer, because it would weaken the already brittle assurancegit diff main feature would provide. Consider:

main  ←  evil  ←  evil  ←  evil  ←  feature

There, a pull request proposes a useful feature on thefeature branch. But the preceding three commits have evil code in them that, if an editor/IDE imports modules to do test discovery (or any other operation the editor reserves for "trusted folders"), will be run. The tip offeature removes the evil code, sogit diff main feature does not reveal it. But becausemaster is checked out at the tip offeature even ifmaster already exists in the reviewer's local or remote repository,the script's reflog-building commands reset to each of the evil commits.

This may not be a serious problem. When reviewing, one should already not rely merely ongit diff main feature, and CI in pull requests is set up to run on thepull-request trigger (which, unlikepull-request-target which is dangerous if not used very carefully, runs with the fork's permissions, not allowing elevation). Furthermore, one may already not havemaster locally or on a remote, in which case the existing checkout code already creates it atHEAD, so it's not like this is a new situation. Furthermore, it's generally unnecessary to run that script while reviewing, evenif one opens up an only partially reviewed PR branch in an editor that may perform unsafe operations based on its contents.

Nonetheless, I wanted to bring this up rather than just adding a commit to change it togit checkout -B master.

black viapre-commit: I'm not sure what's best.

It seems to me that there is actually just one thing that, in hindsight, would have been better to have developed separately. Addingblack topre-commit, and using that on CI, presents choices to be made while reviewing that are practically separate from the other changes.

The core issue is that all the other tools run viapre-commit only perform checks, while the way most users ofblack andpre-commit would expect and prefer they be used together is forpre-commit to actually perform auto-formatting. This is actually no problem on CI; the GitHub Action being used accepts code changes from a hook as a check failure. But there should be a way tojust lint locally, that includes checking forblack-conforming formatting and that does not change any files.

Therefore, I have set uptwo hooks forblack: one that runs by default and formats, and another that does not run by default and that only performs checks. Thetox linting environment andlint.yml CI workflow use the check-only hook and skip the formatting one. But there should also be a way to do this locallywithout building the project and setting uptox environments and without typing in a complexpre-commit command. So I have mademake lint do that.

This is all documented as part of the readme improvements. It is inelegant, though, because everything else done by makefiles in this project is directly related tobuilding. There are a few options, which I present in descending order of my preference, but they are all things I think are reasonable:

  • Use this, for now.
  • Use this, for now, but add a note in the readme about howmake lint may go away. (I think this is not really necessary, because thedevelopment instructions are not implied to be stable. But I am not sure.)
  • Changepre-commit to haveonly the check-onlyblack hook, at least for now, and modify the readme accordingly. Then there is no need formake lint becausepre-commit run --all-files just lints, as it did before, but includingblack. This has the disadvantage that people who likeblack andpre-commit probably prefer that they facilitate auto-formatting, but it is otherwise elegant. I suspect you might prefer this approach, therefore I have made a commit for it on mysh-nofmt branch, which can be easily fast-forward merged into here (or opened as a separate PR if the change is desired after merging this).
  • Remove everything to do withblack from.pre-commit-config.yml on this branch, modifying the readme accordingly, and propose it separately. It could be removed by rebasing, or just by adding a commit to remove it. This is more work, but actually a bigger reason this is not my preferred approach is the same reason I had addedblack topre-commit in the first place: with everything else being done bypre-commit, and with this project being inblack style (aside from this project's very long lines), it is unintuitive and unexpected that it would not, in any form, be usable viapre-commit.

This also reorders the hooks from pre-commit/pre-commit-hooks sothat the overall order of all hooks from all repositories is: lintPython, lint non-Python, non-lint.
Its output is colorized normally, but on CI it is not colorizedwithout this (pre-commit's own output is, but not shellcheck's).
This suppresses ShellCheck SC2016, "Double quote to preventglobbing and word splitting," on the command in the version checkscript that expands $config_opts to build the "-c ..." arguments.It also moves the code repsonsible for getting the latest tag,which this is part of, into a function for that purpose, so it'sclear that building config_opts is specifically for that, and sothat the code is not made harder to read by adding the ShellChecksuppression comment.(The suppression applies only to the immediate next command.)
In the release building script, this changes $1 to "$1", because$1 without quotes means to perform word splitting and globbing(pathname expansion) on the parameter (unless otherwise disabled bythe value of $IFS or "set -f", respectively) and use the result ofdoing so, which isn't the intent of the code.This function is only used from within the script, where it isnot given values that would be changed by these additionalexpansions. So this is mainly about communicating intent.(If in the future it is intended that multiple arguments be usable,then they should be passed as separate arguments to release_with,which should be modified by replacing "$1" with "$@". I have notused "$@" at this time because it is not intuitively obvious thatthe arguments should be to the interpreter, rather than to thebuild module, so I don't think this should be supported unless oruntil a need for it determines that.)
I think this is easier to read, but this is admittedly subjective.This commit also makes the separate change of adjusting commentspacing for consistency within the script. (Two spaces before "#"is not widely regarded as better than one in shell scripting, sounlike Python where PEP-8 recommends that, it would be equally goodto have changed all the other places in the shell scrips to havejust one.)
The script is nonportable to other shells already because of itsuse of trap (and other features, such as implicit assumptions madeabout echo, and the function keyword), which its hashbang alreadyclearly conveys.This uses:- $(<X) in place of $(cat X), to have the shell read the file  directly rather than using cat.- printf -v in one place to set a variable rather than printing.  This eliminates a command substitution that was harder to read.
Because users may have an old version of git without "git switch",init-tests-after-clone.sh should continue to use "git checkout" toattempt to switch to master. But without "--", this suffers fromthe problem that it's ambiguous if master is a branch (so checkoutbehaves like switch) or a path (so checkout behaves like restore).There are two cases where this ambiguity can be a problem. The mostcommon is on a fork with no master branch but also, fortunately, nofile or directory named "master". Then the problem is just theerror message (printed just before the script proceeds to redothe checkout with -b):    error: pathspec 'master' did not match any file(s) known to gitThe real cause of the error is the branch being absent, as happenswhen a fork copies only the main branch and the upstream remote isnot also set up. Adding the "--" improves the error message:    fatal: invalid reference: masterHowever, it is possible, though unlikely, for a file or directorycalled "master" to exist. In that case, if there is also no masterbranch, git discards unstaged changes made to the file or (muchworse!) everywhere in that directory, potentially losing work.This commit adds "--" to the right of "master" so git neverregards it as a path. This is not needed with -b, which is alwaysfollowed by a symbolic name, so I have not added it there.(Note that the command is still imperfect because, for example, inrare cases there could be a master *tag*--and no master branch--inwhich case, as before, HEAD would be detached there and the scriptwould attempt to continue.)
This had been done everywhere except in init-tests-after-clone.sh.
This translates init-tests-after-clone.sh from bash to POSIX sh,changing the hashbang and replacing all bash-specific features, sothat it can be run on any POSIX-compatible ("Bourne style") shell,including but not limited to bash. This allows it to be used evenon systems that don't have any version of bash installed anywhere.Only that script is modified. The other shell scripts make greateruse of (and gain greater benefit from) bash features, and are alsoonly used for building and releasing. In contrast, this script ismeant to be used by most users who clone the repository.
This makes three related changes:- Removes "git fetch --tags" from the instructions in the readme,  because the goal of this command can be achieved in the  init-tests-after-clone.sh script, and because this fetch command,  as written, is probably only achieving that goal in narrow cases.  In clones and fetches, tags on branches are fetched by default,  and the tags needed by the tests are on branches. So the  situations where "git fetch --tags" was helping were (a) when the  remote recently gained the tags, and (b) when the original remote  was cloned in an unusual way, not fetching all tags. In both  cases, the "--tags" option is not what makes that fetch get the  needed tags.- Adds "git fetch --all --tags" to init-tests-after-clone.sh. The  "--all" option causes it to fetch from all remotes, and this is  more significant than "--tags", since the tags needed for testing  are on fetched branches. This achieves what "git fetch --tags"  was achieving, and it also has the benefit of getting tags from  remotes that have been added but not fetched from, as happens  with an upstream remote that was manually added with no further  action. (It also gets branches from those remotes, but if master  is on multiple remotes but at different commits then "git  checkout master" may not be able to pick one. So do this *after*  rather than before that.)- Skips this extra fetch, and also the submodule cloning/updating  step, when running on CI. CI jobs will already have taken care of  cloning the repo with submodules recursively, and fetching all  available tags. In forks without tags, the necessary tags for the  test are not fetched--but the upstream remote is not set up on  CI, so they wouldn't be obtained anyway, not even by refetching  with "--all".
This extracts duplicated code to functions in check-version.sh.One effect is unfortunately that the specific commands being runare less explicitly clear when reading the script. However, smallfuture changes, if accidentally made to one but not the other ineither pair of "git status" commands, would create a much moreconfusing situation. So I think this change is justified overall.
- Because the substitition string after the hyphen is empty,  "${VIRTUAL_ENV:-}" and "${VIRTUAL_ENV-}" have the same effect.  However, the latter, which this changes it to, expresses the  correct idea that the special case being handled is when the  variable is unset: in this case, we expand an empty field rather  than triggering an error due to set -u. When the variable is set  but empty, it already expands to the substitution value, and  including that in the special case with ":" is thus misleading.- Continuing in the vein ofd18d90a (and1e0b3f9), this removes  another explicit newline by adding another echo command to print  the leading blank line before the table.
This adds comments to init-tests-after-clone.sh to explain whateach of the steps is doing that is needed by some of the tests.It also refactors some recently added logic, in a way that lendsitself to greater clarity when combined with these comments.
This is already done in the other shell scripts. Althoughinit-tests-after-clone.sh does not have as many places where a bugcould slip through by an inadvertently nonexistent parameter, itdoes have $answer (and it may have more expansions in the future).
This is a minor improvement to the robustness of the command for"make all", in the face of plausible future target names.- Use [[:alpha:]] instead of [a-z] because, in POSIX BRE and ERE,  whether [a-z] includes capital letters depends on the current  collation order. (The goal here is greater consistency across  systems, and for this it would also be okay to use [[:lower:]].)- Pass -x to the command that filters out the all target itself,  so that the entire name must be "all", because a future target  may contain the substring "all" as part of a larger word.
This seems simpler to me, but I admit it is subjective.
- Add the psf/black-pre-commit-mirror pre-commit hook but have it  just check and report violations rather than fixing them  automatically (to avoid inconsistency with all the other hooks,  and also so that the linting instructions and CI lint workflow  can remain the same and automatically do the black check).- Remove the "black" environment from tox.ini, since it is now part  of the linting done in the "lint" environment. (But README.md  remains the same, as it documented actually auto-formatting with  black, which is still done the same way.)- Add missing "exclude" keys for the shellcheck and black  pre-commit hooks to explicitly avoid traversing into the  submodules.
This splits the pre-commit hook for black into two hooks, bothusing the same repo and id but separately aliased:- black-check, which checks but does not modify files. This only  runs when the manual stage is specified, and it is used by tox  and on CI, with tox.ini and lint.yml modified accordingly.- black-format, which autoformats code. This provides the behavior  most users will expect from a pre-commit hook for black. It runs  automatically along with other hooks. But tox.ini and lint.yml,  in addition to enabling the black-check hook, also explicitly  skip the black-format hook.The effect is that in ordinary development the pre-commit hook forblack does auto-formatting, but that pre-commit continues to beused effectively for running checks in which code should not bechanged.
In the lint workflow, passing extra-args replaced --all-files,rather than keeping it but adding the extra arguments after it.(But --show-diff-on-failure and --color=always were still passed.)
Now that the changes to lint.yml are fixed up, tested, and shownto be capable of revealing formatting errors, the formatting errorI was using for testing (which is from9fa1cee where I hadintroduced it inadvertently) can be fixed.
As on CI and with tox (but not having to build and create and use atox environment).This may not be the best way to do it, since currently theproject's makefiles are otherwise used only for things directlyrelated to building and publishing. However, this seemed like areadily available way to run the moderately complex command thatproduces the same behavior as on CI or with tox, but outside a toxenvironment. It may be possible to improve on this in the future.
This adds BUILDDIR as a variable in the documentation generationmakefile that, along SPHINXOPTS, SPHINXBUILD, and PAPER, defaultsto the usual best value but can be set when invoking make.The main use for choosing a different build output directory is totest building without overwriting or otherwise interfering with anyfiles from a build one may really want to use. tox.ini is thusmodified to pass a path pointing inside its temporary directory asBUILDDIR, so the "html" tox environment now makes no changesoutside the .tox directory. This is thus suitable for being runautomatically, so the "html" environment is now in the envlist.
As well as still checking for Travis, for backward compatibilityand because experience shows that this is safe.The check can be much broader, and would be more accurate, withfewer false negatives. But a false positive can result in localdata loss because the script does hard resets on CI withoutprompting for confirmation. So for now, this just checks $TRAVISand $GITHUB_ACTIONS.Now that GHA is included, the CI workflows no longer need to set$TRAVIS when running the script, so that is removed.
This extends the init script so it tries harder to get versiontags:- As before, locally, "git fetch --all --tags" is always run. (This  also fetches other objects, and the developer experience would be  undesirably inconsistent if that were only sometimes done.)- On CI, run it if version tags are absent. The criterion followed  here and in subsequent steps is that if any existing tag starts  with a digit, or with the letter v followed by a digit, we regard  version tags to be present. This is to balance the benefit of  getting the tags (to make the tests work) against the risk of  creating a very confusing situation in clones of forks that  publish packages or otherwise use their own separate versioning  scheme (especially if those tags later ended up being pushed).- Both locally and on CI, after that, if version tags are absent,  try to copy them from the original gitpython-developers/GitPython  repo, without copying other tags or adding that repo as a remote.  Copy only tags that start with a digit, since v+digit tags aren't  currently used in this project (though forks may use them). This  is a fallback option and it always displays a warning. If it  fails, another message is issued for that. Unexpected failure to  access the repo terminates the script with a failing exit status,  but the absence of version tags in the fallback remote does not,  so CI jobs can continue and reveal which tests fail as a result.On GitHub Actions CI specifically, the Actions syntax for creatinga warning annotation on the workflow is used, but the warning isstill also written to stderr (as otherwise). GHA workflowannotations don't support multi-line warnings, so the message isadjusted into a single line where needed.
In the step output, the warning that produces a workflow annotationis fully readable (and even made to stand out), so it doesn't needto be printed in the plain way as well, which can be reserved forwhen GitHub Actions is not detected.
The tox environments that are not duplicated per Python versionwere set to run on Python 3.9, for consistency with Cygwin, where3.9 is the latest available (through official Cygwin packages), sooutput can be compared between Cygwin and other platforms whiletroubleshooting problems.However, this prevented them from running altogether on systemswhere Python 3.9 was not installed. So I've added all the otherPython versions the project supports as fallback versions for thosetox environments to use, in one of the reasonable precedenceorders, while keeping 3.9 as the first choice for the same reasonsas before.
This changed a while ago but README.md still listed having a mainbranch as a condition for automatic CI linting and testing in afork.
This is probably the *only* way anyone should run that script onWindows, but I don't know of specific bad things that happen if itis run in some other way, such as with WSL bash, aside from messingup line endings, which users are likely to notice anyway.This commit also clarifies the instructions by breaking up anotherparagraph that really represented two separate steps.
@EliahKaganEliahKagan marked this pull request as ready for reviewOctober 4, 2023 04:03
Copy link
Member

@ByronByron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thank youso much!

However, I would be pleased to make further changes as requested.

Even though I think you should do what seems best to you right now, I'd hope that the considerable time that I assume goes into the related issues and PRs could also be channeled towards not making improvements to these scripts necessary as one takes a path towards test isolation, which, as you already pointed out, would make such script unnecessary.

It is inelegant, though, [..]

Maybe you would consider it an improvement to 'upgrade' the Makefile like this:

help:## Display this help@awk'BEGIN {FS = ":.*##"; printf "\nNote: Make is only for specific functionality used by CI - run `just` for developer targets:\n  make \033[36m<target>\033[0m\n"} /^[a-zA-Z_-]+:.*?##/ { printf "  \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) }'$(MAKEFILE_LIST)always:##@ Release Buildsrelease-all: release release-lean release-small## all release buildsrelease: always## the default build, big but pretty (builds in ~2min 35s)cargo build --releaserelease-lean: always## lean and fast, with line renderer (builds in ~1min 30s)cargo build --release --no-default-features --features leanrelease-small: always## minimal dependencies, at cost of performance (builds in ~46s)cargo build --release --no-default-features --features small##@ Debug Buildsdebug: always## the default build, big but prettycargo builddebug-lean: always## lean and fast, with line renderercargo build --no-default-features --features leandebug-small: always## minimal dependencies, at cost of performancecargo build --no-default-features --features small

That way, documentation and headers can be inlined and will be displayed by default.
If the goal is to maximize portability then another direction could also be taken depending on how you see fit. I am pretty sure python projects use python as 'hub' script, too, these days.

head_sha="$(git rev-parse HEAD)"
latest_tag_sha="$(git rev-parse "${latest_tag}^{commit}")"

# Display a table of all the current version, tag, and HEAD commit information.
echo $'\nThe VERSION must be the same in all locations, and so must the HEAD and tag SHA'
echo
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This conveys spacing much better thanks to the separateecho line, even though I have a feeling this was done for portability as$'magic' might not be allowed everywhere.

Copy link
MemberAuthor

@EliahKaganEliahKaganOct 4, 2023
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I did it for spacing, and to to takefuller advantage ofecho to have fewer\n sequences. Although it's true that$'' is not POSIX and shouldn't be used in a#!/bin/sh script, this script is abash script and$'' is a available inbash on all systems. (ksh,zsh, and probably some other shells also have$'', but not all POSIX-compatible shells have it.)

Regarding spacing, the table is currently formatted usingprintf and this has the advantage that its own output spacing can be adjusted by, for example, changing14 to15. But I originally usedprintf for it because I was preferringprintf toecho, because originally I wasdoing this inline inMakefile, and aMakefile runs commands using/bin/sh (unlessSHELL is assigned in theMakefile itself). This was to follow the practice of avoidingecho in portablesh scripts,at least when displaying data read from files.

But that is not necessary in a#!/bin/bash script, where we know howecho works and whereecho doesn't expand escape sequences by default. So it would actually be okay, at this point, to go back to displaying the table this way:

echoecho'The VERSION must be the same in all locations, and so must the HEAD and tag SHA'echo"VERSION file   =$version_version"echo"changes.rst    =$changes_version"echo"Latest tag     =$latest_tag"echo"HEAD SHA       =$head_sha"echo"Latest tag SHA =$latest_tag_sha"

This is arguably more readable, and arguably conveys spacing the best. I'd be happy to change it to this if you like.

(Another option could be to go in the opposite direction and make thischeck-version.sh script, and maybe thebuild-release.sh script as well, a#!/bin/sh script, which is feasible.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This is arguably more readable, and arguably conveys spacing the best. I'd be happy to change it to this if you like.
(Another option could be to go in the opposite direction and make thischeck-version.sh script, and maybe thebuild-release.sh script as well, a#!/bin/sh script, which is feasible.)

Sincetrap is supported in POSIX shells and thussh, making these scripts more portable seems valuable, even though I think that as the project currently is setup there isn't any need as it's only me using them.
Alternatives involve running these on CI which also supportsbash everywhere (at least so it seems).

If you think there is benefits to portability assuming that one day CI can perform trusted publishes, I think it's fair to adjust for portability next time you touch them. Otherwise, it would be just as fair to changeprintf withecho as shown above. It's perfectly optional as well.

Copy link
MemberAuthor

@EliahKaganEliahKaganOct 9, 2023
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The desired interaction oftrap andset -e may be achievable insh. Even if not, we don't have to usetrap; it can be replaced even without making the control-flow logic more complicated or error prone. For example, the script could accept-q to not append the failure message, and otherwise invoke itself with-q, check the result, print the message if it failed, and exit with the original failing exit code. (There is alsoan argument to be made thatset -e shouldn't be relied on at all.)

There are not many systems wherebash doesn't exist and can't be made available. However, due to the problem described in729372f--which only happens when MinGWmake is being used on a native Windows system that has also WSL installed, and does not happen outsidemake nor on other systems--the scripts'bash hashbangs are written as#!/bin/bash rather than#!/usr/bin/env bash, and thus assumebash is in/bin.

Making them#!/bin/sh scripts would also fix that. It is even safer to assumesh is inbin than to assumeenv is in/usr/bin, so#!/usr/bin/env sh is rarely used. (Anyway, there is no corresponding WSL-relatedsh.exe inSystem32.) So, when combined with the other minor reasons for using#!/bin/sh, that is probably worth doing.

As you've suggested, next time I work on those scripts I'll look into making them POSIXsh scripts. However, this would be unrelated to facilitating trusted publishing. CI does supportbash everywhere, at least on runners hosted by GitHub Actions.

Byron reacted with thumbs up emoji
@ByronByron merged commit385c314 intogitpython-developers:mainOct 4, 2023
@EliahKaganEliahKagan deleted the sh branchOctober 5, 2023 01:03
EliahKagan added a commit to EliahKagan/GitPython that referenced this pull requestOct 18, 2023
Since7110bf8 (ingitpython-developers#1693), "git submodule update --init --recursive"was not run on CI, on the mistaken grounds that the CI testworkflows would already have taken care of cloning all submodules(ever since4eef3ec when the "submodules: recursive" option wasadded to the actions/checkout step).This changes the init-tests-after-clone.sh script to again run thatcommand unconditionally, including on CI. The assumption that itwasn't needed on CI was based on the specific content ofGitPython's own GitHub Actions workflows. But this disregarded thatthe test suite is run on CI for *other* projects: specifically, fordownstream projects that package GitPython.
EliahKagan added a commit to EliahKagan/GitPython that referenced this pull requestOct 18, 2023
Since7110bf8 (ingitpython-developers#1693), "git submodule update --init --recursive"was not run on CI, on the mistaken grounds that the CI testworkflows would already have taken care of cloning all submodules(ever since4eef3ec when the "submodules: recursive" option wasadded to the actions/checkout step).This changes the init-tests-after-clone.sh script to again run thatcommand unconditionally, including on CI. The assumption that itwasn't needed on CI was based on the specific content ofGitPython's own GitHub Actions workflows. But this disregarded thatthe test suite is run on CI for *other* projects: specifically, fordownstream projects that package GitPython (gitpython-developers#1713).This also brings back the comment fromfc96980 that says more abouthow the tests rely on submodules being present (specifically, thatthey need a submodule with a submodule). However, that is notspecifically related to the bug being fixed.
@EliahKagan
Copy link
MemberAuthor

as one takes a path towards test isolation, which, as you already pointed out, would make such script unnecessary.

For the benefit of anyone (including future me) finding this while searching for it, the issue for how GitPython's tests currently depend on the GitPython repository being present is#914.

EliahKagan added a commit to EliahKagan/GitPython that referenced this pull requestMar 12, 2024
This is to make it so simple `tox` usage has the expected propertyof leaving all source code files in the working tree unchanged.As noted inc66257e, linting how sometimes performs auto-fixessincegitpython-developers#1862, and the pre-commit command in tox.ini, which had alsorun `black --check`, will do even more file editing withgitpython-developers#1865.The bifurcation for black into separate mutating and non-mutatinghooks, introduced in5d8ddd9 (gitpython-developers#1693), is not carried over into Ruffautoformatting ingitpython-developers#1865. But also it:- Was not necessarily a good approach, and likely should not be  preserved in any form. It is an unusual and unintuitive use of  pre-commit. (It can be brought back if no better approach is  found, though.)- Was done to avoid a situation where it was nontrivial to set up  necessary dependencies for linting in the GitPython virtual  environment itself, because flake8 and its various plugins would  have to be installed.  They were not listed in any existing or newly introduced extra  (for example, they were not added to test-requirements.txt) in  part in the hope that they would all be replaced by Ruff, which  happened ingitpython-developers#1862.- Already does not achieve its goal sincegitpython-developers#1862, since it was  (probably rightly) not extended to Ruff linting to use/omit --fix.Now that Ruff is being used, people can run `pip install ruff` in avirtual environment, then run the `ruff` command however they like.This takes the place of multiple tools and plugins.This commit *avoids* doing any of the following, even though it maybe useful to do them later:- This does not give specific instructions in the readme for  installing and running ruff (andc66257e before this also omits  that). This can be added later and the best way to document it  may depend on some other upcoming decisions (see below).- This does not add ruff to the test extra or as any other kind of  extra or optional dependency. Although the test extra currently  contains some packages not used for running unit tests, such as  pre-commit and mypy, adding Ruff will cause installation to take  a long time and/or or fail on some platforms like Cygwin where  Ruff has to be built from (Rust) source code. This can be solved  properly by reorganizing the extras, but that is likely to wait  until they are expressed completely inside pyproject.toml rather  than one per requirements file (see discussion in comments  ingitpython-developers#1716 for general information about possible forthcoming  changes in how the project is defined).- This does not update tox.ini to run ruff directly, which could be  done by splitting the current lint tox environment into two or  three environments for Python linting, Python autoformatting, and  the miscellaneous other tasks performed by pre-commit hooks, only  the latter of which would use the pre-commit command. Whether and  how this should be done may depend on other forthcoming changes.- This does not remove or update the Makefile "lint" target. That  target should perhaps not have been added, and was always meant  to be improved/replaced, since everything else in the top-level  Makefile is for building and publishing releases. See5d15063  (gitpython-developers#1693). This is likewise not done now since it may depend on  as-yet unmerged changes and tooling decisions not yet made. It  should be feasible to do together when further updating tox.ini.- This does not update tox.ini, Makefile, or the lint.yml GitHub  Actions workflow to omit the manual hook-stage, which will be  unused as ofgitpython-developers#1865. This would complicate integration of changes,  and once it is known that it won't be needed, it can removed.The situation with the tox "lint" environment is thus now similarto that of the tox "html" environment when it was added ine6ec6c8(gitpython-developers#1667), until it was improved inf094909 (gitpython-developers#1693) to run withproper isolation.
EliahKagan added a commit to EliahKagan/GitPython that referenced this pull requestMar 13, 2024
This is to make it so simple `tox` usage has the expected propertyof leaving all source code files in the working tree unchanged.Linting how sometimes performs auto-fixes sincegitpython-developers#1862, and thepre-commit command in tox.ini, which had also run `black --check`,will do even more file editing due to the changes ingitpython-developers#1865.The bifurcation for black into separate mutating and non-mutatinghooks, introduced in5d8ddd9 (gitpython-developers#1693), was not carried over intoRuff autoformatting ingitpython-developers#1865. But also it:- Was not necessarily a good approach, and likely should not be  preserved in any form. It was an unusual and unintuitive use of  pre-commit. (It can be brought back if no better approach is  found, though.)- Was done to avoid a situation where it was nontrivial to set up  necessary dependencies for linting in the GitPython virtual  environment itself, because flake8 and its various plugins would  have to be installed.  They were not listed in any existing or newly introduced extra  (for example, they were not added to test-requirements.txt) in  part in the hope that they would all be replaced by Ruff, which  happened ingitpython-developers#1862.- Already did not achieve its goal as ofgitpython-developers#1862, since it was  (probably rightly) not extended to Ruff linting to use/omit --fix.Now that Ruff is being used, people can run `pip install ruff` in avirtual environment, then run the `ruff` command however they like.This takes the place of multiple tools and plugins.The situation with the tox "lint" environment is thus now similarto that of the tox "html" environment when it was added ine6ec6c8(gitpython-developers#1667), until it was improved inf094909 (gitpython-developers#1693) to run withproper isolation.
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@ByronByronByron approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Tooling is somewhat poorly discoverable Init script presents challenges on some platforms Tests don't readily pass in forks
2 participants
@EliahKagan@Byron

[8]ページ先頭

©2009-2025 Movatter.jp