Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Don't change the meaning of string literals#554

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
nvie merged 1 commit intogitpython-developers:masterfromnvie:master
Dec 8, 2016
Merged

Don't change the meaning of string literals#554

nvie merged 1 commit intogitpython-developers:masterfromnvie:master
Dec 8, 2016

Conversation

nvie
Copy link
Contributor

@nvienvie commentedDec 8, 2016
edited
Loading

This line singlehandedly has caused a slew of painful bugs for us in production since we upgraded from 2.0.8 to 2.0.9, seeing various UnicodeDecodeErrors on repositories with branches/tags/etc that have non-ASCII characters in them. For most repos that only use the ASCII alphabet, implicit encoding and decoding is now happening, and most of the time we get lucky because they accidentally just work.

After reverting this one liner, our production issues all went away.

@ankostis What's the reason this line was included inf11fdf1? It's the only module in the GitPython code base now that uses unicode literals, which makes reasoning about the code related to strings in this module rather confusing. Could we perhaps just change the strings thatshould be unicode for your patch and mark those withu'...' string literals explicitly rather than changing the meaning of all strings at once?

@ankostis
Copy link
Contributor

ankostis commentedDec 8, 2016
edited
Loading

Dear@nvie I deeply apologize for the problems this change has caused you.
You are right, there is no reasons for this change or any explanation in the commit-msg,
so I deduce it could have been a mistake on my part, while trying various fixes for Windows & PY3,
and forgetting to revert them before commit

What is surprising is that no TC fails when reverting this line.
So it indeed there was no real reason for this change.

What is unsettling is that we have no TCs to detect the problems you had.
Can you help into building such a test-case?

@nvie
Copy link
ContributorAuthor

nvie commentedDec 8, 2016
edited
Loading

Dear@nvie I deeply apologize for the problems this change has caused you.

No need to! We've all been there ;)

You are right, there is no reasons for this change or any explanation in the commit-msg,
so I deduce it could have been a mistake on my part, while trying various fixes for Windows & PY3,
and forgetting to revert them before commit.

I thought that might have been the case since it's so isolated, just wanted to double-check this.

What is surprising is that no TC fails when reverting this line.
What is unsettling is that we have no TCs to detect the problems you had.
Can you help into building such a test-case?

I thought of this too, but the bugs are so subtle and hard to replicate that I didn't have a good place to start adding them. The original bugs we have been seeing popped up in a whole different section of the code, so it was quite a journey to end up here, finding this was the root cause.

FWIW, I've included the stack trace that triggered out bug hunt here.

Example git-fetch output that causes this bug to reveal itself:

= [up to date]          feature/mybranch                -> feature/mybranch= [up to date]          feature/foo-‘bar’               -> feature/foo-‘bar’= [up to date]          feature/anotherbranch           -> feature/anotherbranch

Note the "smart quotes" in the 2nd line. That's an example of a repo that will fail.

Stack trace does not reveal any key source code position. I had to trace the data at each stack level and see where data was "wrong", then try to trace back how that data came to be that way.

Traceback (most recent call last):   File "/app/src/foo.py", line 136, in _update     custom_fetch(git_repo, progress, url, '+refs/*:refs/*')   File "/app/src/lib/bar.py", line 48, in custom_fetch     return remote._get_fetch_info_from_stderr(proc, progress)   File "/app/.venv/src/gitpython/git/remote.py", line 638, in _get_fetch_info_from_stderr     proc.wait(stderr=stderr_text)   File "/app/.venv/src/gitpython/git/cmd.py", line 290, in wait     raise GitCommandError(self.args, status, errstr) GitCommandError: Cmd('git') failed due to: exit code(-13)   cmdline: git fetch --progress -v https://****:****@github.com/someuser/somerepo.git +refs/*:refs/*   stderr: 'fatal: repository 'https://****:****@github.com/someuser/somerepo.git/' not found'

As you can see, this stack trace is of no help finding this bug.

jonathanchu reacted with thumbs up emoji

@nvienvie merged commita8437c0 intogitpython-developers:masterDec 8, 2016
@nvie
Copy link
ContributorAuthor

nvie commentedDec 8, 2016

Thanks for the quick response, though. I went ahead and merged this one myself.

Copy link
Contributor

@ankostisankostis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

unicode_literals were forgotten from a Q'nD experiment while fighting with Win+PY3 errors.

@nvie
Copy link
ContributorAuthor

nvie commentedDec 8, 2016

@Byron Could you perhaps roll a release of GitPython to PyPI that includes this fix? Would be appreciated!

@Byron
Copy link
Member

@nvie Will have to work through more emails and PRs, but will cut a new release when done. That should happen today as well.

@nvie
Copy link
ContributorAuthor

nvie commentedDec 8, 2016

Thanks, no rush!

@ankostis
Copy link
Contributor

So@nvie, the unicode problem happens only withfetch cmd?
Or have you encountered problems with unicode-branches in other commands?

Just linking#550 which might also relate to strangely-named branches - so that we can craft a TC for all these problems.

@nvie
Copy link
ContributorAuthor

nvie commentedDec 8, 2016
edited
Loading

@ankostis Yes and no. We thought it was fetch-related for a long time at first, because that's the only place it popped up for us. I think more Git commands could be affected however. The specific bug happenedon this line in the end. Because of theunicode_literals, this was trying to see if a bytes string (input of the function) started with the now-unicode string literals given on that line ('error' and'fatal'). This caused the implicit decoding step and the failure.

This particular one happened in theRemoteProgress class, so anything using that would surely be affected, but the same bug is more wide-spread than just this class, since in a way this class has nothing to do with it. Basically any place that does a.startswith('whatever') (or whatever other method on strings that cause implicit string decoding) is a potential hiding place for this bug.

@nvie
Copy link
ContributorAuthor

nvie commentedDec 8, 2016

Here's one idea to systematically find all places affected:

  1. Find all places where ax.startswith(y) is called.
  2. Put an explicitassert type(x) == type(y), "A bug is lurking here" above the line.
  3. Try touching all these code paths.

Any place where this fails is a similar bug.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@ankostisankostisankostis left review comments

Assignees

@ByronByron

Labels
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

3 participants
@nvie@ankostis@Byron

[8]ページ先頭

©2009-2025 Movatter.jp