Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

feat(api): support file format for repository archive#1561

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
JohnVillalovos merged 1 commit intopython-gitlab:mainfromdAnjou:patch-1
Dec 21, 2021

Conversation

dAnjou
Copy link
Contributor

No description provided.

@codecov-commenter
Copy link

codecov-commenter commentedJul 31, 2021
edited
Loading

Codecov Report

Merging#1561 (3910048) intomaster (ae97196) willdecrease coverage by4.49%.
The diff coverage is33.33%.

@@            Coverage Diff             @@##           master    #1561      +/-   ##==========================================- Coverage   91.16%   86.67%   -4.50%==========================================  Files          74       74                Lines        4177     4179       +2     ==========================================- Hits         3808     3622     -186- Misses        369      557     +188
FlagCoverage Δ
cli_func_v480.71% <33.33%> (-0.04%)⬇️
py_func_v4?
unit82.26% <33.33%> (-0.04%)⬇️

Flags with carried forward coverage won't be shown.Click here to find out more.

Impacted FilesCoverage Δ
gitlab/v4/objects/repositories.py55.73% <33.33%> (-25.62%)⬇️
gitlab/v4/objects/files.py61.76% <0.00%> (-30.89%)⬇️
gitlab/v4/objects/milestones.py71.42% <0.00%> (-28.58%)⬇️
gitlab/v4/objects/tags.py63.88% <0.00%> (-25.00%)⬇️
gitlab/utils.py65.51% <0.00%> (-24.14%)⬇️
gitlab/v4/objects/sidekiq.py80.95% <0.00%> (-19.05%)⬇️
gitlab/v4/objects/commits.py78.26% <0.00%> (-15.95%)⬇️
gitlab/v4/objects/snippets.py82.92% <0.00%> (-14.64%)⬇️
gitlab/v4/objects/clusters.py85.71% <0.00%> (-14.29%)⬇️
... and12 more

Copy link
Member

@JohnVillalovosJohnVillalovos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This will also need a documentation update inhttps://python-gitlab.readthedocs.io/en/stable/gl_objects/projects.html

See "Get the repository archive:"

And can it specify the possible options fromhttps://docs.gitlab.com/ee/api/repositories.html#get-file-archive:
tar.gz, tar.bz2, tbz, tbz2, tb2, bz2, tar, and zip.

@dAnjou
Copy link
ContributorAuthor

Thanks,@JohnVillalovos! Kinda submitted this as a conversation starter because I also wasn't quite sure about whether and where to add tests.

Will make the changes you suggested.

@dAnjou
Copy link
ContributorAuthor

And can it specify the possible options fromhttps://docs.gitlab.com/ee/api/repositories.html#get-file-archive:
tar.gz, tar.bz2, tbz, tbz2, tb2, bz2, tar, and zip.

I was hesitant to explicitly list them in the method documentation because GitLab might change them, so instead I added a note with a link to the example.

@nejch
Copy link
Member

And can it specify the possible options fromhttps://docs.gitlab.com/ee/api/repositories.html#get-file-archive:
tar.gz, tar.bz2, tbz, tbz2, tb2, bz2, tar, and zip.

I was hesitant to explicitly list them in the method documentation because GitLab might change them, so instead I added a note with a link to the example.

That's a good point, maybe this is easier long-term WDYT@JohnVillalovos?

If you want to add tests - maybe extract these lines intotest_repository_archive() and try adding a different format:

archive=project.repository_archive()
assertisinstance(archive,bytes)
archive2=project.repository_archive("master")
assertarchive==archive2

And maybe test that it really worked with e.g.
https://docs.python.org/3/library/tarfile.html#tarfile.is_tarfile
https://docs.python.org/3/library/zipfile.html#zipfile.is_zipfile

@JohnVillalovos
Copy link
Member

JohnVillalovos commentedAug 1, 2021
edited
Loading

And can it specify the possible options fromhttps://docs.gitlab.com/ee/api/repositories.html#get-file-archive:
tar.gz, tar.bz2, tbz, tbz2, tb2, bz2, tar, and zip.

I was hesitant to explicitly list them in the method documentation because GitLab might change them, so instead I added a note with a link to the example.

That's a good point, maybe this is easier long-term WDYT@JohnVillalovos?

Sounds reasonable to me. Could also add in thenote

As of 2021-08, the format options are: tar.gz, tar.bz2, tbz, tbz2, tb2, bz2, tar, and zip.For the formats available, refer tohttps://docs.gitlab.com/ce/api/repositories.html#get-file-archive

I am perfectly fine with the current doc update with only the pointer to the Gitlab docs.

@dAnjoudAnjouforce-pushed thepatch-1 branch 2 times, most recently fromac51f09 to3910048CompareAugust 6, 2021 13:55
@dAnjou
Copy link
ContributorAuthor

I've added a test and ran it locally a few times. For some reason it always got a 404 fortar.gz andtar.bz2. No idea why, because I created a project on the GitLab instance that the tests use and getting these formats via the API worked perfectly fine.

@nejch
Copy link
Member

I've added a test and ran it locally a few times. For some reason it always got a 404 fortar.gz andtar.bz2. No idea why, because I created a project on the GitLab instance that the tests use and getting these formats via the API worked perfectly fine.

@dAnjou Thanks, maybe it's something to do with URL encoding since they include. and we've had issues with this before. I'll take a look.

@JohnVillalovos
Copy link
Member

Thanks@dAnjou

Can you rebase the patch rather than adding a merge commit?

@JohnVillalovos
Copy link
Member

I've added a test and ran it locally a few times. For some reason it always got a 404 fortar.gz andtar.bz2. No idea why, because I created a project on the GitLab instance that the tests use and getting these formats via the API worked perfectly fine.

Strange. I wonder why that is happening.@nejch did you ever have a chance to investigate?

@JohnVillalovos
Copy link
Member

JohnVillalovos commentedDec 18, 2021
edited
Loading

Strange. I wonder why that is happening.@nejch did you ever have a chance to investigate?

I think I figured it out. See#1758 for my "fix" and can see the functional tests pass.

@nejch would like to get your feedback on this.

I really don't think that we should be converting periods to%2E as I believe we have other issues about that causing problems.

@nejch
Copy link
Member

Ok, so itis encoding drama again 😅

I really don't think that we should be converting periods to %2E as I believe we have other issues about that causing problems.

Could be, would love to get rid of more custom code. Do you have a link to the other issue? We just need to make sure that this is not needed anymore:

# Requests assumes that `.` should not be encoded as %2E and will make
# changes to urls using this encoding. Using a prepped request we can
# get the desired behavior.
# The Requests behavior is right but it seems that web servers don't
# always agree with this decision (this is the case with a default
# gitlab installation)

So maybe need to check again whata221d7b was trying to achieve. And possibly test this with paths that may include dots so we don't have a regression 🤔

JohnVillalovos added a commit that referenced this pull requestDec 19, 2021
Forcing the encoding of '.' to '%2E' causes issues. It also goesagainst the RFC:https://datatracker.ietf.org/doc/html/rfc3986.html#section-2.3From the RFC:  For consistency, percent-encoded octets in the ranges of ALPHA  (%41-%5A and %61-%7A), DIGIT (%30-%39), hyphen (%2D), period (%2E),  underscore (%5F), or tilde (%7E) should not be created by URI  producers...Related#1006Related#1356Related#1561BREAKING CHANGE: This possibly could be a breaking for users who haveincorrectly configured GitLab servers which don't handle period '.'characters correctly.
JohnVillalovos added a commit that referenced this pull requestDec 19, 2021
Forcing the encoding of '.' to '%2E' causes issues. It also goesagainst the RFC:https://datatracker.ietf.org/doc/html/rfc3986.html#section-2.3From the RFC:  For consistency, percent-encoded octets in the ranges of ALPHA  (%41-%5A and %61-%7A), DIGIT (%30-%39), hyphen (%2D), period (%2E),  underscore (%5F), or tilde (%7E) should not be created by URI  producers...Closes#1006Related#1356Related#1561BREAKING CHANGE: This possibly could be a breaking for users who haveincorrectly configured GitLab servers which don't handle period '.'characters correctly.
JohnVillalovos added a commit that referenced this pull requestDec 20, 2021
Forcing the encoding of '.' to '%2E' causes issues. It also goesagainst the RFC:https://datatracker.ietf.org/doc/html/rfc3986.html#section-2.3From the RFC:  For consistency, percent-encoded octets in the ranges of ALPHA  (%41-%5A and %61-%7A), DIGIT (%30-%39), hyphen (%2D), period (%2E),  underscore (%5F), or tilde (%7E) should not be created by URI  producers...Closes#1006Related#1356Related#1561BREAKING CHANGE: This could potentially be a breaking change for userswho have incorrectly configured GitLab servers which don't handleperiod '.' characters correctly.
JohnVillalovos added a commit that referenced this pull requestDec 20, 2021
Forcing the encoding of '.' to '%2E' causes issues. It also goesagainst the RFC:https://datatracker.ietf.org/doc/html/rfc3986.html#section-2.3From the RFC:  For consistency, percent-encoded octets in the ranges of ALPHA  (%41-%5A and %61-%7A), DIGIT (%30-%39), hyphen (%2D), period (%2E),  underscore (%5F), or tilde (%7E) should not be created by URI  producers...Closes#1006Related#1356Related#1561BREAKING CHANGE: This could potentially be a breaking change for userswho have incorrectly configured GitLab servers which don't handleperiod '.' characters correctly.
@JohnVillalovos
Copy link
Member

Ok, so itis encoding drama again 😅

I really don't think that we should be converting periods to %2E as I believe we have other issues about that causing problems.

Could be, would love to get rid of more custom code. Do you have a link to the other issue? We just need to make sure that this is not needed anymore:

I created a PR#1766 to get rid of it and referenced the issues I found, though I believe I had seen other things related to this but didn't find them in my quick search.

So maybe need to check again whata221d7b was trying to achieve. And possibly test this with paths that may include dots so we don't have a regression 🤔

I thought we cleaned that up somewhat ind56a434 But I'm not exactly sure what you are asking for, probably because I'm a bit tired at the moment and not reading everything 😟

Not sure if you really meant this commitd7c7911 which first added that '.' -> '%2E' translation. Very little info there besides saying it didn't work with a default gitlab installation. But my PR above is passing all the functional tests without an issues.

@nejch
Copy link
Member

Not sure if you really meant this commitd7c7911 which first added that '.' -> '%2E' translation. Very little info there besides saying it didn't work with a default gitlab installation. But my PR above is passing all the functional tests without an issues.

Sorry yes this is what I meant, I just did a quick blame there. 🤦

Ahh interesting: it seems like the underlying issue was fixed in GitLab 13.2 by upgrading the Grape library. Previously GitLab needed%2E encoding for the API it seems, If I understand this MR and the MRs linked correctly:https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36038. So looks like it's being tested for regressions upstream as well 👍

@JohnVillalovos
Copy link
Member

Ahh interesting: it seems like the underlying issue was fixed in GitLab 13.2 by upgrading the Grape library. Previously GitLab needed%2E encoding for the API it seems, If I understand this MR and the MRs linked correctly:https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36038. So looks like it's being tested for regressions upstream as well 👍

Nice research!!! 👍

JohnVillalovos added a commit that referenced this pull requestDec 20, 2021
Forcing the encoding of '.' to '%2E' causes issues. It also goesagainst the RFC:https://datatracker.ietf.org/doc/html/rfc3986.html#section-2.3From the RFC:  For consistency, percent-encoded octets in the ranges of ALPHA  (%41-%5A and %61-%7A), DIGIT (%30-%39), hyphen (%2D), period (%2E),  underscore (%5F), or tilde (%7E) should not be created by URI  producers...Closes#1006Related#1356Related#1561BREAKING CHANGE: stop encoding '.' to '%2E'. This could potentially bea breaking change for users who have incorrectly configured GitLabservers which don't handle period '.' characters correctly.
JohnVillalovos added a commit that referenced this pull requestDec 20, 2021
Forcing the encoding of '.' to '%2E' causes issues. It also goesagainst the RFC:https://datatracker.ietf.org/doc/html/rfc3986.html#section-2.3From the RFC:  For consistency, percent-encoded octets in the ranges of ALPHA  (%41-%5A and %61-%7A), DIGIT (%30-%39), hyphen (%2D), period (%2E),  underscore (%5F), or tilde (%7E) should not be created by URI  producers...Closes#1006Related#1356Related#1561BREAKING CHANGE: stop encoding '.' to '%2E'. This could potentially bea breaking change for users who have incorrectly configured GitLabservers which don't handle period '.' characters correctly.
@JohnVillalovos
Copy link
Member

@dAnjou Would you be willing to rebase this? I think we fixed the issue that caused the functional tests to fail in a PR that got merged today PR#1766

@JohnVillalovos
Copy link
Member

@dAnjou Would you be willing to rebase this? I think we fixed the issue that caused the functional tests to fail in a PR that got merged today PR#1766

I got impatient 😊So I did it myself.

@JohnVillalovosJohnVillalovosenabled auto-merge (rebase)December 21, 2021 16:57
@JohnVillalovosJohnVillalovos merged commit83dcabf intopython-gitlab:mainDec 21, 2021
@dAnjou
Copy link
ContributorAuthor

Thank you,@JohnVillalovos! Glad the change made it 😊

JohnVillalovos reacted with thumbs up emoji

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@JohnVillalovosJohnVillalovosJohnVillalovos approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

4 participants
@dAnjou@codecov-commenter@nejch@JohnVillalovos

[8]ページ先頭

©2009-2025 Movatter.jp