Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

feat: Add compression option ZSTD.#1890

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
chalmerlowe merged 13 commits intomainfrompatch-1
Apr 11, 2024
Merged

feat: Add compression option ZSTD.#1890

chalmerlowe merged 13 commits intomainfrompatch-1
Apr 11, 2024

Conversation

@chalmerlowe
Copy link
Collaborator

Based on a PR submitted by EthanSteinberg.

One of BigQuery's neat features is how it supports ZSTD compression for exports.
See:https://cloud.google.com/bigquery/docs/exporting-data#parquet_export_details
This commit simply adds ZSTD to the list of enums allowed for the compression type.

I added a test to confirm that the enum is correctly populated with only the allowed options.

For future me, including this link directly to the list ofcurrent export formats and the allowable compression types.

Closing Ethan's PR.

@chalmerlowechalmerlowe requested review froma team ascode ownersApril 8, 2024 14:32
@product-auto-labelproduct-auto-labelbot added size: sPull request size is small. api: bigqueryIssues related to the googleapis/python-bigquery API. labelsApr 8, 2024
@chalmerlowechalmerlowe self-assigned thisApr 8, 2024
@chalmerlowechalmerlowe added the do not mergeIndicates a pull request not ready for merge, due to either quality or timing. labelApr 8, 2024
@chalmerlowe
Copy link
CollaboratorAuthor

Before we merge, there are a couple items I wanna investigate.
More to come.

@chalmerlowechalmerlowe requested a review fromtswastApril 8, 2024 16:04
# limitations under the License.


deftest_compression_enums():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I wonder how useful this test is? Seems an awful lot like achange-detector test to me. I'd be fine adding the constant without adding the test.

Alternatively, maybe there's a system test we could write to make sure this is synced with thebigquery discovery document? But even then, compression isn't a true enum. The allowed values are only listed in the documentation string from what I can tell.

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@tswast

I will remove this test.

In terms of attempting to ensure that this matches the discovery docs... the terms referenced by the docs are present in the description, as you note, so we would need some mechanism to extract them from the discovery doc, which feels somewhat fragile (ie extract all words that are ALL CAPS and deduplicate them). Thoughts?

"JobConfigurationExtract": {      ...      "properties": {        "compression": {          "description": "Optional. The compression type to use for exported files.Possible values include DEFLATE, GZIP, NONE, SNAPPY, and ZSTD. Thedefault value is NONE. Not all compression formats are support for allfile formats. DEFLATE is only supported for Avro. ZSTD is only supportedfor Parquet. Not applicable when extracting models.",          "type": "string"        },

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Agreed. Without a structured representation of the allowed values, it's too fragile.

@chalmerlowechalmerlowe added automergeMerge the pull request once unit tests and other checks pass. and removed do not mergeIndicates a pull request not ready for merge, due to either quality or timing. labelsApr 11, 2024
@chalmerlowechalmerlowe merged commit5ed9cce intomainApr 11, 2024
@chalmerlowechalmerlowe deleted the patch-1 branchApril 11, 2024 19:33
@gcf-merge-on-greengcf-merge-on-greenbot removed the automergeMerge the pull request once unit tests and other checks pass. labelApr 11, 2024
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@tswasttswasttswast approved these changes

@LinchinLinchinAwaiting requested review from LinchinLinchin was automatically assigned from googleapis/api-bigquery

Assignees

@chalmerlowechalmerlowe

Labels

api: bigqueryIssues related to the googleapis/python-bigquery API.size: sPull request size is small.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@chalmerlowe@tswast@EthanSteinberg

[8]ページ先頭

©2009-2025 Movatter.jp