NotificationsYou must be signed in to change notification settings
Fork321
Star786

feat: add`reference_file_schema_uri` to LoadJobConfig, ExternalConfig#1399

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Jump to bottom

Merged

aribray merged 29 commits intogoogleapis:mainfromaribray:aribray--federated-formats

Nov 14, 2022

Merged

feat: add`reference_file_schema_uri` to LoadJobConfig, ExternalConfig#1399

aribray merged 29 commits intogoogleapis:mainfromaribray:aribray--federated-formats

Nov 14, 2022

Conversation

Copy link

Contributor

aribray commentedNov 4, 2022•
edited
Loading

Current behavior:

for load jobs from federated formats like AVRO, PARQUET, and ORC, BigQuery uses the schema of whichever file is lexicographically last.

Example:

source_uris = [    "gs://{project}/{bucket_name}/c-file.avro",     "gs://{project}/{bucket_name}/b-file.avro",    "gs://{project}/{bucket_name}/r-file.avro",]

"gs://{project}/{bucket_name}/r-file.avro" is lexicographically last

New behavior:

Thereference_file_schema_uri field allows users to specify the schema
Thereference_file_schema_uri does not have to be a file from thesource_uris list
To prevent data loss, thereference_file_schema_uri should be a superset of the schemas in thesource_uris list

Googlers see 246809557

aribray added4 commits

October 29, 2022 23:56

add property for reference_file_schema_uri

051aae4

feat: add 'reference_file_schema_uri' to LoadJobConfig

205b988

add 'reference_file_schema_uri' and tests to external config, loadjob…

484b36e

…config

Merge branch 'main' of github.com:aribray/python-bigquery into aribra…

dad486d

…y--federated-formats

product-auto-labelbot added size: m

Pull request size is medium.

api: bigqueryIssues related to the googleapis/python-bigquery API. labels

Nov 4, 2022

aribray added5 commits

November 3, 2022 21:51

use _make_dataset_id

d74e28d

add unit test

4f903ca

remove unit tests

25d3dee

add tests to test_load.py

c9e9ef6

Merge branch 'main' of github.com:aribray/python-bigquery into aribra…

8fa6d41

…y--federated-formats

aribray marked this pull request as ready for review

November 4, 2022 15:56

aribray requested a review froma team

November 4, 2022 15:56

aribray requested a review froma team as acode owner

November 4, 2022 15:56

aribray requested a review fromNeenu1995

November 4, 2022 15:56

aribray added2 commits

November 4, 2022 11:53

fix linting

dd0dc8f

add attribute assertions to system tests

e2f7b89

product-auto-labelbot added size: lPull request size is large. and removed size: mPull request size is medium. labels

Nov 4, 2022

aribray added2 commits

November 4, 2022 13:27

remove unit test

5f45057

add attribute to unit test

ee570bd

Neenu1995 approved these changes

Nov 4, 2022

View reviewed changes

leahecole approved these changes

Nov 9, 2022

View reviewed changes

Copy link

Contributor

leahecole left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Yeah with the nits, I'm honestly torn. Use your best judgment - it's nbd if it's not changed.

tests/system/test_client.py OutdatedShow resolvedHide resolved

add URIs as constants

efe60f1

product-auto-labelbot added size: mPull request size is medium. and removed size: lPull request size is large. labels

Nov 9, 2022

leahecole approved these changes

Nov 9, 2022

View reviewed changes

aribray added2 commits

November 10, 2022 09:06

fix linting

e4583f3

fix file format

40b5004

product-auto-labelbot removed the size: mPull request size is medium. label

Nov 10, 2022