This PR fixes the problem when converting query results to Pandas withpyarrow when data contains timestamps that would fall out ofpyarrow's nanoseconds precision.

The fix requirespyarrow>=1.0.0, thus it only works on Python 3.

PR checklist

Make sure to open an issue as abug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

plamut requested a review fromtswast

August 1, 2020 16:10

google-clabot added the cla: yesThis human has signed the Contributor License Agreement. label

Aug 1, 2020

Copy link

ContributorAuthor

plamut commentedAug 1, 2020

@tswast There is inconsistency with the existingdate_as_object option thatis exposed to the users, while thetimestamp_as_object option is hidden. Let me know if you want to unify these two approaches to a similar problem.

tswast reviewed

Aug 3, 2020

View reviewed changes

Copy link

Contributor

tswast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks!

Regardingdate_as_object, it's a little different in that case, because it doesn't throw an error for dates. They just come back as strings if it's not set.

If we do providetimestamp_as_object, I think it needs to be 3 states:

(default) the behavior in this fix
(explicitly false) let the error happen, since they want to use native pandas Timestamp (maybe for performance reasons)
(explicitly true) always convert to datetime objects

google/cloud/bigquery/table.py OutdatedShow resolvedHide resolved

fix: converting timestamps/datetimes out of pyarrow bounds to dataframe

796eb94

plamut force-pushed theiss-168 branch from485fbc2 to796eb94Compare

August 6, 2020 12:26

plamut marked this pull request as ready for review

August 6, 2020 12:29

plamut requested review fromshollyman andtswast

August 6, 2020 12:29

Copy link

ContributorAuthor

plamut commentedAug 6, 2020

Let me know if I should also add an explicittimestamp_as_object parameter as envisioned by@tswast, or should we leave it out from this fix and (maybe) add it in a separate feature PR.