- Notifications
You must be signed in to change notification settings - Fork322
Closed
Description
Certain values ofdatetime lose a single microsecond on their way into BigQuery. This appears to happen because of the JSON rendering of timestamps as a floating point number of seconds.
Environment details
- OS type and version: Ubuntu Bionic
- Python version: 3.6.9
- pip version: 9.0.1
google-cloud-bigqueryversion: 1.28.0
Steps to reproduce
Here is an example of adatetime value that loses a microsecond:
datetime(2020,11,17,1,6,52,353795,tzinfo=datetime.timezone(datetime.timedelta(0),'UTC'))
As a full integration repro, you could try something like this:
- Create a BigQuery table with a
TIMESTAMPcolumn. - Use
insert_rowsto insert the value above. - Inspect the results in the Console, where it will show as
2020-11-17 01:06:52.353794 UTCinstead of2020-11-17 01:06:52.353795 UTC.
Code example
An easier repro hints at the root cause. By looking at the output of_timestamp_to_json_row, it appears that the problematic values are rounded down to microsecond precision.
importdatetimefromgoogle.cloudimportbigqueryasbqts=datetime.datetime(2020,11,17,1,6,52,353795,tzinfo=datetime.timezone(datetime.timedelta(0),'UTC'))print(bq._helpers._timestamp_to_json_row(ts))
This will print:
1605575212.3537948Perhaps the solution is to add a half microsecond (0.5e-6) so that truncating results in rounding to the nearest microsecond.