- Notifications
You must be signed in to change notification settings - Fork321
Open
Description
Is your feature request related to a problem? Please describe.
The problematic field name is not returned in the error for theaugment_schema function like it is elsewhere. This can then show up in places likeload_table_from_dataframe sinceaugment_schema is called.
pyarrow.lib.ArrowTypeError: Expected bytes, got a 'float' object File "...", line 57, in load_table_bq job = client.load_table_from_dataframe( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.../.venv/lib/python3.12/site-packages/google/cloud/bigquery/client.py", line 2781, in load_table_from_dataframe new_job_config.schema = _pandas_helpers.dataframe_to_bq_schema( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.../.venv/lib/python3.12/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 491, in dataframe_to_bq_schema bq_schema_out = augment_schema(dataframe, bq_schema_out) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.../.venv/lib/python3.12/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 520, in augment_schema arrow_table = pyarrow.array(dataframe.reset_index()[field.name]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "pyarrow/array.pxi", line 360, in pyarrow.lib.array File "pyarrow/array.pxi", line 87, in pyarrow.lib._ndarray_to_array File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_statusDescribe the solution you'd like
Fix could be similar to#1836
defaugment_schema(dataframe,current_bq_schema): ...forfieldincurrent_bq_schema:iffield.field_typeisnotNone:augmented_schema.append(field)continuetry:arrow_table=pyarrow.array(dataframe.reset_index()[field.name])exceptArrowTypeError:msg=f"""Error converting Pandas column with name: "{field.name}" and datatype: "{field.dtype}" to an appropriate pyarrow datatype: ..."""_LOGGER.error(msg)raiseArrowTypeError(msg)
Happy to submit a PR if this would be approved?