- Notifications
You must be signed in to change notification settings - Fork126
Open
Assignees
Description
When creating aConnection, currently it's possible to configure some aspects of pandas / arrow, such as_use_arrow_native_complex_types and_use_arrow_native_timestamps, but_convert_arrow_table uses a hardcoded type mapper that's impossible to change:
databricks-sql-python/src/databricks/sql/client.py
Lines 1349 to 1362 in0947b9a
| dtype_mapping= { | |
| pyarrow.int8():pandas.Int8Dtype(), | |
| pyarrow.int16():pandas.Int16Dtype(), | |
| pyarrow.int32():pandas.Int32Dtype(), | |
| pyarrow.int64():pandas.Int64Dtype(), | |
| pyarrow.uint8():pandas.UInt8Dtype(), | |
| pyarrow.uint16():pandas.UInt16Dtype(), | |
| pyarrow.uint32():pandas.UInt32Dtype(), | |
| pyarrow.uint64():pandas.UInt64Dtype(), | |
| pyarrow.bool_():pandas.BooleanDtype(), | |
| pyarrow.float32():pandas.Float32Dtype(), | |
| pyarrow.float64():pandas.Float64Dtype(), | |
| pyarrow.string():pandas.StringDtype(), | |
| } |
Similarly, there's no way to pass other parameters to theto_pandas call here:
databricks-sql-python/src/databricks/sql/client.py
Lines 1366 to 1370 in0947b9a
| df=table_renamed.to_pandas( | |
| types_mapper=dtype_mapping.get, | |
| date_as_object=True, | |
| timestamp_as_object=True, | |
| ) |
Please consider extending the connection parameters to add more flexibility.
Metadata
Metadata
Assignees
Labels
No labels