- Notifications
You must be signed in to change notification settings - Fork126
Closed
Description
It's a great project, and truly helps me in a variety of ways. However, I am having a few issues when utilizing it with Pandas.
When reading a table I would like to do:
defread(self,catalog_name:str,schema_name:str,table_name:str )->DataFrame:withself._get_connection(catalog_name)asconnection:iterator=pd.read_sql_table(schema=schema_name,table_name=table_name,con=connection, )
This should return a pandas dataframe.
However, I am required to do the following lacking features from your otherwise great connection.
defread(self,catalog_name:str,schema_name:str,table_name:str )->pd.DataFrame:withself._get_connection(catalog_name)asconnection:query=f"SELECT * FROM{catalog_name}.{schema_name}.{table_name}"self.logger.debug("Running query '%s'",query)df=pd.read_sql(query,connection)self.logger.debug(df)returndf
Similarly I cannot use theto_sql on a dataframe where i'd like to do something like the following:
defwrite(self,dataframe:pd.DataFrame,catalog_name:str,schema_name:str,table_name:str, )->pd.DataFrame:withself._get_connection(catalog_name)asconnection:dataframe.to_sql(name=table_name,con=connection,schema=schema_name)
Both fail
databricks.sql.exc.NotSupportedError: Transactions are not supported on DatabricksVersions:
python 3.10.6databricks-sql-connector==2.2.1pandas==1.5.2Metadata
Metadata
Assignees
Labels
No labels