- Notifications
You must be signed in to change notification settings - Fork126
SEA: Normalise Columns returned by Metadata Queries#627
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Closed
Uh oh!
There was an error while loading.Please reload this page.
Closed
Changes fromall commits
Commits
Show all changes
316 commits Select commitHold shift + click to select a range
484064e remove filtering, metadata ops
varun-edachali-dbx030edf8 raise NotImplementedErrror for metadata ops
varun-edachali-dbx4e07f1e align SeaResultSet with new structure
varun-edachali-dbx65e7c6b correct sea res set tests
varun-edachali-dbx30f8266 add metadata commands
varun-edachali-dbx033ae73 formatting (black)
varun-edachali-dbx33821f4 add metadata command unit tests
varun-edachali-dbx71b451a minimal fetch phase intro
varun-edachali-dbx170f339 Merge branch 'exec-resp-norm' into fetch-json-inline
varun-edachali-dbx40f79b5 Merge branch 'sea-res-set' into fetch-json-inline
varun-edachali-dbxc038d5a working JSON + INLINE
varun-edachali-dbx3e22c6c change to valid table name
varun-edachali-dbx716304b rmeove redundant queue init
varun-edachali-dbxe96e5b8 large query results
varun-edachali-dbx787f1f7 Merge branch 'sea-migration' into sea-test-scripts
varun-edachali-dbx165c4f3 remove un-necessary changes
varun-edachali-dbxa6e40d0 simplify test module
varun-edachali-dbx52e3088 logging -> debug level
varun-edachali-dbx641c09b change table name in log
varun-edachali-dbx8bd12d8 Merge branch 'sea-migration' into exec-models-sea
varun-edachali-dbxffded6e remove un-necessary changes
varun-edachali-dbx227f6b3 remove un-necessary backend cahnges
varun-edachali-dbx68657a3 remove un-needed GetChunksResponse
varun-edachali-dbx3940eec remove un-needed GetChunksResponse
varun-edachali-dbx37813ba reduce code duplication in response parsing
varun-edachali-dbx267c9f4 reduce code duplication
varun-edachali-dbx2967119 more clear docstrings
varun-edachali-dbx47fd60d introduce strongly typed ChunkInfo
varun-edachali-dbx982fdf2 remove is_volume_operation from response
varun-edachali-dbx9e14d48 add is_volume_op and more ResultData fields
varun-edachali-dbxbe1997e Merge branch 'exec-models-sea' into exec-phase-sea
varun-edachali-dbxe8e8ee7 Merge branch 'sea-test-scripts' into exec-phase-sea
varun-edachali-dbx05ee4e7 add test scripts
varun-edachali-dbx3ffa898 Merge branch 'exec-models-sea' into metadata-sea
varun-edachali-dbx2952d8d Revert "Merge branch 'sea-migration' into exec-models-sea"
varun-edachali-dbx89e2aa0 Merge branch 'exec-phase-sea' into metadata-sea
varun-edachali-dbxcbace3f Revert "Merge branch 'exec-models-sea' into exec-phase-sea"
varun-edachali-dbxc075b07 change logging level
varun-edachali-dbxc62f76d remove un-necessary changes
varun-edachali-dbx199402e remove excess changes
varun-edachali-dbx8ac574b remove excess changes
varun-edachali-dbx398ca70 Merge branch 'sea-migration' into exec-phase-sea
varun-edachali-dbxb1acc5b remove _get_schema_bytes (for now)
varun-edachali-dbxef2a7ee redundant comments
varun-edachali-dbx699942d Merge branch 'sea-migration' into exec-phase-sea
varun-edachali-dbxaf8f74e remove fetch phase methods
varun-edachali-dbx5540c5c reduce code repetititon + introduce gaps after multi line pydocs
varun-edachali-dbxefe3881 remove unused imports
varun-edachali-dbx36ab59b move description extraction to helper func
varun-edachali-dbx1d57c99 formatting (black)
varun-edachali-dbxdf6dac2 add more unit tests
varun-edachali-dbxad0e527 streamline unit tests
varun-edachali-dbxed446a0 test getting the list of allowed configurations
varun-edachali-dbx38e4b5c reduce diff
varun-edachali-dbx94879c0 reduce diff
varun-edachali-dbx1809956 house constants in enums for readability and immutability
varun-edachali-dbxda5260c add note on hybrid disposition
varun-edachali-dbx6ec265f [squashed from cloudfetch-sea] introduce external links + arrow funct…
varun-edachali-dbxb2ad5e6 reduce responsibility of Queue
varun-edachali-dbx66d0df6 reduce repetition in arrow tablee creation
varun-edachali-dbxeb7ec80 reduce redundant code in CloudFetchQueue
varun-edachali-dbxa3a8a4a move chunk link progression to separate func
varun-edachali-dbxea79bc8 remove redundant log
varun-edachali-dbx5b49405 improve logging
varun-edachali-dbx015fb76 remove reliance on schema_bytes in SEA
varun-edachali-dbx0385ffb remove redundant note on arrow_schema_bytes
varun-edachali-dbx5380c7a use more fetch methods
varun-edachali-dbx27b781f remove redundant schema_bytes from parent constructor
varun-edachali-dbx238dc0a only call get_chunk_link with non null chunk index
varun-edachali-dbxb3bb07e align SeaResultSet structure with ThriftResultSet
varun-edachali-dbx13e6346 remvoe _fill_result_buffer from SeaResultSet
varun-edachali-dbxf90b4d4 reduce code repetition
varun-edachali-dbx23963fc align SeaResultSet with ext-links-sea
varun-edachali-dbxdd43715 remove redundant methods
varun-edachali-dbx34a7f66 update unit tests
varun-edachali-dbx715cc13 remove accidental venv changes
varun-edachali-dbxfb53dd9 pre-fetch next chunk link on processing current
varun-edachali-dbxd893877 reduce nesting
varun-edachali-dbxa165f1c line break after multi line pydoc
varun-edachali-dbxd68e4ea re-introduce schema_bytes for better abstraction (likely temporary)
varun-edachali-dbxbe17812 Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbxa0705bc add fetchmany_arrow and fetchall_arrow
varun-edachali-dbx1b90c4a Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbxf7c11b9 remove accidental changes in sea backend tests
varun-edachali-dbx349c021 Merge branch 'exec-phase-sea' into metadata-sea
varun-edachali-dbx6229848 remove irrelevant changes
varun-edachali-dbxfd52356 remove un-necessary test changes
varun-edachali-dbx64e58b0 remove un-necessary changes in thrift backend tests
varun-edachali-dbx2903473 remove unimplemented methods test
varun-edachali-dbxb300709 Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbx021ff4c remove unimplemented method tests
varun-edachali-dbxadecd53 modify example scripts to include fetch calls
varun-edachali-dbxd33e5fd Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbxe3cef5c add GetChunksResponse
varun-edachali-dbxac50669 remove changes to sea test
varun-edachali-dbx03cdc4f re-introduce accidentally removed description extraction method
varun-edachali-dbxe1842d8 fix type errors (ssl_options, CHUNK_PATH_WITH_ID..., etc.)
varun-edachali-dbx89a46af access ssl_options through connection
varun-edachali-dbx1d0b28b DEBUG level
varun-edachali-dbxc8820d4 remove explicit multi chunk test
varun-edachali-dbxfe47787 move cloud fetch queues back into utils.py
varun-edachali-dbx74f59b7 remove excess docstrings
varun-edachali-dbx4b456b2 move ThriftCloudFetchQueue above SeaCloudFetchQueue
varun-edachali-dbxbfc1f01 fix sea connector tests
varun-edachali-dbxa4447a1 Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbx4883aff correct patch module path in cloud fetch queue tests
varun-edachali-dbx0a2cdfd remove unimplemented methods test
varun-edachali-dbxcd3378c correct add_link docstring
varun-edachali-dbx90bb09c Merge branch 'sea-migration' into exec-phase-sea
varun-edachali-dbxcd22389 remove invalid import
varun-edachali-dbx82e0f8b Merge branch 'sea-migration' into exec-phase-sea
varun-edachali-dbxe64b81b Merge branch 'exec-phase-sea' into metadata-sea
varun-edachali-dbx27564ca Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbxbc467d1 Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbx5ab9bbe better align queries with JDBC impl
varun-edachali-dbx1ab6e87 line breaks after multi-line PRs
varun-edachali-dbxf469c24 remove unused imports
varun-edachali-dbx68ec65f fix: introduce ExecuteResponse import
varun-edachali-dbxffd478e Merge branch 'sea-migration' into metadata-sea
varun-edachali-dbxf6d873d remove unimplemented metadata methods test, un-necessary imports
varun-edachali-dbx28675f5 introduce unit tests for metadata methods
varun-edachali-dbx3578659 remove verbosity in ResultSetFilter docstring
varun-edachali-dbx8713023 remove un-necessary info in ResultSetFilter docstring
varun-edachali-dbx22dc252 remove explicit type checking, string literals around forward annotat…
varun-edachali-dbx390f592 house SQL commands in constants
varun-edachali-dbxdd7dc6a convert complex types to string if not _use_arrow_native_complex_types
varun-edachali-dbx28308fe Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbx2712d1c introduce unit tests for altered functionality
varun-edachali-dbxdabba55 Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbx48ad7b3 Revert "Merge branch 'fetch-json-inline' into ext-links-sea"
varun-edachali-dbxa1f9b9c reduce verbosity of ResultSetFilter docstring
varun-edachali-dbx984e8ee remove unused imports
varun-edachali-dbx3a999c0 Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbxc313c2b Revert "Merge branch 'fetch-json-inline' into ext-links-sea"
varun-edachali-dbx3bc615e Revert "reduce verbosity of ResultSetFilter docstring"
varun-edachali-dbxb6e1a10 Reapply "Merge branch 'fetch-json-inline' into ext-links-sea"
varun-edachali-dbx2df3d39 Revert "Merge branch 'fetch-json-inline' into ext-links-sea"
varun-edachali-dbx5e75fb5 remove un-necessary filters changes
varun-edachali-dbx20822e4 remove un-necessary backend changes
varun-edachali-dbx802d045 remove constants changes
varun-edachali-dbxf3f795a remove changes in filters tests
varun-edachali-dbxf6c5950 remove unit test backend and JSON queue changes
varun-edachali-dbxd210ccd remove changes in sea result set testing
varun-edachali-dbx22a953e Revert "remove changes in sea result set testing"
varun-edachali-dbx3aed144 Revert "remove unit test backend and JSON queue changes"
varun-edachali-dbx0fe4da4 Revert "remove changes in filters tests"
varun-edachali-dbx0e3c0a1 Revert "remove constants changes"
varun-edachali-dbx93edb93 Revert "remove un-necessary backend changes"
varun-edachali-dbx871a44f Revert "remove un-necessary filters changes"
varun-edachali-dbx0ce144d remove unused imports
varun-edachali-dbx08ca60d Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbx8c5cc77 working version
varun-edachali-dbx7f5c715 adopy _wait_until_command_done
varun-edachali-dbx9ef5fad introduce metadata commands
varun-edachali-dbx44183db use new backend structure
varun-edachali-dbxd59b351 constrain backend diff
varun-edachali-dbx1edc80a remove changes to filters
varun-edachali-dbxf82658a make _parse methods in models internal
varun-edachali-dbx54eb0a4 reduce changes in unit tests
varun-edachali-dbx50cc1e2 run small queries with SEA during integration tests
varun-edachali-dbx242307a run some tests for sea
varun-edachali-dbx8a138e8 allow empty schema bytes for alignment with SEA
varun-edachali-dbx82f9d6b pass is_vl_op to Sea backend ExecuteResponse
varun-edachali-dbx35f1ef0 remove catalog requirement in get_tables
varun-edachali-dbxa515d26 move filters.py to SEA utils
varun-edachali-dbx59b1330 ensure SeaResultSet
varun-edachali-dbx293e356 Merge branch 'sea-migration' into metadata-sea
varun-edachali-dbxdd40beb prevent circular imports
varun-edachali-dbx14057ac remove unused imports
varun-edachali-dbxa4d5bdb remove cast, throw error if not SeaResultSet
varun-edachali-dbx156421a Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbxeb1a9b4 pass param as TSparkParameterValue
varun-edachali-dbx9000666 Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbxa3ca7c7 remove failing test (temp)
varun-edachali-dbx2c22010 remove SeaResultSet type assertion
varun-edachali-dbxc09508e change errors to align with spec, instead of arbitrary ValueError
varun-edachali-dbxe9b1314 make SEA backend methods return SeaResultSet
varun-edachali-dbx8ede414 use spec-aligned Exceptions in SEA backend
varun-edachali-dbx09a1b11 remove defensive row type check
varun-edachali-dbx5e01e7b Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbx3becefe Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbxa026d31 raise ProgrammingError for invalid id
varun-edachali-dbx4446a9e make is_volume_operation strict bool
varun-edachali-dbx138359d remove complex types code
varun-edachali-dbxb99d0c4 Revert "remove complex types code"
varun-edachali-dbx21c389d introduce type conversion for primitive types for JSON + INLINE
varun-edachali-dbx734321a Merge branch 'sea-migration' into fetch-json-inline
varun-edachali-dbx9f0f969 remove SEA running on metadata queries (known failures
varun-edachali-dbx04a1936 remove un-necessary docstrings
varun-edachali-dbx278b8cd align expected types with databricks sdk
varun-edachali-dbx91b7f7f link rest api reference to validate types
varun-edachali-dbx7a5ae13 remove test_catalogs_returns_arrow_table test
varun-edachali-dbxf1776f3 fix fetchall_arrow and fetchmany_arrow
varun-edachali-dbx6143331 remove thrift aligned test_cancel_during_execute from SEA tests
varun-edachali-dbx8949d0c Merge branch 'sea-migration' into fetch-json-inline
varun-edachali-dbx5eaded4 remove un-necessary changes in example scripts
varun-edachali-dbxeeed9a1 remove un-necessary chagnes in example scripts
varun-edachali-dbxf233886 _convert_json_table -> _create_json_table
varun-edachali-dbx68ac437 remove accidentally removed test
varun-edachali-dbx7fd0845 remove new unit tests (to be re-added based on new arch)
varun-edachali-dbxea7ff73 remove changes in sea_result_set functionality (to be re-added)
varun-edachali-dbx563da71 introduce more integration tests
varun-edachali-dbxa018273 remove SEA tests in parameterized queries
varun-edachali-dbxc0e98f4 remove partial parameter fix changes
varun-edachali-dbx7343035 remove un-necessary timestamp tests
varun-edachali-dbxec500b6 slightly stronger typing of _convert_json_types
varun-edachali-dbx0b3e91d stronger typing of json utility func s
varun-edachali-dbx7664e44 stronger typing of fetch*_json
varun-edachali-dbxdb7b8e5 remove unused helper methods in SqlType
varun-edachali-dbxf75f2b5 line breaks after multi line pydocs, remove excess logs
varun-edachali-dbxe2d4ef5 line breaks after multi line pydocs, reduce diff of redundant changes
varun-edachali-dbx21e3078 reduce diff of redundant changes
varun-edachali-dbxbb015e6 mandate ResultData in SeaResultSet constructor
varun-edachali-dbx3944e39 Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbxb3273c7 remove complex type conversion
varun-edachali-dbx38c2b88 correct fetch*_arrow
varun-edachali-dbxb77acbe Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbxfa2359d recover old sea tests
varun-edachali-dbxc07f709 move queue and result set into SEA specific dir
varun-edachali-dbx9e4ef2e pass ssl_options into CloudFetchQueue
varun-edachali-dbxb00c06c reduce diff
varun-edachali-dbx10f55f0 remove redundant conversion.py
varun-edachali-dbxcd119e9 fix type issues
varun-edachali-dbxd79638b ValueError not ProgrammingError
varun-edachali-dbxf84578a reduce diff
varun-edachali-dbxc621c0c introduce SEA cloudfetch e2e tests
varun-edachali-dbx7958cd9 allow empty cloudfetch result
varun-edachali-dbxe2d17ff add unit tests for CloudFetchQueue and SeaResultSet
varun-edachali-dbxd348b35 skip pyarrow dependent tests
varun-edachali-dbx811205e Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbx4bd290e simplify download process: no pre-fetching
varun-edachali-dbxdfbbf79 correct class name in logs
varun-edachali-dbx548e395 introduce column normalisation for SEA metadat queries
varun-edachali-dbx61a9200 add refs, correct COLUMN_COLUMNS mappings
varun-edachali-dbxc5d97bd remove TYPE_NAME mapping
varun-edachali-dbx48bb680 move test-column-mapping into tests/unit/
varun-edachali-dbx8f12c33 formatting (black)
varun-edachali-dbxed4d7ab Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbxa5e9cdf align with old impl
varun-edachali-dbxbe16634 align next_n_rows with prev imple
varun-edachali-dbx6ec8656 align remaining_rows with prev impl
varun-edachali-dbx7ea7b75 remove un-necessary Optional params
varun-edachali-dbx64be07b remove un-necessary changes in thrift field if tests
varun-edachali-dbx165644c remove unused imports
varun-edachali-dbxabef941 run large queries
varun-edachali-dbxe939d9e Merge branch 'ext-links-sea' into sea-norm-cols
varun-edachali-dbx8aa4649 Merge branch 'sea-migration' into sea-norm-cols
varun-edachali-dbxf76d740 remove excess changes
varun-edachali-dbx004bcce test_get_catalogs
varun-edachali-dbxb8e39e2 Merge branch 'sea-migration' into sea-norm-cols
varun-edachali-dbxFile filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
There are no files selected for viewing
110 changes: 110 additions & 0 deletionssrc/databricks/sql/backend/column_mapping.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| """ | ||
| Column name mappings between different backend protocols. | ||
| This module provides mappings between column names returned by different backends | ||
| to ensure a consistent interface for metadata operations. | ||
| """ | ||
| from enum import Enum | ||
| class MetadataOp(Enum): | ||
| """Enum for metadata operations.""" | ||
| CATALOGS = "catalogs" | ||
| SCHEMAS = "schemas" | ||
| TABLES = "tables" | ||
| COLUMNS = "columns" | ||
| # Mappings from column names to standard column names | ||
| # ref: CATALOG_COLUMNS in JDBC: https://github.com/databricks/databricks-jdbc/blob/e3d0d8dad683146a3afc3d501ddf0864ba086309/src/main/java/com/databricks/jdbc/common/MetadataResultConstants.java#L219 | ||
| CATALOG_COLUMNS = { | ||
| "catalog": "TABLE_CAT", | ||
| } | ||
| # ref: SCHEMA_COLUMNS in JDBC: https://github.com/databricks/databricks-jdbc/blob/e3d0d8dad683146a3afc3d501ddf0864ba086309/src/main/java/com/databricks/jdbc/common/MetadataResultConstants.java#L221 | ||
| SCHEMA_COLUMNS = { | ||
| "databaseName": "TABLE_SCHEM", | ||
| "catalogName": "TABLE_CATALOG", | ||
| } | ||
| # ref: TABLE_COLUMNS in JDBC: https://github.com/databricks/databricks-jdbc/blob/e3d0d8dad683146a3afc3d501ddf0864ba086309/src/main/java/com/databricks/jdbc/common/MetadataResultConstants.java#L224 | ||
| TABLE_COLUMNS = { | ||
| "catalogName": "TABLE_CAT", | ||
| "namespace": "TABLE_SCHEM", | ||
| "tableName": "TABLE_NAME", | ||
| "tableType": "TABLE_TYPE", | ||
| "remarks": "REMARKS", | ||
| "TYPE_CATALOG_COLUMN": "TYPE_CAT", | ||
| "TYPE_SCHEMA_COLUMN": "TYPE_SCHEM", | ||
| "TYPE_NAME": "TYPE_NAME", | ||
| "SELF_REFERENCING_COLUMN_NAME": "SELF_REFERENCING_COL_NAME", | ||
| "REF_GENERATION_COLUMN": "REF_GENERATION", | ||
| } | ||
| # ref: COLUMN_COLUMNS in JDBC: https://github.com/databricks/databricks-jdbc/blob/e3d0d8dad683146a3afc3d501ddf0864ba086309/src/main/java/com/databricks/jdbc/common/MetadataResultConstants.java#L192 | ||
| # TYPE_NAME is not included because it is a duplicate target for columnType, and COLUMN_DEF is known to be returned by Thrift. | ||
| # TODO: check if TYPE_NAME is to be returned / also used by Thrift. | ||
| COLUMN_COLUMNS = { | ||
| "catalogName": "TABLE_CAT", | ||
| "namespace": "TABLE_SCHEM", | ||
| "tableName": "TABLE_NAME", | ||
| "col_name": "COLUMN_NAME", | ||
| "dataType": "DATA_TYPE", | ||
| "columnSize": "COLUMN_SIZE", | ||
| "bufferLength": "BUFFER_LENGTH", | ||
| "decimalDigits": "DECIMAL_DIGITS", | ||
| "radix": "NUM_PREC_RADIX", | ||
| "Nullable": "NULLABLE", | ||
| "remarks": "REMARKS", | ||
| "columnType": "COLUMN_DEF", | ||
| "SQLDataType": "SQL_DATA_TYPE", | ||
| "SQLDatetimeSub": "SQL_DATETIME_SUB", | ||
| "CharOctetLength": "CHAR_OCTET_LENGTH", | ||
| "ordinalPosition": "ORDINAL_POSITION", | ||
| "isNullable": "IS_NULLABLE", | ||
| "ScopeCatalog": "SCOPE_CATALOG", | ||
| "ScopeSchema": "SCOPE_SCHEMA", | ||
| "ScopeTable": "SCOPE_TABLE", | ||
| "SourceDataType": "SOURCE_DATA_TYPE", | ||
| "isAutoIncrement": "IS_AUTOINCREMENT", | ||
| "isGenerated": "IS_GENERATEDCOLUMN", | ||
| } | ||
| def normalise_metadata_result(result_set, operation: MetadataOp): | ||
| """ | ||
| Normalise column names in a result set based on the operation type. | ||
| This function modifies the result set in place. | ||
| Args: | ||
| result_set: The result set object to normalise | ||
| operation: The metadata operation (from MetadataOp enum) | ||
| """ | ||
| # Select the appropriate mapping based on the operation | ||
| mapping = None | ||
| if operation == MetadataOp.CATALOGS: | ||
| mapping = CATALOG_COLUMNS | ||
| elif operation == MetadataOp.SCHEMAS: | ||
| mapping = SCHEMA_COLUMNS | ||
| elif operation == MetadataOp.TABLES: | ||
| mapping = TABLE_COLUMNS | ||
| elif operation == MetadataOp.COLUMNS: | ||
| mapping = COLUMN_COLUMNS | ||
| if mapping is None: | ||
| return | ||
| # Normalize column names in the description | ||
| new_description = [] | ||
| for col_desc in result_set.description: | ||
| col_name = col_desc[0] | ||
| if col_name in mapping: | ||
| # Create a new column description tuple with the normalized name | ||
| new_col_desc = (mapping[col_name],) + col_desc[1:] | ||
| new_description.append(new_col_desc) | ||
| else: | ||
| new_description.append(col_desc) | ||
| result_set.description = new_description |
14 changes: 14 additions & 0 deletionssrc/databricks/sql/backend/sea/backend.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
13 changes: 11 additions & 2 deletionstests/e2e/test_driver.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
194 changes: 194 additions & 0 deletionstests/unit/test_column_mapping.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,194 @@ | ||
| """ | ||
| Tests for the column mapping module. | ||
| """ | ||
| import pytest | ||
| from unittest.mock import MagicMock | ||
| from enum import Enum | ||
| from databricks.sql.backend.column_mapping import ( | ||
| normalise_metadata_result, | ||
| MetadataOp, | ||
| ) | ||
| class TestColumnMapping: | ||
| """Tests for the column mapping module.""" | ||
| def test_normalize_metadata_result_catalogs(self): | ||
| """Test normalizing catalog column names.""" | ||
| # Create a mock result set with a description | ||
| mock_result = MagicMock() | ||
| mock_result.description = [ | ||
| ("catalog", "string", None, None, None, None, True), | ||
| ("other_column", "string", None, None, None, None, True), | ||
| ] | ||
| # Normalize the result set | ||
| normalise_metadata_result(mock_result, MetadataOp.CATALOGS) | ||
| # Check that the column names were normalized | ||
| assert mock_result.description[0][0] == "TABLE_CAT" | ||
| assert mock_result.description[1][0] == "other_column" | ||
| def test_normalize_metadata_result_schemas(self): | ||
| """Test normalizing schema column names.""" | ||
| # Create a mock result set with a description | ||
| mock_result = MagicMock() | ||
| mock_result.description = [ | ||
| ("databaseName", "string", None, None, None, None, True), | ||
| ("catalogName", "string", None, None, None, None, True), | ||
| ("other_column", "string", None, None, None, None, True), | ||
| ] | ||
| # Normalize the result set | ||
| normalise_metadata_result(mock_result, MetadataOp.SCHEMAS) | ||
| # Check that the column names were normalized | ||
| assert mock_result.description[0][0] == "TABLE_SCHEM" | ||
| assert mock_result.description[1][0] == "TABLE_CATALOG" | ||
| assert mock_result.description[2][0] == "other_column" | ||
| def test_normalize_metadata_result_tables(self): | ||
| """Test normalizing table column names.""" | ||
| # Create a mock result set with a description | ||
| mock_result = MagicMock() | ||
| mock_result.description = [ | ||
| ("catalogName", "string", None, None, None, None, True), | ||
| ("namespace", "string", None, None, None, None, True), | ||
| ("tableName", "string", None, None, None, None, True), | ||
| ("tableType", "string", None, None, None, None, True), | ||
| ("remarks", "string", None, None, None, None, True), | ||
| ("TYPE_CATALOG_COLUMN", "string", None, None, None, None, True), | ||
| ("TYPE_SCHEMA_COLUMN", "string", None, None, None, None, True), | ||
| ("TYPE_NAME", "string", None, None, None, None, True), | ||
| ("SELF_REFERENCING_COLUMN_NAME", "string", None, None, None, None, True), | ||
| ("REF_GENERATION_COLUMN", "string", None, None, None, None, True), | ||
| ("other_column", "string", None, None, None, None, True), | ||
| ] | ||
| # Normalize the result set | ||
| normalise_metadata_result(mock_result, MetadataOp.TABLES) | ||
| # Check that the column names were normalized | ||
| assert mock_result.description[0][0] == "TABLE_CAT" | ||
| assert mock_result.description[1][0] == "TABLE_SCHEM" | ||
| assert mock_result.description[2][0] == "TABLE_NAME" | ||
| assert mock_result.description[3][0] == "TABLE_TYPE" | ||
| assert mock_result.description[4][0] == "REMARKS" | ||
| assert mock_result.description[5][0] == "TYPE_CAT" | ||
| assert mock_result.description[6][0] == "TYPE_SCHEM" | ||
| assert mock_result.description[7][0] == "TYPE_NAME" | ||
| assert mock_result.description[8][0] == "SELF_REFERENCING_COL_NAME" | ||
| assert mock_result.description[9][0] == "REF_GENERATION" | ||
| assert mock_result.description[10][0] == "other_column" | ||
| def test_normalize_metadata_result_columns(self): | ||
| """Test normalizing column column names.""" | ||
| # Create a mock result set with a description | ||
| mock_result = MagicMock() | ||
| mock_result.description = [ | ||
| ("catalogName", "string", None, None, None, None, True), | ||
| ("namespace", "string", None, None, None, None, True), | ||
| ("tableName", "string", None, None, None, None, True), | ||
| ("col_name", "string", None, None, None, None, True), | ||
| ("dataType", "string", None, None, None, None, True), | ||
| ("columnSize", "string", None, None, None, None, True), | ||
| ("bufferLength", "string", None, None, None, None, True), | ||
| ("decimalDigits", "string", None, None, None, None, True), | ||
| ("radix", "string", None, None, None, None, True), | ||
| ("Nullable", "string", None, None, None, None, True), | ||
| ("remarks", "string", None, None, None, None, True), | ||
| ("columnType", "string", None, None, None, None, True), | ||
| ("SQLDataType", "string", None, None, None, None, True), | ||
| ("SQLDatetimeSub", "string", None, None, None, None, True), | ||
| ("CharOctetLength", "string", None, None, None, None, True), | ||
| ("ordinalPosition", "string", None, None, None, None, True), | ||
| ("isNullable", "string", None, None, None, None, True), | ||
| ("ScopeCatalog", "string", None, None, None, None, True), | ||
| ("ScopeSchema", "string", None, None, None, None, True), | ||
| ("ScopeTable", "string", None, None, None, None, True), | ||
| ("SourceDataType", "string", None, None, None, None, True), | ||
| ("isAutoIncrement", "string", None, None, None, None, True), | ||
| ("isGenerated", "string", None, None, None, None, True), | ||
| ("other_column", "string", None, None, None, None, True), | ||
| ] | ||
| # Normalize the result set | ||
| normalise_metadata_result(mock_result, MetadataOp.COLUMNS) | ||
| # Check that the column names were normalized | ||
| assert mock_result.description[0][0] == "TABLE_CAT" | ||
| assert mock_result.description[1][0] == "TABLE_SCHEM" | ||
| assert mock_result.description[2][0] == "TABLE_NAME" | ||
| assert mock_result.description[3][0] == "COLUMN_NAME" | ||
| assert mock_result.description[4][0] == "DATA_TYPE" | ||
| assert mock_result.description[5][0] == "COLUMN_SIZE" | ||
| assert mock_result.description[6][0] == "BUFFER_LENGTH" | ||
| assert mock_result.description[7][0] == "DECIMAL_DIGITS" | ||
| assert mock_result.description[8][0] == "NUM_PREC_RADIX" | ||
| assert mock_result.description[9][0] == "NULLABLE" | ||
| assert mock_result.description[10][0] == "REMARKS" | ||
| assert mock_result.description[11][0] == "COLUMN_DEF" | ||
| assert mock_result.description[12][0] == "SQL_DATA_TYPE" | ||
| assert mock_result.description[13][0] == "SQL_DATETIME_SUB" | ||
| assert mock_result.description[14][0] == "CHAR_OCTET_LENGTH" | ||
| assert mock_result.description[15][0] == "ORDINAL_POSITION" | ||
| assert mock_result.description[16][0] == "IS_NULLABLE" | ||
| assert mock_result.description[17][0] == "SCOPE_CATALOG" | ||
| assert mock_result.description[18][0] == "SCOPE_SCHEMA" | ||
| assert mock_result.description[19][0] == "SCOPE_TABLE" | ||
| assert mock_result.description[20][0] == "SOURCE_DATA_TYPE" | ||
| assert mock_result.description[21][0] == "IS_AUTOINCREMENT" | ||
| assert mock_result.description[22][0] == "IS_GENERATEDCOLUMN" | ||
| assert mock_result.description[23][0] == "other_column" | ||
| def test_normalize_metadata_result_unknown_operation(self): | ||
| """Test normalizing with an unknown operation type.""" | ||
| # Create a mock result set with a description | ||
| mock_result = MagicMock() | ||
| mock_result.description = [ | ||
| ("column1", "string", None, None, None, None, True), | ||
| ("column2", "string", None, None, None, None, True), | ||
| ] | ||
| # Save the original description | ||
| original_description = mock_result.description.copy() | ||
| # Create a separate enum for testing | ||
| class TestOp(Enum): | ||
| UNKNOWN = "unknown" | ||
| # Normalize the result set with an unknown operation | ||
| normalise_metadata_result(mock_result, TestOp.UNKNOWN) | ||
| # Check that the description was not modified | ||
| assert mock_result.description == original_description | ||
| def test_normalize_metadata_result_preserves_other_fields(self): | ||
| """Test that normalization preserves other fields in the description.""" | ||
| # Create a mock result set with a description | ||
| mock_result = MagicMock() | ||
| mock_result.description = [ | ||
| ( | ||
| "catalog", | ||
| "string", | ||
| "display_size", | ||
| "internal_size", | ||
| "precision", | ||
| "scale", | ||
| True, | ||
| ), | ||
| ] | ||
| # Normalize the result set | ||
| normalise_metadata_result(mock_result, MetadataOp.CATALOGS) | ||
| # Check that the column name was normalized but other fields preserved | ||
| assert mock_result.description[0][0] == "TABLE_CAT" | ||
| assert mock_result.description[0][1] == "string" | ||
| assert mock_result.description[0][2] == "display_size" | ||
| assert mock_result.description[0][3] == "internal_size" | ||
| assert mock_result.description[0][4] == "precision" | ||
| assert mock_result.description[0][5] == "scale" | ||
| assert mock_result.description[0][6] == True |
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.