- Notifications
You must be signed in to change notification settings - Fork126
Normalise type code#652
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Uh oh!
There was an error while loading.Please reload this page.
Merged
Changes fromall commits
Commits
Show all changes
84 commits Select commitHold shift + click to select a range
5bf5d4c Separate Session related functionality from Connection class (#571)
varun-edachali-dbx400a8bd Introduce Backend Interface (DatabricksClient) (#573)
varun-edachali-dbx3c78ed7 Implement ResultSet Abstraction (backend interfaces for fetch phase) …
varun-edachali-dbx9625229 Introduce Sea HTTP Client and test script (#583)
varun-edachali-dbx0887bc1 Introduce `SeaDatabricksClient` (Session Implementation) (#582)
varun-edachali-dbx6d63df0 Normalise Execution Response (clean backend interfaces) (#587)
varun-edachali-dbxba8d9fd Introduce models for `SeaDatabricksClient` (#595)
varun-edachali-dbxbb3f15a Introduce preliminary SEA Result Set (#588)
varun-edachali-dbx19f1fae Merge branch 'main' into sea-migration
varun-edachali-dbx6c5ba6d remove invalid ExecuteResponse import
varun-edachali-dbx5e5147b Separate Session related functionality from Connection class (#571)
varun-edachali-dbx57370b3 Introduce Backend Interface (DatabricksClient) (#573)
varun-edachali-dbx75752bf Implement ResultSet Abstraction (backend interfaces for fetch phase) …
varun-edachali-dbx450b80d remove un-necessary initialisation assertions
varun-edachali-dbxa926f02 remove un-necessary line break s
varun-edachali-dbx55ad001 more un-necessary line breaks
varun-edachali-dbxfa15730 constrain diff of test_closing_connection_closes_commands
varun-edachali-dbx019c7fb reduce diff of test_closing_connection_closes_commands
varun-edachali-dbx726abe7 use pytest-like assertions for test_closing_connection_closes_commands
varun-edachali-dbxbf6d41c ensure command_id is not None
varun-edachali-dbx5afa733 line breaks after multi-line pyfocs
varun-edachali-dbxe3dfd36 ensure non null operationHandle for commandId creation
varun-edachali-dbx63360b3 use command_id methods instead of explicit guid_to_hex_id conversion
varun-edachali-dbx13ffb8d remove un-necessary artifacts in test_session, add back assertion
varun-edachali-dbxa74d279 Implement SeaDatabricksClient (Complete Execution Spec) (#590)
varun-edachali-dbxd759050 add from __future__ import annotations to remove string literals arou…
varun-edachali-dbx1e21434 move docstring of DatabricksClient within class
varun-edachali-dbxcd4015b move ThriftResultSet import to top of file
varun-edachali-dbxed8b610 make backend/utils __init__ file empty
varun-edachali-dbx94d951e use from __future__ import annotations to remove string literals arou…
varun-edachali-dbxc20058e use lazy logging
varun-edachali-dbxfe3acb1 replace getters with property tag
varun-edachali-dbx9fb6a76 Merge branch 'main' into backend-refactors
varun-edachali-dbx61dfc4d set active_command_id to None, not active_op_handle
varun-edachali-dbx64fb9b2 align test_session with pytest instead of unittest
varun-edachali-dbxcbf63f9 Merge branch 'main' into sea-migration
varun-edachali-dbx59b4825 remove duplicate test, correct active_command_id attribute
varun-edachali-dbxe380654 SeaDatabricksClient: Add Metadata Commands (#593)
varun-edachali-dbx677a7b0 SEA volume operations fix: assign `manifest.is_volume_operation` to `…
varun-edachali-dbx45585d4 Introduce manual SEA test scripts for Exec Phase (#589)
varun-edachali-dbx70c7dc8 Complete Fetch Phase (for `INLINE` disposition and `JSON_ARRAY` forma…
varun-edachali-dbxabf9aab Merge branch 'main' into sea-migration
varun-edachali-dbx9b4b606 Merge branch 'main' into backend-refactors
varun-edachali-dbx4f11ff0 Introduce `row_limit` param (#607)
varun-edachali-dbx45f5c26 Merge branch 'main' into backend-refactors
varun-edachali-dbx2c9368a formatting (black)
varun-edachali-dbx9b1b1f5 remove repetition from Session.__init__
varun-edachali-dbx77e23d3 Merge branch 'backend-refactors' into sea-migration
varun-edachali-dbx3bd3aef fix merge artifacts
varun-edachali-dbx6d4701f correct patch paths
varun-edachali-dbxdc1cb6d fix type issues
varun-edachali-dbx5d04cd0 Merge branch 'main' into sea-migration
varun-edachali-dbx922c448 explicitly close result queue
varun-edachali-dbx1a0575a Complete Fetch Phase (`EXTERNAL_LINKS` disposition and `ARROW` format…
varun-edachali-dbxc07beb1 SEA Session Configuration Fix: Explicitly convert values to `str` (#…
varun-edachali-dbx640cc82 SEA: add support for `Hybrid` disposition (#631)
varun-edachali-dbx8fbca9d SEA: Reduce network calls for synchronous commands (#633)
varun-edachali-dbx806e5f5 SEA: Decouple Link Fetching (#632)
varun-edachali-dbxb57c3f3 Chunk download latency (#634)
saishreeeeeef5836b acquire lock before notif + formatting (black)
varun-edachali-dbx4fd2a3f Merge branch 'main' into sea-migration
varun-edachali-dbx26f8947 fix imports
varun-edachali-dbx2d44596 add get_chunk_link s
varun-edachali-dbx99e7435 simplify description extraction
varun-edachali-dbx54ec080 pass session_id_hex to ThriftResultSet
varun-edachali-dbxf9f9f31 revert to main's extract description
varun-edachali-dbx51cef2b validate row count for sync query tests as well
varun-edachali-dbx387102d guid_hex -> hex_guid
varun-edachali-dbxd53d1ea reduce diff
varun-edachali-dbxc7810aa reduce diff
varun-edachali-dbxb3072bd reduce diff
varun-edachali-dbx8be5264 set .value in compression
varun-edachali-dbx80692e3 reduce diff
varun-edachali-dbx83e45ae is_direct_results -> has_more_rows
varun-edachali-dbxa636bdc type normalisation for SEA
varun-edachali-dbxc23d540 Merge branch 'main' into normalise-code
varun-edachali-dbx152f565 fix type codes by using Thrift ttypes
varun-edachali-dbxbd2e83b remove excess call to session_id_hex
varun-edachali-dbxb4f9e2d remove session_id_hex args
varun-edachali-dbx533f74b document disparity mapping
varun-edachali-dbx29e4846 ensure valid interval return
varun-edachali-dbx6481851 more verbose logging for type conversion fail
varun-edachali-dbx1332a43 Revert "more verbose logging for type conversion fail"
varun-edachali-dbx44be1ac stop throwing errors from type conversion
varun-edachali-dbxFile filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
There are no files selected for viewing
6 changes: 6 additions & 0 deletionssrc/databricks/sql/backend/sea/backend.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
19 changes: 9 additions & 10 deletionssrc/databricks/sql/backend/sea/result_set.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
75 changes: 44 additions & 31 deletionssrc/databricks/sql/backend/sea/utils/conversion.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
50 changes: 50 additions & 0 deletionssrc/databricks/sql/backend/sea/utils/normalize.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| """ | ||
| Type normalization utilities for SEA backend. | ||
| This module provides functionality to normalize SEA type names to match | ||
| Thrift type naming conventions. | ||
| """ | ||
| from typing import Dict, Any | ||
| # SEA types that need to be translated to Thrift types | ||
| # The list of all SEA types is available in the REST reference at: | ||
| # https://docs.databricks.com/api/workspace/statementexecution/executestatement | ||
| # The list of all Thrift types can be found in the ttypes.TTypeId definition | ||
| # The SEA types that do not align with Thrift are explicitly mapped below | ||
| SEA_TO_THRIFT_TYPE_MAP = { | ||
| "BYTE": "TINYINT", | ||
| "SHORT": "SMALLINT", | ||
| "LONG": "BIGINT", | ||
| "INTERVAL": "INTERVAL", # Default mapping, will be overridden if type_interval_type is present | ||
| } | ||
| def normalize_sea_type_to_thrift(type_name: str, col_data: Dict[str, Any]) -> str: | ||
| """ | ||
| Normalize SEA type names to match Thrift type naming conventions. | ||
| Args: | ||
| type_name: The type name from SEA (e.g., "BYTE", "LONG", "INTERVAL") | ||
| col_data: The full column data dictionary from manifest (for accessing type_interval_type) | ||
| Returns: | ||
| Normalized type name matching Thrift conventions | ||
| """ | ||
| # Early return if type doesn't need mapping | ||
| if type_name not in SEA_TO_THRIFT_TYPE_MAP: | ||
| return type_name | ||
| normalized_type = SEA_TO_THRIFT_TYPE_MAP[type_name] | ||
| # Special handling for interval types | ||
| if type_name == "INTERVAL": | ||
| type_interval_type = col_data.get("type_interval_type") | ||
| if type_interval_type: | ||
| return ( | ||
| "INTERVAL_YEAR_MONTH" | ||
| if any(t in type_interval_type.upper() for t in ["YEAR", "MONTH"]) | ||
| else "INTERVAL_DAY_TIME" | ||
| ) | ||
| return normalized_type |
4 changes: 1 addition & 3 deletionstests/unit/test_client.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
6 changes: 4 additions & 2 deletionstests/unit/test_downloader.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
60 changes: 60 additions & 0 deletionstests/unit/test_sea_backend.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.