- Notifications
You must be signed in to change notification settings - Fork126
SEA: re-fetch links in case of expiry#635
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Draft
varun-edachali-dbx wants to merge382 commits intoless-defensive-downloadChoose a base branch fromsea-link-expiry
base:less-defensive-download
Could not load branches
Branch not found:{{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline, and old review comments may become outdated.
Uh oh!
There was an error while loading.Please reload this page.
Draft
Changes fromall commits
Commits
Show all changes
382 commits Select commitHold shift + click to select a range
5380c7a use more fetch methods
varun-edachali-dbx27b781f remove redundant schema_bytes from parent constructor
varun-edachali-dbx238dc0a only call get_chunk_link with non null chunk index
varun-edachali-dbxb3bb07e align SeaResultSet structure with ThriftResultSet
varun-edachali-dbx13e6346 remvoe _fill_result_buffer from SeaResultSet
varun-edachali-dbxf90b4d4 reduce code repetition
varun-edachali-dbx23963fc align SeaResultSet with ext-links-sea
varun-edachali-dbxdd43715 remove redundant methods
varun-edachali-dbx34a7f66 update unit tests
varun-edachali-dbx715cc13 remove accidental venv changes
varun-edachali-dbxfb53dd9 pre-fetch next chunk link on processing current
varun-edachali-dbxd893877 reduce nesting
varun-edachali-dbxa165f1c line break after multi line pydoc
varun-edachali-dbxd68e4ea re-introduce schema_bytes for better abstraction (likely temporary)
varun-edachali-dbxbe17812 Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbxa0705bc add fetchmany_arrow and fetchall_arrow
varun-edachali-dbx1b90c4a Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbxf7c11b9 remove accidental changes in sea backend tests
varun-edachali-dbx349c021 Merge branch 'exec-phase-sea' into metadata-sea
varun-edachali-dbx6229848 remove irrelevant changes
varun-edachali-dbxfd52356 remove un-necessary test changes
varun-edachali-dbx64e58b0 remove un-necessary changes in thrift backend tests
varun-edachali-dbx2903473 remove unimplemented methods test
varun-edachali-dbxb300709 Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbx021ff4c remove unimplemented method tests
varun-edachali-dbxadecd53 modify example scripts to include fetch calls
varun-edachali-dbxd33e5fd Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbxe3cef5c add GetChunksResponse
varun-edachali-dbxac50669 remove changes to sea test
varun-edachali-dbx03cdc4f re-introduce accidentally removed description extraction method
varun-edachali-dbxe1842d8 fix type errors (ssl_options, CHUNK_PATH_WITH_ID..., etc.)
varun-edachali-dbx89a46af access ssl_options through connection
varun-edachali-dbx1d0b28b DEBUG level
varun-edachali-dbxc8820d4 remove explicit multi chunk test
varun-edachali-dbxfe47787 move cloud fetch queues back into utils.py
varun-edachali-dbx74f59b7 remove excess docstrings
varun-edachali-dbx4b456b2 move ThriftCloudFetchQueue above SeaCloudFetchQueue
varun-edachali-dbxbfc1f01 fix sea connector tests
varun-edachali-dbxa4447a1 Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbx4883aff correct patch module path in cloud fetch queue tests
varun-edachali-dbx0a2cdfd remove unimplemented methods test
varun-edachali-dbxcd3378c correct add_link docstring
varun-edachali-dbx90bb09c Merge branch 'sea-migration' into exec-phase-sea
varun-edachali-dbxcd22389 remove invalid import
varun-edachali-dbx82e0f8b Merge branch 'sea-migration' into exec-phase-sea
varun-edachali-dbxe64b81b Merge branch 'exec-phase-sea' into metadata-sea
varun-edachali-dbx27564ca Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbxbc467d1 Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbx5ab9bbe better align queries with JDBC impl
varun-edachali-dbx1ab6e87 line breaks after multi-line PRs
varun-edachali-dbxf469c24 remove unused imports
varun-edachali-dbx68ec65f fix: introduce ExecuteResponse import
varun-edachali-dbxffd478e Merge branch 'sea-migration' into metadata-sea
varun-edachali-dbxf6d873d remove unimplemented metadata methods test, un-necessary imports
varun-edachali-dbx28675f5 introduce unit tests for metadata methods
varun-edachali-dbx3578659 remove verbosity in ResultSetFilter docstring
varun-edachali-dbx8713023 remove un-necessary info in ResultSetFilter docstring
varun-edachali-dbx22dc252 remove explicit type checking, string literals around forward annotat…
varun-edachali-dbx390f592 house SQL commands in constants
varun-edachali-dbxdd7dc6a convert complex types to string if not _use_arrow_native_complex_types
varun-edachali-dbx28308fe Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbx2712d1c introduce unit tests for altered functionality
varun-edachali-dbxdabba55 Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbx48ad7b3 Revert "Merge branch 'fetch-json-inline' into ext-links-sea"
varun-edachali-dbxa1f9b9c reduce verbosity of ResultSetFilter docstring
varun-edachali-dbx984e8ee remove unused imports
varun-edachali-dbx3a999c0 Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbxc313c2b Revert "Merge branch 'fetch-json-inline' into ext-links-sea"
varun-edachali-dbx3bc615e Revert "reduce verbosity of ResultSetFilter docstring"
varun-edachali-dbxb6e1a10 Reapply "Merge branch 'fetch-json-inline' into ext-links-sea"
varun-edachali-dbx2df3d39 Revert "Merge branch 'fetch-json-inline' into ext-links-sea"
varun-edachali-dbx5e75fb5 remove un-necessary filters changes
varun-edachali-dbx20822e4 remove un-necessary backend changes
varun-edachali-dbx802d045 remove constants changes
varun-edachali-dbxf3f795a remove changes in filters tests
varun-edachali-dbxf6c5950 remove unit test backend and JSON queue changes
varun-edachali-dbxd210ccd remove changes in sea result set testing
varun-edachali-dbx22a953e Revert "remove changes in sea result set testing"
varun-edachali-dbx3aed144 Revert "remove unit test backend and JSON queue changes"
varun-edachali-dbx0fe4da4 Revert "remove changes in filters tests"
varun-edachali-dbx0e3c0a1 Revert "remove constants changes"
varun-edachali-dbx93edb93 Revert "remove un-necessary backend changes"
varun-edachali-dbx871a44f Revert "remove un-necessary filters changes"
varun-edachali-dbx0ce144d remove unused imports
varun-edachali-dbx08ca60d Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbx8c5cc77 working version
varun-edachali-dbx7f5c715 adopy _wait_until_command_done
varun-edachali-dbx9ef5fad introduce metadata commands
varun-edachali-dbx44183db use new backend structure
varun-edachali-dbxd59b351 constrain backend diff
varun-edachali-dbx1edc80a remove changes to filters
varun-edachali-dbxf82658a make _parse methods in models internal
varun-edachali-dbx54eb0a4 reduce changes in unit tests
varun-edachali-dbx50cc1e2 run small queries with SEA during integration tests
varun-edachali-dbx242307a run some tests for sea
varun-edachali-dbx8a138e8 allow empty schema bytes for alignment with SEA
varun-edachali-dbx82f9d6b pass is_vl_op to Sea backend ExecuteResponse
varun-edachali-dbx35f1ef0 remove catalog requirement in get_tables
varun-edachali-dbxa515d26 move filters.py to SEA utils
varun-edachali-dbx59b1330 ensure SeaResultSet
varun-edachali-dbx293e356 Merge branch 'sea-migration' into metadata-sea
varun-edachali-dbxdd40beb prevent circular imports
varun-edachali-dbx14057ac remove unused imports
varun-edachali-dbxa4d5bdb remove cast, throw error if not SeaResultSet
varun-edachali-dbx156421a Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbxeb1a9b4 pass param as TSparkParameterValue
varun-edachali-dbx9000666 Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbxa3ca7c7 remove failing test (temp)
varun-edachali-dbx2c22010 remove SeaResultSet type assertion
varun-edachali-dbxc09508e change errors to align with spec, instead of arbitrary ValueError
varun-edachali-dbxe9b1314 make SEA backend methods return SeaResultSet
varun-edachali-dbx8ede414 use spec-aligned Exceptions in SEA backend
varun-edachali-dbx09a1b11 remove defensive row type check
varun-edachali-dbx5e01e7b Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbx3becefe Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbxa026d31 raise ProgrammingError for invalid id
varun-edachali-dbx4446a9e make is_volume_operation strict bool
varun-edachali-dbx138359d remove complex types code
varun-edachali-dbxb99d0c4 Revert "remove complex types code"
varun-edachali-dbx21c389d introduce type conversion for primitive types for JSON + INLINE
varun-edachali-dbx734321a Merge branch 'sea-migration' into fetch-json-inline
varun-edachali-dbx9f0f969 remove SEA running on metadata queries (known failures
varun-edachali-dbx04a1936 remove un-necessary docstrings
varun-edachali-dbx278b8cd align expected types with databricks sdk
varun-edachali-dbx91b7f7f link rest api reference to validate types
varun-edachali-dbx7a5ae13 remove test_catalogs_returns_arrow_table test
varun-edachali-dbxf1776f3 fix fetchall_arrow and fetchmany_arrow
varun-edachali-dbx6143331 remove thrift aligned test_cancel_during_execute from SEA tests
varun-edachali-dbx8949d0c Merge branch 'sea-migration' into fetch-json-inline
varun-edachali-dbx5eaded4 remove un-necessary changes in example scripts
varun-edachali-dbxeeed9a1 remove un-necessary chagnes in example scripts
varun-edachali-dbxf233886 _convert_json_table -> _create_json_table
varun-edachali-dbx68ac437 remove accidentally removed test
varun-edachali-dbx7fd0845 remove new unit tests (to be re-added based on new arch)
varun-edachali-dbxea7ff73 remove changes in sea_result_set functionality (to be re-added)
varun-edachali-dbx563da71 introduce more integration tests
varun-edachali-dbxa018273 remove SEA tests in parameterized queries
varun-edachali-dbxc0e98f4 remove partial parameter fix changes
varun-edachali-dbx7343035 remove un-necessary timestamp tests
varun-edachali-dbxec500b6 slightly stronger typing of _convert_json_types
varun-edachali-dbx0b3e91d stronger typing of json utility func s
varun-edachali-dbx7664e44 stronger typing of fetch*_json
varun-edachali-dbxdb7b8e5 remove unused helper methods in SqlType
varun-edachali-dbxf75f2b5 line breaks after multi line pydocs, remove excess logs
varun-edachali-dbxe2d4ef5 line breaks after multi line pydocs, reduce diff of redundant changes
varun-edachali-dbx21e3078 reduce diff of redundant changes
varun-edachali-dbxbb015e6 mandate ResultData in SeaResultSet constructor
varun-edachali-dbx3944e39 Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbxb3273c7 remove complex type conversion
varun-edachali-dbx38c2b88 correct fetch*_arrow
varun-edachali-dbxb77acbe Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbxfa2359d recover old sea tests
varun-edachali-dbxc07f709 move queue and result set into SEA specific dir
varun-edachali-dbx9e4ef2e pass ssl_options into CloudFetchQueue
varun-edachali-dbxb00c06c reduce diff
varun-edachali-dbx10f55f0 remove redundant conversion.py
varun-edachali-dbxcd119e9 fix type issues
varun-edachali-dbxd79638b ValueError not ProgrammingError
varun-edachali-dbxf84578a reduce diff
varun-edachali-dbxc621c0c introduce SEA cloudfetch e2e tests
varun-edachali-dbx7958cd9 allow empty cloudfetch result
varun-edachali-dbxe2d17ff add unit tests for CloudFetchQueue and SeaResultSet
varun-edachali-dbxd348b35 skip pyarrow dependent tests
varun-edachali-dbx811205e Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbx4bd290e simplify download process: no pre-fetching
varun-edachali-dbxdfbbf79 correct class name in logs
varun-edachali-dbxed4d7ab Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbxa5e9cdf align with old impl
varun-edachali-dbxbe16634 align next_n_rows with prev imple
varun-edachali-dbx6ec8656 align remaining_rows with prev impl
varun-edachali-dbx7ea7b75 remove un-necessary Optional params
varun-edachali-dbx64be07b remove un-necessary changes in thrift field if tests
varun-edachali-dbx165644c remove unused imports
varun-edachali-dbx4a9ba21 init hybrid
varun-edachali-dbxabef941 run large queries
varun-edachali-dbxe74ccd1 hybrid disposition
varun-edachali-dbx06a5d54 remove un-ncessary log
varun-edachali-dbx9cf6e02 formatting (black)
varun-edachali-dbxb5578ca remove redundant tests
varun-edachali-dbx7f67cfc multi frame decompression of lz4
varun-edachali-dbx2202057 ensure no compression (temp)
varun-edachali-dbx92fe5fd introduce separate link fetcher
varun-edachali-dbx54b43fa log time to create table
varun-edachali-dbx2750c5e add chunk index to table creation time log
varun-edachali-dbx239cd8d remove custom multi-frame decompressor for lz4
varun-edachali-dbx57ebf06 remove excess logs
varun-edachali-dbx6894b4c remove redundant tests (temp)
varun-edachali-dbx4abb3ad add link to download manager before notifying consumer
varun-edachali-dbxfce324b move link fetching immediately before table creation so link expiry i…
varun-edachali-dbx671dbca Merge branch 'ext-links-sea' into sea-hybrid
varun-edachali-dbx7257168 Merge branch 'sea-hybrid' into sea-decouple-link-fetch
varun-edachali-dbx9590af7 resolve merge artifacts
varun-edachali-dbx82fc0b6 remove redundant methods
varun-edachali-dbx39469fa Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbxb2d1579 formatting (black)
varun-edachali-dbxbd51b1c introduce callback to handle link expiry
varun-edachali-dbx077a71c fix types
varun-edachali-dbx7985639 fix param type in unit tests
varun-edachali-dbx0e1abfa Merge branch 'ext-links-sea' into sea-hybrid
varun-edachali-dbxa736bb4 Merge branch 'sea-hybrid' into sea-decouple-link-fetch
varun-edachali-dbx0868fe3 formatting + minor type fixes
varun-edachali-dbx3f42a03 Revert "introduce callback to handle link expiry"
varun-edachali-dbxd038d84 remove unused callback (to be introduced later)
varun-edachali-dbxdfc32b4 Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbx0a0643b correct param extraction
varun-edachali-dbxf374f5f init sea link retry func
varun-edachali-dbx0829e67 correct unit tests for download management
varun-edachali-dbxa18be78 create strongly typed Future for download tasks
varun-edachali-dbxf7fd1d9 remove common constructor for databricks client abc
varun-edachali-dbx510b0a3 make SEA Http Client instance a private member
varun-edachali-dbxdd2864b make GetChunksResponse model more robust
varun-edachali-dbxc32b281 add link to doc of GetChunk response model
varun-edachali-dbx0b1eba5 pass result_data instead of "initial links" into SeaCloudFetchQueue
varun-edachali-dbx777f7c1 move download_manager init into parent CloudFetchQueue
varun-edachali-dbx130b0d3 raise ServerOperationError for no 0th chunk
varun-edachali-dbx1920375 unused iports
varun-edachali-dbxd882c6e Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbx5a43686 return None in case of empty respose
varun-edachali-dbx28c6bb1 ensure table is empty on no initial link s
varun-edachali-dbx4dd9434 Merge branch 'ext-links-sea' into sea-hybrid
varun-edachali-dbx5791745 account for total chunk count
varun-edachali-dbx2701e5d iterate by chunk index instead of link
varun-edachali-dbxa618115 Merge branch 'sea-hybrid' into sea-decouple-link-fetch
varun-edachali-dbxeb24409 Merge branch 'sea-migration' into sea-decouple-link-fetch
varun-edachali-dbxf56497f make LinkFetcher convert link static
varun-edachali-dbx76fd190 add helper for link addition, check for edge case to prevent inf wait
varun-edachali-dbxab2c2c4 add unit tests for LinkFetcher
varun-edachali-dbx3af2333 remove un-necessary download manager check
varun-edachali-dbx480528e remove un-necessary string literals around param type
varun-edachali-dbxfba1812 remove duplicate download_manager init
varun-edachali-dbxf2ea729 Merge branch 'sea-migration' into sea-decouple-link-fetch
varun-edachali-dbx23dba80 account for empty response in LinkFetcher init
varun-edachali-dbx26aa45c make get_chunk_link return mandatory ExternalLink
varun-edachali-dbx9fd57d8 set shutdown_event instead of breaking on completion so get_chunk_lin…
varun-edachali-dbxdd009df docstrings, logging, pydoc
varun-edachali-dbx7c360d1 use total_chunk_cound > 0
varun-edachali-dbx6b1b972 clarify that link has already been submitted on getting row_offset
varun-edachali-dbx2ca7c2b return None for out of range
varun-edachali-dbx00db613 default link_fetcher to None
varun-edachali-dbx47bb758 Merge branch 'sea-decouple-link-fetch' into sea-link-expiry
varun-edachali-dbx2cd802e reduce repeated init
varun-edachali-dbx0475abf Merge branch 'sea-migration' into sea-link-expiry
varun-edachali-dbxb5fbeb6 Merge branch 'less-defensive-download' into sea-link-expiry
varun-edachali-dbxb114599 add_link -> add_links for less heavy link additions
varun-edachali-dbx38c9343 Merge branch 'less-defensive-download' into sea-link-expiry
varun-edachali-dbx92821e7 move param order
varun-edachali-dbx63deb04 type issues
varun-edachali-dbxade7054 Merge branch 'less-defensive-download' into sea-link-expiry
varun-edachali-dbxfefb5ec Merge branch 'less-defensive-download' into sea-link-expiry
varun-edachali-dbxac04e66 introduce create_download_handler
varun-edachali-dbxFile filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
There are no files selected for viewing
12 changes: 6 additions & 6 deletionsexamples/experimental/tests/test_sea_sync_query.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
35 changes: 32 additions & 3 deletionssrc/databricks/sql/backend/sea/queue.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
89 changes: 74 additions & 15 deletionssrc/databricks/sql/cloudfetch/download_manager.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
12 changes: 6 additions & 6 deletionssrc/databricks/sql/cloudfetch/downloader.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
12 changes: 10 additions & 2 deletionssrc/databricks/sql/utils.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
4 changes: 4 additions & 0 deletionstests/unit/test_download_manager.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.