- Notifications
You must be signed in to change notification settings - Fork126
Chunk download latency#634
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Uh oh!
There was an error while loading.Please reload this page.
@jayantsing-db can you take a look, this PR adds latency logs (merges into sea-migration) as there are some refactorings that this PR leverages |
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
LGTM. Thanks for making the changes
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
LGTM
b57c3f3 intosea-migrationUh oh!
There was an error while loading.Please reload this page.
This reverts commitb57c3f3.
* allow empty schema bytes for alignment with SEASigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass is_vl_op to Sea backend ExecuteResponseSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove catalog requirement in get_tablesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* move filters.py to SEA utilsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* ensure SeaResultSetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* prevent circular importsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove unused importsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove cast, throw error if not SeaResultSetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass param as TSparkParameterValueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove failing test (temp)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove SeaResultSet type assertionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* change errors to align with spec, instead of arbitrary ValueErrorSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make SEA backend methods return SeaResultSetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* use spec-aligned Exceptions in SEA backendSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove defensive row type checkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* raise ProgrammingError for invalid idSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make is_volume_operation strict boolSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove complex types codeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* Revert "remove complex types code"This reverts commit138359d.* introduce type conversion for primitive types for JSON + INLINESigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove SEA running on metadata queries (known failuresSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary docstringsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align expected types with databricks sdkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* link rest api reference to validate typesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove test_catalogs_returns_arrow_table testmetadata commands not expected to passSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix fetchall_arrow and fetchmany_arrowSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove thrift aligned test_cancel_during_execute from SEA testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary changes in example scriptsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary chagnes in example scriptsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* _convert_json_table -> _create_json_tableSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove accidentally removed testSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove new unit tests (to be re-added based on new arch)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove changes in sea_result_set functionality (to be re-added)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* introduce more integration testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove SEA tests in parameterized queriesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove partial parameter fix changesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary timestamp tests(pass with minor disparity)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* slightly stronger typing of _convert_json_typesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* stronger typing of json utility func sSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* stronger typing of fetch*_jsonSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove unused helper methods in SqlTypeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* line breaks after multi line pydocs, remove excess logsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* line breaks after multi line pydocs, reduce diff of redundant changesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diff of redundant changesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* mandate ResultData in SeaResultSet constructorSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove complex type conversionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* correct fetch*_arrowSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* recover old sea testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* move queue and result set into SEA specific dirSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass ssl_options into CloudFetchQueueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant conversion.pySigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix type issuesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* ValueError not ProgrammingErrorSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* introduce SEA cloudfetch e2e testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* allow empty cloudfetch resultSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add unit tests for CloudFetchQueue and SeaResultSetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* skip pyarrow dependent testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* simplify download process: no pre-fetchingSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* correct class name in logsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align with old implSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align next_n_rows with prev impleSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align remaining_rows with prev implSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary Optional paramsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary changes in thrift field if testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove unused importsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* init hybrid* run large queriesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* hybrid dispositionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-ncessary logSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* formatting (black)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* multi frame decompression of lz4Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* ensure no compression (temp)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* introduce separate link fetcherSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* log time to create tableSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add chunk index to table creation time logSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove custom multi-frame decompressor for lz4Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove excess logs* remove redundant tests (temp)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add link to download manager before notifying consumerSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* move link fetching immediately before table creation so link expiry is not an issueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* resolve merge artifactsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant methodsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* formatting (black)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* introduce callback to handle link expirySigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix typesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix param type in unit testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* formatting + minor type fixesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* Revert "introduce callback to handle link expiry"This reverts commitbd51b1c.* remove unused callback (to be introduced later)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* correct param extractionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove common constructor for databricks client abcSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make SEA Http Client instance a private memberSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make GetChunksResponse model more robustSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add link to doc of GetChunk response modelSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass result_data instead of "initial links" into SeaCloudFetchQueueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* move download_manager init into parent CloudFetchQueueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* raise ServerOperationError for no 0th chunkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* unused iportsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* return None in case of empty resposeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* ensure table is empty on no initial link sSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* account for total chunk countSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* iterate by chunk index instead of linkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make LinkFetcher convert link staticSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add helper for link addition, check for edge case to prevent inf waitSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add unit tests for LinkFetcherSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary download manager checkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary string literals around param typeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove duplicate download_manager initSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* account for empty response in LinkFetcher initSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make get_chunk_link return mandatory ExternalLinkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* set shutdown_event instead of breaking on completion so get_chunk_link is informedSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* docstrings, logging, pydocSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* use total_chunk_cound > 0Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* clarify that link has already been submitted on getting row_offsetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* return None for out of rangeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* default link_fetcher to NoneSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>---------Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* Chunk download latency (#634)* chunk download latencySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* test fixesSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* sea-migration static type checking fixesSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* check types fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* fix type issuesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* type fix revertSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* -Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* statement id in get metadata functionsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed result set extractorSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* databricks client typeSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* remove defaults, fix chunk idSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added statement type to command idSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* check types fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* renamed chunk_id to num_downloaded_chunksSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* set statement type to query for chunk downloadSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* comment fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed dup check for trowsetSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>---------Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* acquire lock before notif + formatting (black)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix importsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add get_chunk_link sSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* simplify description extractionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass session_id_hex to ThriftResultSetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* revert to main's extract descriptionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* validate row count for sync query tests as wellSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* guid_hex -> hex_guidSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* set .value in compressionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* formatting (black)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant testSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* move extra_params to the backSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* is_direct_results -> has_more_rowsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* Revert "is_direct_results -> has_more_rows"This reverts commit0e87374.* stop passing session_id_hexSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant commentSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add extra_params paramSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass extra_params into test_...unset...Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove excess session_id_heSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce changes in DatabricksRetryPolicySigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diff in DatabricksRetryPolicySigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* simple comments on proxy settingSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* link docs for getproxies)(Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* rename proxy specific attrs with proxy prefixSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>---------Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* remove redundant conversion.pySigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix type issuesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* ValueError not ProgrammingErrorSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* introduce SEA cloudfetch e2e testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* allow empty cloudfetch resultSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add unit tests for CloudFetchQueue and SeaResultSetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* skip pyarrow dependent testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* simplify download process: no pre-fetchingSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* correct class name in logsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align with old implSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align next_n_rows with prev impleSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align remaining_rows with prev implSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary Optional paramsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary changes in thrift field if testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove unused importsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* init hybrid* run large queriesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* hybrid dispositionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-ncessary logSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* formatting (black)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* multi frame decompression of lz4Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* ensure no compression (temp)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* introduce separate link fetcherSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* log time to create tableSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add chunk index to table creation time logSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove custom multi-frame decompressor for lz4Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove excess logs* remove redundant tests (temp)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add link to download manager before notifying consumerSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* move link fetching immediately before table creation so link expiry is not an issueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* resolve merge artifactsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant methodsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* formatting (black)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* introduce callback to handle link expirySigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix typesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix param type in unit testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* formatting + minor type fixesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* Revert "introduce callback to handle link expiry"This reverts commitbd51b1c.* remove unused callback (to be introduced later)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* correct param extractionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove common constructor for databricks client abcSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make SEA Http Client instance a private memberSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make GetChunksResponse model more robustSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add link to doc of GetChunk response modelSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass result_data instead of "initial links" into SeaCloudFetchQueueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* move download_manager init into parent CloudFetchQueueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* raise ServerOperationError for no 0th chunkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* unused iportsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* return None in case of empty resposeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* ensure table is empty on no initial link sSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* account for total chunk countSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* iterate by chunk index instead of linkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make LinkFetcher convert link staticSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add helper for link addition, check for edge case to prevent inf waitSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add unit tests for LinkFetcherSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary download manager checkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary string literals around param typeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove duplicate download_manager initSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* account for empty response in LinkFetcher initSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make get_chunk_link return mandatory ExternalLinkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* set shutdown_event instead of breaking on completion so get_chunk_link is informedSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* docstrings, logging, pydocSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* use total_chunk_cound > 0Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* clarify that link has already been submitted on getting row_offsetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* return None for out of rangeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* default link_fetcher to NoneSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>---------Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* Chunk download latency (#634)* chunk download latencySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* test fixesSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* sea-migration static type checking fixesSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* check types fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* fix type issuesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* type fix revertSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* -Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* statement id in get metadata functionsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed result set extractorSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* databricks client typeSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* remove defaults, fix chunk idSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added statement type to command idSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* check types fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* renamed chunk_id to num_downloaded_chunksSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* set statement type to query for chunk downloadSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* comment fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed dup check for trowsetSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>---------Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* acquire lock before notif + formatting (black)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix importsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add get_chunk_link sSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* simplify description extractionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass session_id_hex to ThriftResultSetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* revert to main's extract descriptionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* validate row count for sync query tests as wellSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* guid_hex -> hex_guidSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* set .value in compressionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* is_direct_results -> has_more_rowsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* preliminary large metadata resultsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* account for empty table in arrow table filterSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align flowsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align flow of json with arrowSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* case sensitive support for arrow tableSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary commentSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix merge artifactsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant methodSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove incorrect docstringSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove deepcopySigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>---------Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Uh oh!
There was an error while loading.Please reload this page.
What type of PR is this?
Description
Record chunk download latency
Added
chunk_idtoSqlExecutionEventandResultDownloadHandlerHow is this tested?
Ran the query:
SELECT * FROM RANGE(20000000)Latency was recorded for 4 chunks.
The latency log for the 1st chunk:
Related Tickets & Documents
PECOBLR-653