Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Chunk download latency#634

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
saishreeeee merged 22 commits intosea-migrationfromchunk-latency
Jul 21, 2025
Merged

Chunk download latency#634

saishreeeee merged 22 commits intosea-migrationfromchunk-latency
Jul 21, 2025

Conversation

@saishreeeee
Copy link
Contributor

@saishreeeeesaishreeeee commentedJul 13, 2025
edited
Loading

What type of PR is this?

  • Refactor
  • Feature
  • Bug Fix
  • Other

Description

Record chunk download latency
Addedchunk_id toSqlExecutionEvent andResultDownloadHandler

How is this tested?

  • Unit tests
  • E2E Tests
  • Manually
  • N/A

Ran the query:SELECT * FROM RANGE(20000000)
Latency was recorded for 4 chunks.
The latency log for the 1st chunk:

{  "frontend_log_event_id": "e684ff49-9026-49af-abb6-687a6a16e565",  "context": {    "client_context": {      "timestamp_millis": 1752467452769,      "user_agent": "PyDatabricksSqlConnector/4.0.5"    }  },  "entry": {    "sql_driver_log": {      "session_id": "01f0606b-4825-118d-abd2-87f5748c7dd3",      "system_configuration": {        "driver_version": "4.0.5",        "os_name": "Darwin",        "os_version": "24.5.0",        "os_arch": "arm64",        "runtime_name": "Python 3.13.3",        "runtime_version": "3.13.3",        "runtime_vendor": "CPython",        "driver_name": "Databricks SQL Python Connector",        "char_set_encoding": "utf-8",        "locale_name": "en_US"      },      "driver_connection_params": {        "http_path": <HTTP_PATH>,        "mode": "THRIFT",        "host_info": {          "host_url": <SERVER_HOSTNAME>,          "port": 443        },        "auth_mech": "PAT"      },      "sql_statement_id": "01f0606b-4854-1f89-a0cb-d5860332ef81",      "sql_operation": {        "statement_type": "QUERY",        "is_compressed": true,        "execution_result": "EXTERNAL_LINKS",        "chunk_id": 0      },      "operation_latency_ms": 15882    }  }}

Related Tickets & Documents

PECOBLR-653

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
varun-edachali-dbxand others added2 commitsJuly 14, 2025 09:35
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
@saishreeeeesaishreeeee marked this pull request as ready for reviewJuly 14, 2025 04:46
@saishreeeeesaishreeeee self-assigned thisJul 14, 2025
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
@vikrantpuppala
Copy link
Contributor

@jayantsing-db can you take a look, this PR adds latency logs (merges into sea-migration) as there are some refactorings that this PR leverages

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Copy link
Contributor

@jprakash-dbjprakash-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM. Thanks for making the changes

Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Copy link
Contributor

@varun-edachali-dbxvarun-edachali-dbx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM

@saishreeeeesaishreeeee merged commitb57c3f3 intosea-migrationJul 21, 2025
23 checks passed
varun-edachali-dbx added a commit that referenced this pull requestJul 22, 2025
@varun-edachali-dbxvarun-edachali-dbx mentioned this pull requestJul 23, 2025
5 tasks
varun-edachali-dbx added a commit that referenced this pull requestAug 4, 2025
* allow empty schema bytes for alignment with SEASigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass is_vl_op to Sea backend ExecuteResponseSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove catalog requirement in get_tablesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* move filters.py to SEA utilsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* ensure SeaResultSetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* prevent circular importsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove unused importsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove cast, throw error if not SeaResultSetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass param as TSparkParameterValueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove failing test (temp)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove SeaResultSet type assertionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* change errors to align with spec, instead of arbitrary ValueErrorSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make SEA backend methods return SeaResultSetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* use spec-aligned Exceptions in SEA backendSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove defensive row type checkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* raise ProgrammingError for invalid idSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make is_volume_operation strict boolSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove complex types codeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* Revert "remove complex types code"This reverts commit138359d.* introduce type conversion for primitive types for JSON + INLINESigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove SEA running on metadata queries (known failuresSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary docstringsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align expected types with databricks sdkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* link rest api reference to validate typesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove test_catalogs_returns_arrow_table testmetadata commands not expected to passSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix fetchall_arrow and fetchmany_arrowSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove thrift aligned test_cancel_during_execute from SEA testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary changes in example scriptsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary chagnes in example scriptsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* _convert_json_table -> _create_json_tableSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove accidentally removed testSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove new unit tests (to be re-added based on new arch)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove changes in sea_result_set functionality (to be re-added)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* introduce more integration testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove SEA tests in parameterized queriesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove partial parameter fix changesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary timestamp tests(pass with minor disparity)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* slightly stronger typing of _convert_json_typesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* stronger typing of json utility func sSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* stronger typing of fetch*_jsonSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove unused helper methods in SqlTypeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* line breaks after multi line pydocs, remove excess logsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* line breaks after multi line pydocs, reduce diff of redundant changesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diff of redundant changesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* mandate ResultData in SeaResultSet constructorSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove complex type conversionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* correct fetch*_arrowSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* recover old sea testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* move queue and result set into SEA specific dirSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass ssl_options into CloudFetchQueueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant conversion.pySigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix type issuesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* ValueError not ProgrammingErrorSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* introduce SEA cloudfetch e2e testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* allow empty cloudfetch resultSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add unit tests for CloudFetchQueue and SeaResultSetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* skip pyarrow dependent testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* simplify download process: no pre-fetchingSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* correct class name in logsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align with old implSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align next_n_rows with prev impleSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align remaining_rows with prev implSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary Optional paramsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary changes in thrift field if testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove unused importsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* init hybrid* run large queriesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* hybrid dispositionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-ncessary logSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* formatting (black)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* multi frame decompression of lz4Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* ensure no compression (temp)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* introduce separate link fetcherSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* log time to create tableSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add chunk index to table creation time logSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove custom multi-frame decompressor for lz4Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove excess logs* remove redundant tests (temp)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add link to download manager before notifying consumerSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* move link fetching immediately before table creation so link expiry is not an issueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* resolve merge artifactsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant methodsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* formatting (black)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* introduce callback to handle link expirySigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix typesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix param type in unit testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* formatting + minor type fixesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* Revert "introduce callback to handle link expiry"This reverts commitbd51b1c.* remove unused callback (to be introduced later)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* correct param extractionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove common constructor for databricks client abcSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make SEA Http Client instance a private memberSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make GetChunksResponse model more robustSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add link to doc of GetChunk response modelSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass result_data instead of "initial links" into SeaCloudFetchQueueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* move download_manager init into parent CloudFetchQueueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* raise ServerOperationError for no 0th chunkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* unused iportsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* return None in case of empty resposeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* ensure table is empty on no initial link sSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* account for total chunk countSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* iterate by chunk index instead of linkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make LinkFetcher convert link staticSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add helper for link addition, check for edge case to prevent inf waitSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add unit tests for LinkFetcherSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary download manager checkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary string literals around param typeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove duplicate download_manager initSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* account for empty response in LinkFetcher initSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make get_chunk_link return mandatory ExternalLinkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* set shutdown_event instead of breaking on completion so get_chunk_link is informedSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* docstrings, logging, pydocSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* use total_chunk_cound > 0Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* clarify that link has already been submitted on getting row_offsetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* return None for out of rangeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* default link_fetcher to NoneSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>---------Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* Chunk download latency (#634)* chunk download latencySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* test fixesSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* sea-migration static type checking fixesSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* check types fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* fix type issuesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* type fix revertSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* -Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* statement id in get metadata functionsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed result set extractorSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* databricks client typeSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* remove defaults, fix chunk idSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added statement type to command idSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* check types fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* renamed chunk_id to num_downloaded_chunksSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* set statement type to query for chunk downloadSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* comment fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed dup check for trowsetSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>---------Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* acquire lock before notif + formatting (black)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix importsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add get_chunk_link sSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* simplify description extractionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass session_id_hex to ThriftResultSetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* revert to main's extract descriptionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* validate row count for sync query tests as wellSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* guid_hex -> hex_guidSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* set .value in compressionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* formatting (black)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant testSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* move extra_params to the backSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* is_direct_results -> has_more_rowsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* Revert "is_direct_results -> has_more_rows"This reverts commit0e87374.* stop passing session_id_hexSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant commentSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add extra_params paramSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass extra_params into test_...unset...Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove excess session_id_heSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce changes in DatabricksRetryPolicySigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diff in DatabricksRetryPolicySigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* simple comments on proxy settingSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* link docs for getproxies)(Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* rename proxy specific attrs with proxy prefixSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>---------Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
varun-edachali-dbx added a commit that referenced this pull requestAug 4, 2025
* remove redundant conversion.pySigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix type issuesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* ValueError not ProgrammingErrorSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* introduce SEA cloudfetch e2e testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* allow empty cloudfetch resultSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add unit tests for CloudFetchQueue and SeaResultSetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* skip pyarrow dependent testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* simplify download process: no pre-fetchingSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* correct class name in logsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align with old implSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align next_n_rows with prev impleSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align remaining_rows with prev implSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary Optional paramsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary changes in thrift field if testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove unused importsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* init hybrid* run large queriesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* hybrid dispositionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-ncessary logSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* formatting (black)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* multi frame decompression of lz4Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* ensure no compression (temp)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* introduce separate link fetcherSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* log time to create tableSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add chunk index to table creation time logSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove custom multi-frame decompressor for lz4Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove excess logs* remove redundant tests (temp)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add link to download manager before notifying consumerSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* move link fetching immediately before table creation so link expiry is not an issueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* resolve merge artifactsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant methodsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* formatting (black)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* introduce callback to handle link expirySigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix typesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix param type in unit testsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* formatting + minor type fixesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* Revert "introduce callback to handle link expiry"This reverts commitbd51b1c.* remove unused callback (to be introduced later)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* correct param extractionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove common constructor for databricks client abcSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make SEA Http Client instance a private memberSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make GetChunksResponse model more robustSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add link to doc of GetChunk response modelSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass result_data instead of "initial links" into SeaCloudFetchQueueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* move download_manager init into parent CloudFetchQueueSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* raise ServerOperationError for no 0th chunkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* unused iportsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* return None in case of empty resposeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* ensure table is empty on no initial link sSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* account for total chunk countSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* iterate by chunk index instead of linkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make LinkFetcher convert link staticSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add helper for link addition, check for edge case to prevent inf waitSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add unit tests for LinkFetcherSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary download manager checkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary string literals around param typeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove duplicate download_manager initSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* account for empty response in LinkFetcher initSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* make get_chunk_link return mandatory ExternalLinkSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* set shutdown_event instead of breaking on completion so get_chunk_link is informedSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* docstrings, logging, pydocSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* use total_chunk_cound > 0Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* clarify that link has already been submitted on getting row_offsetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* return None for out of rangeSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* default link_fetcher to NoneSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>---------Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* Chunk download latency (#634)* chunk download latencySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* test fixesSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* sea-migration static type checking fixesSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* check types fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* fix type issuesSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* type fix revertSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* -Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* statement id in get metadata functionsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed result set extractorSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* databricks client typeSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* remove defaults, fix chunk idSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added statement type to command idSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* check types fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* renamed chunk_id to num_downloaded_chunksSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* set statement type to query for chunk downloadSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* comment fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed dup check for trowsetSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>---------Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* acquire lock before notif + formatting (black)Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix importsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* add get_chunk_link sSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* simplify description extractionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* pass session_id_hex to ThriftResultSetSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* revert to main's extract descriptionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* validate row count for sync query tests as wellSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* guid_hex -> hex_guidSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* set .value in compressionSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* reduce diffSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* is_direct_results -> has_more_rowsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* preliminary large metadata resultsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* account for empty table in arrow table filterSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align flowsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* align flow of json with arrowSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* case sensitive support for arrow tableSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove un-necessary commentSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* fix merge artifactsSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove redundant methodSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove incorrect docstringSigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>* remove deepcopySigned-off-by: varun-edachali-dbx <varun.edachali@databricks.com>---------Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@vikrantpuppalavikrantpuppalavikrantpuppala approved these changes

@jprakash-dbjprakash-dbjprakash-db approved these changes

@jayantsing-dbjayantsing-dbAwaiting requested review from jayantsing-db

+1 more reviewer

@varun-edachali-dbxvarun-edachali-dbxvarun-edachali-dbx approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@saishreeeeesaishreeeee

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@saishreeeee@vikrantpuppala@jprakash-db@varun-edachali-dbx

[8]ページ先頭

©2009-2025 Movatter.jp