Movatterモバイル変換

Skip to content

databricks/databricks-sql-pythonPublic

NotificationsYou must be signed in to change notification settings
Fork126
Star210

Fix : Unable to poll results in async API#515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged

jprakash-db merged 5 commits intorelease/v3.7.3fromjprakash-db/async-api-direct-results

Feb 28, 2025

Merged

Fix : Unable to poll results in async API#515

jprakash-db merged 5 commits intorelease/v3.7.3fromjprakash-db/async-api-direct-results

Feb 28, 2025

Conversation

@jprakash-db

Copy link

Contributor

jprakash-db commentedFeb 25, 2025•
edited
Loading

Description

Polling using the async API returned theRESOURCE_NOT_FOUND error for async executed queries

Fix

Force retention of data on the server side by not returning the results in thedirect results within theexecute_query response

Testing

Added e2e tests

@jprakash-db

Fixed the async issue

ed0f73b

@jprakash-db

jprakash-db requested review fromjayantsing-db andsamikshya-db

February 25, 2025 15:45

@jprakash-db

jprakash-db requested review fromandrefurlan-db,benc-db,jackyhu-db,rcypher-databricks andyunbodeng-db ascode owners

February 25, 2025 15:45

@jprakash-db

jprakash-db self-assigned this

samikshya-db

samikshya-db reviewed

View reviewed changes

src/databricks/sql/thrift_backend.py OutdatedShow resolvedHide resolved

jayantsing-db

jayantsing-db reviewed

View reviewed changes

src/databricks/sql/thrift_backend.py Outdated

		# For async operation we don't want the direct results
		getDirectResults=ttypes.TSparkGetDirectResults(
		maxRows=max_rows,maxBytes=max_bytes
		maxRows=0ifasync_opelsemax_rows,

Copy link

Contributor

jayantsing-dbFeb 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Instead of setting maxRows as 0, I would recommend not specifying the direct result object in the request altogether. This is to make the intention clear of not expecting the direct results.

Copy link

ContributorAuthor

jprakash-dbFeb 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Have made the object None

@jprakash-db

Added unit tests

3b51b4e

@jprakash-db

jprakash-db had a problem deploying to azure-prod

February 27, 2025 06:49

— with

GitHub Actions Failure

@jprakash-db

jprakash-db requested a review fromsamikshya-db

February 27, 2025 06:57

jayantsing-db

jayantsing-db reviewed

View reviewed changes

src/databricks/sql/thrift_backend.py

Comment on lines +902 to 907

		getDirectResults=None
		ifasync_op
		elsettypes.TSparkGetDirectResults(
		maxRows=max_rows,
		maxBytes=max_bytes,
		),

Copy link

Contributor

jayantsing-dbFeb 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

do we have a linter for python driver yet? if not, could you please open an issue to community or internally?

Copy link

ContributorAuthor

jprakash-dbFeb 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@jayantsing-db I can raise a new PR for that

@jprakash-db

Minor change

d84c15b

@jprakash-db

jprakash-db had a problem deploying to azure-prod

February 27, 2025 11:40

— with

GitHub Actions Failure

@jprakash-db

Changed to 5

0c0ea0c

@jprakash-db

jprakash-db had a problem deploying to azure-prod

February 27, 2025 12:15

— with

GitHub Actions Failure

@jprakash-db

Increased time

f2a01b1

@jprakash-db

jprakash-db temporarily deployed to azure-prod

February 27, 2025 16:04

— with

GitHub Actions Inactive

jayantsing-db

jayantsing-db reviewed

View reviewed changes

tests/e2e/common/large_queries_mixin.py

		withself.cursor()ascursor:
		whileduration<min_duration:
		assertscale_factor<512,"Detected infinite loop"
		assertscale_factor<1024,"Detected infinite loop"

Copy link

Contributor

jayantsing-dbFeb 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Could you explain this change?

Copy link

ContributorAuthor

jprakash-dbFeb 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

For long running queries the previous upper limit of scale factor was not sufficient to run queries for atleast 5 minutes, hence increased it

@jprakash-db

jprakash-db requested a review fromjayantsing-db

February 28, 2025 04:08

samikshya-db

samikshya-db approved these changes

View reviewed changes

@jprakash-db

jprakash-db merged commitfb3e32c intorelease/v3.7.3

2 of 3 checks passed

@jprakash-db

jprakash-db mentioned this pull request

Forward porting all changes into 4.x.x. uptil v3.7.3#529

Merged

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Labels

None yet

4 participants

@jprakash-db

@samikshya-db

@jayantsing-db

[8]ページ先頭

©2009-2025 Movatter.jp