Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

feat: make QueryJob.done() method more performant#544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
tswast merged 1 commit intogoogleapis:masterfromplamut:iss-534
Mar 10, 2021

Conversation

@plamut
Copy link
Contributor

@plamutplamut commentedMar 9, 2021
edited
Loading

Fixes#534.

This PR removes refreshing query results from theQueryJob.done() method, the latter is now just thedone() method inherited from the_AsyncJob base class that at most reloads the job itself and checks if its state is DONE.

Since blocking poll from thePollingFuture base class repeatedly invokesdone(), the change would cause too many job reload requests while waiting for the query results. TheQueryJob class thus overrides the_done_or_raise() method repeatedly used by the blocking poll so that the polling is actually performed by fetching the query results. The latter call can block for up to 10 seconds, meaning that fewer polling requests are made than if reload the job was used.

How to test

Set logging level to DEBUG to see what HTTP requests are made. Then run a query job that normally takes more than 10 seconds to complete, for example:

SELECTCONCAT('https://stackoverflow.com/questions/',    CAST(idas STRING))as url,view_count,1AS fooFROM`bigquery-public-data.stackoverflow.posts_questions`""""""ORDER BY view_countDESC

Tip: If running the query in multiple test runs in a row, change1 AS foo to a different value so that the query is re-run and cached query results arenot used)

While the query is running, callquery_job.result() - query results should be fetched, but with a reasonable amount of requests.

Beside testing the.result() method, the.done() method should be checked, too - if it is run repeatedly while the query is running, each call should finish "fast", i.e. it should block for significantly less time than 10 seconds, because all it needs to do is to reload the job data itself.

PR checklist:

  • Make sure to open an issue as abug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

@plamutplamut added the do not mergeIndicates a pull request not ready for merge, due to either quality or timing. labelMar 9, 2021
@google-clagoogle-clabot added the cla: yesThis human has signed the Contributor License Agreement. labelMar 9, 2021
@product-auto-labelproduct-auto-labelbot added the api: bigqueryIssues related to the googleapis/python-bigquery API. labelMar 9, 2021
@plamutplamut removed the do not mergeIndicates a pull request not ready for merge, due to either quality or timing. labelMar 10, 2021
@plamutplamut marked this pull request as ready for reviewMarch 10, 2021 14:09
@plamutplamut requested review froma team,stephaniewang526 andtswast and removed request fora teamMarch 10, 2021 14:09
Copy link
Contributor

@tswasttswast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Magnificent!

@tswasttswast merged commita3ab9ef intogoogleapis:masterMar 10, 2021
@plamutplamut deleted the iss-534 branchMarch 10, 2021 20:52
gcf-merge-on-greenbot pushed a commit that referenced this pull requestMar 16, 2021
🤖 I have created a release \*beep\* \*boop\*---## [2.12.0](https://www.github.com/googleapis/python-bigquery/compare/v2.11.0...v2.12.0) (2021-03-16)### Features* make QueryJob.done() method more performant ([#544](https://www.github.com/googleapis/python-bigquery/issues/544)) ([a3ab9ef](https://www.github.com/googleapis/python-bigquery/commit/a3ab9efdd0758829845cfcb6ca0ac1f03ab44f64))### Bug Fixes* remove DB-API dependency on pyarrow with decimal query parameters ([#551](https://www.github.com/googleapis/python-bigquery/issues/551)) ([1b946ba](https://www.github.com/googleapis/python-bigquery/commit/1b946ba23ee7df86114c6acb338ec34e6c92af6d))---This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This was referencedMar 17, 2021
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@tswasttswasttswast approved these changes

@stephaniewang526stephaniewang526Awaiting requested review from stephaniewang526

Assignees

No one assigned

Labels

api: bigqueryIssues related to the googleapis/python-bigquery API.cla: yesThis human has signed the Contributor License Agreement.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Improve performance ofQueryJob.done() method

2 participants

@plamut@tswast

[8]ページ先頭

©2009-2025 Movatter.jp