Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings
This repository was archived by the owner on Oct 29, 2024. It is now read-only.

Fix chunked query to return chunk resultsets#753

Merged
sebito91 merged 1 commit intoinfluxdata:masterfromhrbonz:wip/chunked_query
Apr 10, 2020

Conversation

hrbonz
Copy link
Contributor

@hrbonzhrbonz commentedSep 8, 2019•
edited
Loading

When querying large data sets, it's vital to get a chunked responses to manage memory usage. Wrapping the query response in a generator and streaming the request provides the desired result.
It also fixesInfluxDBClient.query() behavior for chunked queries that is currently not working according tospecs

close#585
close#531
close#538

@hrbonz
Copy link
ContributorAuthor

I'm currently running tox tests and will fix whatever fails in travisCI

@hrbonzhrbonzforce-pushed thewip/chunked_query branch 2 times, most recently from2a2d459 toa6bc05dCompareSeptember 8, 2019 13:28
hrbonz added a commit to hrbonz/influxdump that referenced this pull requestSep 14, 2019
Option to dump data in a folder in fragment files of a given chunk size.This works only if the chunksize option is properly implemented ininfluxdb-python (seeinfluxdata/influxdb-python#753)
@psy0rz
Copy link

Does this also close#538 ?

@hrbonz
Copy link
ContributorAuthor

Yes, it covers it as well. I've had issues with my laptop so got caught in a bit of a snag, will get back to fixing the tests.
The tests were designed to pass rather than reflect the method documented behavior.

@hrbonzhrbonz marked this pull request as ready for reviewMarch 23, 2020 08:07
@hrbonz
Copy link
ContributorAuthor

I had to change the test substantially as it was testing a broken behavior not the documented (and better one imho).
I'm testing that I'm getting the proper number of chunkedResultSets and testing that eachResultset returns the proper elements.

@hrbonz
Copy link
ContributorAuthor

hrbonz commentedMar 23, 2020•
edited
Loading

On a side note, I've been using this code to move data from an old version of influxdb to 1.7 (and doing some type casting in the middle) with no issue. About 50G of data across 1500 measurements.
Without the proper chunking and generator, my memory was going through the roof.

When querying large data sets, it's vital to get a chunked responses tomanage memory usage. Wrapping the query response in a generator andstreaming the request provides the desired result.It also fixes `InfluxDBClient.query()` behavior for chunked queries thatis currently not working according to[specs](https://github.com/influxdata/influxdb-python/blob/master/influxdb/client.py#L429)closeinfluxdata#585closeinfluxdata#531
@russorat
Copy link
Contributor

@hrbonz Thank you! i've reached out to@sebito91 to take a look.

@hrbonz
Copy link
ContributorAuthor

Thanks@russorat, hopefully this PR is getting some attention while the new library is getting all the love ;)

xiandong79 reacted with thumbs up emojisebito91 reacted with heart emoji

@sebito91sebito91 self-assigned thisApr 8, 2020
Copy link
Contributor

@sebito91sebito91 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This is great stuff, thank you@hrbonz!

@sebito91sebito91 merged commitc903d73 intoinfluxdata:masterApr 10, 2020
ocworld pushed a commit to AhnLab-OSS/influxdb-python that referenced this pull requestApr 13, 2020
When querying large data sets, it's vital to get a chunked responses tomanage memory usage. Wrapping the query response in a generator andstreaming the request provides the desired result.It also fixes `InfluxDBClient.query()` behavior for chunked queries thatis currently not working according to[specs](https://github.com/influxdata/influxdb-python/blob/master/influxdb/client.py#L429)Closesinfluxdata#585.Closesinfluxdata#531.Closesinfluxdata#538.
@hrbonz
Copy link
ContributorAuthor

@sebito91 quite happy to see it finally done and merged! Now to

ping@psy0rz for the inspiration

sebito91 and psy0rz reacted with hooray emoji

@psy0rz
Copy link

awesome! thank YOU for actually implementing it all the way :)

Sign up for freeto subscribe to this conversation on GitHub. Already have an account?Sign in.
Reviewers

@sebito91sebito91sebito91 approved these changes

@aviauaviauAwaiting requested review from aviauaviau is a code owner

@xginn8xginn8Awaiting requested review from xginn8xginn8 is a code owner

Assignees

@sebito91sebito91

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Out of memory error when querying large data set with chunked=True
4 participants
@hrbonz@psy0rz@russorat@sebito91
[8]ページ先頭
©2009-2025 Movatter.jp