fix: handle null values in data#636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Jump to bottom

Merged

bednar merged 10 commits intomasterfromfix/issue-621

Jan 31, 2024

Merged

fix: handle null values in data#636

bednar merged 10 commits intomasterfromfix/issue-621

Jan 31, 2024

Conversation

Copy link

Contributor

alespour commentedJan 30, 2024•
edited
Loading

Closes#621

Proposed Changes

Handles data with missing values when querying to data frames. The query functionsquery_data_frame... have new optional parameteruse_extension_dtypes.

defquery_data_frame(self,query:str,org=None,data_frame_index:List[str]=None,params:dict=None,use_extension_dtypes:bool=False):    ...defquery_data_frame_stream(self,query:str,org=None,data_frame_index:List[str]=None,params:dict=None,use_extension_dtypes:bool=False):    ...

when set toTrue, missing values are represented aspandas.NA and dtype of columns containing<NA> is of correspondingnullable extension dtypes frompandas package (ie.Int64,Float64,Boolean etc). Missing value can be checked usingpandas.isna() function.
whenFalse (default), missing values are represented asNone, and dtype of columns with missing values is either'object' or'float64' when type of values is numeric. This is a standard conversion behavior of data frames, see
- https://note.nkmk.me/en/python-pandas-nan-none-na/
- https://pandas.pydata.org/docs/user_guide/missing_data.html

Example output (with data from#621):

use_extension_dtypes=True

<bound method NDFrame.head of     result  table                    _start                     _stop                            _time _measurement  test_double  test_long0  _result      0 2023-12-15 13:19:54+00:00 2023-12-15 13:19:57+00:00 2023-12-15 13:19:55.372000+00:00test          4.0<NA>1  _result      0 2023-12-15 13:19:54+00:00 2023-12-15 13:19:57+00:00        2023-12-15 13:19:56+00:00test<NA>1>

use_extension_dtypes=False

<bound method NDFrame.head of     result  table                    _start                     _stop                            _time _measurement  test_double  test_long0  _result      0 2023-12-15 13:19:54+00:00 2023-12-15 13:19:57+00:00 2023-12-15 13:19:55.372000+00:00test          4.0        NaN1  _result      0 2023-12-15 13:19:54+00:00 2023-12-15 13:19:57+00:00        2023-12-15 13:19:56+00:00test          NaN        1.0>

Note: the conversion of numeric values to extension dtypes works properly withpandas>=2.0, so in Python 3.7 environment, where the latest available pandas is 1.3.5, dtype of columns with NA values is'object' ie. same as without the use extension types. For Python 3.8+,pandas 2.x is available.

Checklist

CHANGELOG.md updated
Rebased/mergeable
A test has been added if appropriate
pytest tests completes successfully
Commit messages areconventional

alespour added5 commits

January 30, 2024 15:00

fix: handle null values in Flux data

0c99bfb

test: add tests for null value handling and extension dtypes

01ce750

test: fix failures with empty warnings cases

8030d84

test: comment out dtype some extra assertion until solved

5f7a1a9

test: skip extension dtypes test on pythn 3.7

074c013

Copy link

codecov-commenter commentedJan 30, 2024•
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base(27777d1) 90.19% compared to head(17ab3b1) 90.40%.

❗ Your organization needs to install theCodecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@##           master     #636      +/-   ##==========================================+ Coverage   90.19%   90.40%   +0.21%==========================================  Files          39       39                Lines        3467     3503      +36     ==========================================+ Hits         3127     3167      +40+ Misses        340      336       -4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report?Share it here.