Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork137
Test Stream::parseFloat() with many input digits#133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
matthijskooijman commentedDec 10, 2020
|
edgar-bonet commentedDec 10, 2020
@matthijskooijman: The overflow affects a local variable, not exposed to the API. |
aentinger commentedDec 11, 2020 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
While I technically agree with@matthijskooijman I've just yesterday participated in a discussion with@tigoe in which he expressed his dismay on some prototype code using @edgar-bonet Thank you very much for providing this first step on fixing your issue (A failing test is a great was to start!). Considering floating point comparison you can use the REQUIRE(mock.parseFloat() == Approx(-3.141592654f); Here you can find more documentation how to perform such floating point comparisons. |
edgar-bonet commentedDec 11, 2020
@aentinger: Thanks for the info! I did not know about this library. The Now, I have a question for you. If the stream provides an integer larger than In the first case, I would add the relevant test to this PR, and then submit a PR than handles both situations. In the second case, I would wait for this issue to be sorted out before moving to the “parse large integer” problem. |
aentinger commentedDec 11, 2020
Thank you integrating |
matthijskooijman commentedDec 11, 2020
Right, that would make it even easier to change from |
edgar-bonet commentedDec 11, 2020
@aentinger wrote:
Sorry, I was not clear enough. It is not about
My question is whether these two points should be considered parts of the same issue or separate issues. @matthijskooijman: I do think it is off-topic. |
aentinger commentedDec 14, 2020
Good morning@edgar-bonet 👋 ☕ |
edgar-bonet commentedDec 14, 2020
Hi@aentinger, thanks for your answer. I just added a failing test case with a number that has too many digits before the decimal point. This is in addition to the previous one that has too many digits after the decimal point. Should I squash the three commits together and force-push? |
…low when parsing float values.However, we still need to ensure against too large values contained in streams. This should be possible because the maximum length of a float value pre-comma is known to be 38 digits (FLT_MAX_10_EXP).
aentinger commentedDec 14, 2020 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Please no. I already added a first possible solution on how to address this issue. Please to a pull first before your next commit or you've got disentangle it with the commit I pushed 😉 |
codecov-io commentedDec 14, 2020 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Codecov Report
@@ Coverage Diff @@## master #133 +/- ##==========================================- Coverage 96.41% 96.06% -0.35%========================================== Files 14 14 Lines 837 839 +2 ==========================================- Hits 807 806 -1- Misses 30 33 +3
Continue to review full report at Codecov.
|
api/Stream.cpp Outdated
| double value =0.0; | ||
| int c; | ||
| float fraction =1.0; | ||
| unsignedint digits_post_comma =0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Ifvalue is a floating point number, there is no need for this extra variable. It could be folded intovalue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
That's a good question here. What's more computing time expensive -pow or repeatedly doing a double multiplication?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
However, a further argument againstpow is that there's a question of availability (and with what argument types) across all supported platforms.
api/Stream.cpp Outdated
| returnvalue* fraction; | ||
| else | ||
| return value; | ||
| value/=pow(10, digits_post_comma); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
pow() involves a logarithm and an exponential, which is very expensive on MMU-less processors.
edgar-bonet commentedDec 14, 2020
I had considered parsing straight into a float, but I then assumed the original code used an integer for performance reasons. My tests show parsing into a float takes about 50% more processing time an AVR. |
aentinger commentedDec 14, 2020 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Right now EDIT: |
api/Stream.cpp Outdated
| elseif (c =='.') | ||
| isFraction =true; | ||
| elseif(c >='0' && c <='9') {// is c a digit? | ||
| value = value *10 + c -'0'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
value may overflow toINFINITY on AVR if there are more than 38 digits (maybe that's so many we do not care about that case?), event with a number smaller thanFLT_MAX. I suggest instead:
if(isFraction) { fraction *=0.1; value = value + fraction * (c -'0'); }else { value = value *10 + c -'0'; }
Note that on something other than AVR one would need more than 308 digits to demonstrate the problem.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I like this form, however in that case we've got to commit to using adouble forvalue. I personally would not mind since we are talking about stream parsing after all and the onlyStream channels currently available are Serial and various networking streams and which are slow even compared to floating point multiplication.
I can further support my argument pro floating point implementation that if you want to run performance-critical/floating-point applications on an 8-Bit architecture you've selected the wrong MCU alltogether.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Since I got no feedback over quite a long period of time (unfortunate but not unexpected given what we are currently swamped in) I'm moving forward with merging this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Let's hold on for a second though@edgar-bonet can you integrate your last change suggested above by yourself:
if(isFraction) { fraction *=0.1; value = value + fraction * (c -'0'); }else { value = value *10 + c -'0'; }
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
@aentinger: OK, I'll do it today.
Give a 311-digit number to Stream::parseFloat(). This makes the localvariable `value' overflow to infinity. With so many digits, the numbercannot be parsed into an integer, not even into an integer stored as a`double'.Note that 40 digits would be enough to unveil this issue on AVR.
If more than 309 digits are provided to Stream::parseFloat() (more than39 on AVR), the internal variable 'value' would overflow to infinity. Weavoid this by not storing the parsed number as an integer-in-a-float.
edgar-bonet commentedJan 25, 2021
As requested, I just pushed my proposed change. Prior to this, I took the liberty to modify the test “A float is provided with too many digits after the decimal point” by adding more digits, in order to evidence the issue the last commit is fixing. |
aentinger left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
LGTM 👍 Thank you@edgar-bonet 🚀
As a followup to issue#129: “Stream::parseFloat() fails if it reads too many digits”, this pull request adds a failing test to test_parseFloat.cpp that reveals the problem. I ran into a couple of issues when writing the test:
Ensuring correct rounding when parsing a decimal representation of a number is not trivial, and
Stream::parseFloat()does not attempt to provide such guarantee. Testing for exact equality to the expected result is thus prone to spurious failures.The existing tests in test_parseFloat.cpp do test for exact equality, and it seems to work. However, the more digits are processed, the greater the chances that rounding errors pile up and the end result differs from the correctly rounded one. Since this test requires processing many decimal digits, I had to use fuzzy floating point comparison in order to avoid the test failing for the wrong reason.
It would seem these tests are meant to be compiled and run on 64-bit Linux hosts, where a
longis 64-bits wide. Most (all?) Arduino platforms, in contrast, have 32-bitlongs. Since this test is meant to reveal the effect of alongvariable overflowing, I had to add many more digits than would be needed to trigger the overflow on an Arduino. As shown in issueStream::parseFloat() fails if it reads too many digits #129, 10 digits of π are enough to makeStream::parseFloat()fail badly on an actual Arduino.