- Notifications
You must be signed in to change notification settings - Fork5
Commitaad663a
committed
Reduce number of bytes examined by convert_one_string_to_scalar().
Previously, convert_one_string_to_scalar() would examine up to 20 bytes ofthe input string, producing a scalar conversion with theoretical precisionfar greater than is of any possible use considering the other limitationson the accuracy of the resulting selectivity estimate. (I think thischoice might pre-date the caller-level logic that strips any common prefixof the strings; before that, there could have been value in scanning thestrings far enough to use all the precision available in a double.)Aside from wasting cycles to little purpose, this choice meant that the"denom" variable could grow to as much as 256^21 = 3.74e50, which couldoverflow in some non-IEEE float arithmetics. While we don't really supportany machines with non-IEEE arithmetic anymore, this still seems like quitean unnecessary platform dependency. Limit the scan to 12 bytes instead,thus limiting "denom" to 256^13 = 2.03e31, a value more likely to becomputable everywhere.Per testing by Greg Stark, which showed overflow failures in our standardregression tests on VAX.1 parent44ed65a commitaad663a
1 file changed
+9
-3
lines changedLines changed: 9 additions & 3 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
3861 | 3861 |
| |
3862 | 3862 |
| |
3863 | 3863 |
| |
3864 |
| - | |
| 3864 | + | |
| 3865 | + | |
| 3866 | + | |
| 3867 | + | |
| 3868 | + | |
| 3869 | + | |
| 3870 | + | |
3865 | 3871 |
| |
3866 |
| - | |
3867 |
| - | |
| 3872 | + | |
| 3873 | + | |
3868 | 3874 |
| |
3869 | 3875 |
| |
3870 | 3876 |
| |
|
0 commit comments
Comments
(0)