Created attachment 137066 [details] Example errors on big integers In attachments, there are some big numbers, and arithmetic operations on them (integer division by 2, and remainder of that division). With red are marked wrong results (where MOD() errors out, or integer division gives off-by-one error). Given that the numbers are all representable in double, the operations should succeed. Please note, that there's no some threshold, because there are numbers of same magnitude, which produce correct results. Tested with Version: 5.4.2.2 (x64) Build ID: 22b09f6418e8c2d508a9eaf86b2399209b0990f4 CPU threads: 4; OS: Windows 6.19; UI render: default; Locale: ru-RU (ru_RU); Calc: CL
A fix is proposed: https://gerrit.libreoffice.org/43477 I don't know if just zeroing ~3 least significant binary bits would be better maybe?
(In reply to Mike Kaganski from comment #1) > I don't know if just zeroing ~3 least significant binary bits would be > better maybe? Clarification: I suppose that it is only useful to zero some trailing bits when they are not a part of integers, and contribute to some small fraction (say, <10^-5..10^-10)... but I can be wrong. Eike, Markus, could you please advise, possibly clarify which problems the rounding solves that I might break with this change?
The approxValue() rounding as used in approxFloor() and approxCeil() solves the problem that due to precision and rounding errors a floor(1.99999...) that should had been floor(2.0) actually results in 2.0 instead of 1.0 If the change affects *only* integer values to return early then it should be fine.
Mike Kaganski committed a patch related to this issue. It has been pushed to "master": http://cgit.freedesktop.org/libreoffice/core/commit/?id=334a9f16cd1d1f9694f885c759903a41aa3d4833 tdf#113211: fix calculations with big integers It will be available in 6.0.0. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
A polite ping to Mike Kaganski: is this bug fixed? if so, could you please close it as RESOLVED FIXED ? Thanks
Let's call it fixed. :) Thanks Xisco!