Bug 154871

Summary: ROUNDDOWN returns invalid results when working with larger numbers
Product: LibreOffice Reporter: brianjheisler <brianjheisler>
Component: CalcAssignee: Not Assigned <libreoffice-bugs>
Status: CLOSED DUPLICATE    
Severity: normal    
Priority: medium    
Version: 7.4.5.1 release   
Hardware: All   
OS: All   
Whiteboard:
Crash report or crash signature: Regression By:

Description brianjheisler@gmail.com 2023-04-17 21:17:35 UTC 
Example I:

A1 =28147497671265.2
A2 =ROUNDDOWN(A1,-2)
A3 =ROUNDDOWN(A1,0)
A4 =ROUNDDOWN(A1,2)

The result in A2 should be 28147497671200 but it is instead 28147497671300.
The result in A3 should be 28147497671265 but it is instead 28147497671300.
The result in A4 should be 28147497671265 but it is instead 28147497671300.

====

Example II:

B1 =2814749767126.2
B2 =ROUNDDOWN(($B$1),-2)
B3 =ROUNDDOWN(($B$1),0)
B4 =ROUNDDOWN(($B$1),2)

The result in B2 should be 2814749767100 and it is.
The result in B3 should be 2814749767126 but it is instead 2814749767130.
The result in B4 should be 2814749767126 but it is instead 2814749767130.

====

I may be wrong but I suspect some kind of disconnect between Calc's IEEE 754 Double and the mechanics of the ROUNDDOWN function because as I experiment with different values, closing in on operands requiring no more than 32-bits for representation sees this problem fade away.

IEEE 754 Double should give us 15 digits (and sometimes, 16 digits) of precision in the significand.

====

Version: 7.4.5.1 (x64) / LibreOffice Community
Build ID: 9c0871452b3918c1019dde9bfac75448afc4b57f
CPU threads: 8; OS: Windows 6.1 Service Pack 1 Build 7601; UI render: Skia/Raster; VCL: win
Locale: en-US (en_US); UI: en-US
Calc: CL
Comment 1 Eike Rathke 2023-04-17 21:58:47 UTC 
> A4 should be 28147497671265
Should be 28147497671265.2

> B4 should be 2814749767126
Should be 2814749767126.2

Anyway, fixed with bug 154792.

*** This bug has been marked as a duplicate of bug 154792 ***
Comment 2 brianjheisler@gmail.com 2023-04-18 14:13:28 UTC 
I wrote up Bug 154871 thinking that the problem described in Bug 154792 was a different issue (losing several digits of significance vs. a single count in the least significant digit*). *This is how I read it anyway.

(ROUNDDOWN(28147497671265.2,0) results in 28147497671300 instead of 28147497671265 (notice ...1300 vs ...1265, a two digit error of 35) and scaling up by x10 scales up the lost data by x10 (a three digit error).)

Reading the notes in Bug 154792 suggests that this subject can get a bit deep. I see this function has been modified and will be available with 7.6.0. For now, I'll shelve my observations and defer to the better-informed. When 7.6.0 is released, I will revisit and test.