Bug 96918 - Rounding display error for 15 digits integers
Summary: Rounding display error for 15 digits integers
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
Inherited From OOo
Hardware: All All
: medium normal
Assignee: Eike Rathke
URL:
Whiteboard: target:5.2.0 target:5.1.0
Keywords:
Depends on:
Blocks:
 
Reported: 2016-01-06 07:23 UTC by Cœur
Modified: 2024-05-14 13:29 UTC (History)
3 users (show)

See Also:
Crash report or crash signature:


Attachments
File with list of inputted numbers and how they are shown (10.49 KB, application/vnd.oasis.opendocument.spreadsheet)
2016-01-08 17:03 UTC, Usama
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Cœur 2016-01-06 07:23:28 UTC
Found by DarthGizka on StackOverflow: http://stackoverflow.com/questions/34499255/spreadsheet-calculations-with-at-least-the-accuracy-of-a-c-double

Even though internal values are at least precise to 2^52 or 2^53 (precision of a IEEE 754 double), displayed values are less than 15 digits-precise.

* inputing  999999999999999 shows 1000000000000000
* inputing 1125899906842623 shows 1125899906842620
* inputing 4503599627370495 shows 4503599627370490
* inputing 4503599627370496 shows 4503599627370500

The displayed values should match the precision of internal values.
Comment 1 Usama 2016-01-08 17:03:40 UTC
Created attachment 121803 [details]
File with list of inputted numbers and how they are shown

I can confirm this on Libreoffice 5.1.0.1 Linux. Attached is a file containing the value inputted and the result shown.

You might have to change the format from default to -12345 to see all the numbers
Comment 2 m_a_riosv 2016-01-10 02:09:33 UTC
You can find some information in https://bugs.documentfoundation.org/show_bug.cgi?id=94394#c1 the comment is from a main calc developer.
Comment 3 Eike Rathke 2016-01-11 13:15:03 UTC
This is slightly different though because the entered number can be represented and displayed as is, but our conversion to decimal digits stops the bin-dec correction after 14 decimal digits and starts normal rounding after that.
Comment 4 Commit Notification 2016-01-13 13:50:20 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=0f6203edf74832f84d8263d7a544d679203a4efc

tdf#96918 display accurate integer double values up to (2^53)-1

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 5 Commit Notification 2016-01-13 13:50:24 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=a3f5f4f6369552a3da870de1ed4ea9d8c628c7a8

unit test for tdf#96918 display accurate integer double values

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 6 Eike Rathke 2016-01-13 14:33:40 UTC
Pending review
https://gerrit.libreoffice.org/21429 for 5-1
https://gerrit.libreoffice.org/21435 for 5-1-0
https://gerrit.libreoffice.org/21430 for 5-0
Comment 7 Commit Notification 2016-01-14 19:55:38 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=fdf982f70b2944053d995baaa3d78c7cdc4bbc4b

handle negative decimal places for rounding, tdf#96918 related

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 8 Commit Notification 2016-01-14 19:55:42 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=5299400e5cce3060a0de85bb4dedd065b5ad1f41

unit test for negative decimal places rounding, tdf#96918 related

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 9 Commit Notification 2016-01-15 13:25:43 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "libreoffice-5-1":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e480172cb91b4adca6a7fcd42f68c2ff2874fe54&h=libreoffice-5-1

tdf#96918 display accurate integer double values up to (2^53)-1

It will be available in 5.1.1.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 10 Commit Notification 2016-01-19 12:45:54 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "libreoffice-5-1-0":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=ab0ef85c7dbc0f8f25cb39ebcce4d5aa038d2f56&h=libreoffice-5-1-0

tdf#96918 display accurate integer double values up to (2^53)-1

It will be available in 5.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.