Bug 130728 - Change Calc to display in appropriate precision
Summary: Change Calc to display in appropriate precision
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
(earliest affected)
6.0 all versions
Hardware: All All
: medium normal
Assignee: Not Assigned
Depends on:
Reported: 2020-02-17 10:19 UTC by Jonny Grant
Modified: 2022-11-14 13:40 UTC (History)
1 user (show)

See Also:
Crash report or crash signature:

Google Docs doesn't have this problem (20.19 KB, image/png)
2020-02-24 17:54 UTC, Jonny Grant
Another easy example of this working correctly in Google Docs but not Calc (24.18 KB, image/png)
2020-03-03 11:19 UTC, Jonny Grant

Note You need to log in before you can comment on or make changes to this bug.
Description Jonny Grant 2020-02-17 10:19:34 UTC
There's a well known issue, Calc will display this result as 1E-04

What was expected
Expected 0.0001



If we expand the column it presently shows show 0.0000999999999999994

What's gone wrong
There are various ways to fix this bug, the best is to calcuate accurately, and not display raw Double precision.

Calc seems to be displaying raw double precision, and not using it's own accurate calculation of decimal numbers.
It's pretty easy to simply have a BigNum class with an int32 and an uint32 representing the decimal precision to avoid these issues.

The alternative, is to select appropriate formatting 
eg in this case it would be #,##0.0000

Personally I favour (A), just calculate and show exactly correct values. 

My $5 Casio calculator displays the result correctly. There is no reason to ever show a user  1E-04 or 0.0000999999999999994

Build ID: 1:6.0.7-0ubuntu0.18.04.10
CPU threads: 8; OS: Linux 4.15; UI render: default; VCL: gtk3; 
Locale: en-GB (en_GB.UTF-8); Calc: group
Comment 1 Oliver Brinzing 2020-02-17 17:32:57 UTC

*** This bug has been marked as a duplicate of bug 130725 ***
Comment 2 Jonny Grant 2020-02-24 17:54:30 UTC
Created attachment 158146 [details]
Google Docs doesn't have this problem
Comment 3 Jonny Grant 2020-03-03 11:19:41 UTC
Created attachment 158335 [details]
Another easy example of this working correctly in Google Docs but not Calc

Another easy example of this working correctly in Google Docs but not Calc
Comment 4 Eike Rathke 2020-09-27 15:28:50 UTC
Welcome to the wonderful world of IEEE 754 double floating point values.
Go to http://weitz.de/ieee/ and input the numbers
and click the - minus button; see the result is 1.0000000000000026E-4

For more information visit https://erack.de/bookmarks/D.html#010203 and pick your reading.

For the values in attachment 158335 [details] Calc also displays 0.8, there's no problem.

Meanwhile, for Calc 7.0,
for the values in attachment 158146 [details] Calc also displays 0.0001, there's no problem anymore.