Bug 96406 - Wrong zero calculation
Summary: Wrong zero calculation
Status: RESOLVED NOTABUG
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
5.0.3.2 release
Hardware: x86-64 (AMD64) All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-12-11 10:36 UTC by Martin
Modified: 2015-12-11 19:04 UTC (History)
1 user (show)

See Also:
Crash report or crash signature:


Attachments
Wrong zero calculation (13.71 KB, application/vnd.oasis.opendocument.spreadsheet)
2015-12-11 10:36 UTC, Martin
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Martin 2015-12-11 10:36:40 UTC
Created attachment 121218 [details]
Wrong zero calculation

Simple formula (=C26+A27-B27) calculates "5,6843418860808E-14" instead of expected "0".

System: Linux Mint 17 (Linux 3.13.0-24-generic (x86_64))
Version: 5.0.3.2

Same behavior using LO 5.0.2.2 with Windows XP (32).

See cell C26 in the attachement.

Changing cell A10 to "181,000005" and B26 to "178,500005" results in correct value "0" in cell C26. With values "181,000006" and "178,500006" the error occurs.
Comment 1 MM 2015-12-11 12:34:08 UTC
It's a rounding problem with floating point numbers. Annoying maybe, but not a bug I guess.
When set options > lo calc > calculate > 'limit decimals ....' on and 'decimal places' to 2, the problem is gone.
Comment 2 m_a_riosv 2015-12-11 19:04:58 UTC
I think the issue usually occurs with numbers that are periodic in binary base, so doing operations on them results on the precision problems. If you set up 15 decimal places for the all cell's format, you can see that C27 is different than B27 on the last decimals.

The precision in calc is fifteen numbers plus sign, and this is a hardware limitation.
https://bugs.documentfoundation.org/show_bug.cgi?id=50299
take special attention to comments 4 and 6 of Kohei, one of the most expert developers with calc.

Combining the MM mentioned option with 12 decimal places, with Menu/Tools/LibreOffice calc/Calculate - Precision as Shown  seems to work fine in this case.
What usually for accounting purposes it's enough precision.

Please if you are not agree with resolution, reopen it.