Bug 142134 - Basic math calculation errors
Summary: Basic math calculation errors
Status: RESOLVED NOTABUG
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
7.1.2.2 release
Hardware: x86-64 (AMD64) Windows (All)
: medium normal
Assignee: Not Assigned
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-05-06 17:50 UTC by Emily Tilden
Modified: 2021-05-09 06:00 UTC (History)
2 users (show)

See Also:
Crash report or crash signature:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Emily Tilden 2021-05-06 17:50:15 UTC 
Description:
I have a cell that contains "=10.2-9.3" and the value that is returned is 0.899999999999999 instead of 0.9 exactly.

I have another cell that contains "=(16+16+10.2)-(16+16+9.3)", and it returns 0.900000000000006 instead of 0.9 exactly.



Steps to Reproduce:
1. Create a cell that contains "=10.2-9.3".
2. Create a cell that contains "=(16+16+10.2)-(16+16+9.3)".
3.

Actual Results:
Happens in new empty worksheets

Expected Results:
I should be getting exactly 0.9 or 0.900000000


Reproducible: Always


User Profile Reset: No



Additional Info:
Version: 7.1.2.2 (x64) / LibreOffice Community
Build ID: 8a45595d069ef5570103caea1b71cc9d82b2aae4
CPU threads: 6; OS: Windows 10.0 Build 19042; UI render: Skia/Vulkan; VCL: win
Locale: en-US (en_US); UI: en-US
Calc: CL
Comment 1 Aron Budea 2021-05-09 06:00:23 UTC 
This is due to floating point math, and not a bug, for details please see:
https://wiki.documentfoundation.org/Faq/Calc/Accuracy

It is possible to either use rounding functions, or simply set display format to the desired number of decimals.