Bug 86174 - logic error with mod() and int() function
Summary: logic error with mod() and int() function
Status: RESOLVED NOTABUG
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
4.2.4.2 release
Hardware: x86 (IA32) Windows (All)
: medium normal
Assignee: Not Assigned
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-11-12 00:52 UTC by Brian Prasse
Modified: 2014-11-12 01:19 UTC (History)
2 users (show)

See Also:
Crash report or crash signature:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Brian Prasse 2014-11-12 00:52:09 UTC 
Mod(2.01,int(2.01))=0.01 returns false
Comment 1 m.a.riosv 2014-11-12 01:14:31 UTC 
Hi @Brian, thanks for reporting.

Really not a bug, but a hardware limitation, please take a look to https://bugs.freedesktop.org/show_bug.cgi?id=50299#c4.
Comment 2 Robinson Tryon (qubit) 2014-11-12 01:19:29 UTC 
Another symptom of floating-point arithmetic being imperfect.

If you bump-up precision to 20 places, you see:

Value:                  2.01000000000000000000
INT(Value):             2.00000000000000000000
MOD(Value, INT(Value)): 0.00999999999999979000

When you compare 0.01000000000000000000 and 0.00999999999999979000, Calc (correctly) says that they're not the same.

I'm not sure if Calc has a built-in EQUAL_WITHIN_TOLERANCE() function, but you can use this for the purpose:

=(ABS($CELL_OF_FIRST_VALUE - $CELL_OF_SECOND_VALUE) < 0.0000000000001)

Making sure that two values are less than 1e-12 different should be close enough for most purposes :-)

See:
Bug 67026 - Other: Limitations in Calc precision need to be documented