Bug 138424 - calc: calculation: rounding of results: bad influence of unspecific rounding if terms in different dyadic ranges
Summary: calc: calculation: rounding of results: bad influence of unspecific rounding ...
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
3.5.1 release
Hardware: All All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: Calc-Cells
  Show dependency treegraph
 
Reported: 2020-11-23 07:37 UTC by b.
Modified: 2024-12-03 15:15 UTC (History)
1 user (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description b. 2020-11-23 07:37:07 UTC
Description:
calc doe's some 'tie to zero' rounding, acc. @erAck intended for integer values but imho affecting all calculations, 

this rounding sometimes has evil effects, e.g. '=(2^53+31-2^53)+(2^53-31-2^53)+(2^53+31-2^53)+(2^53-31-2^53)' should mathematical result in '0' (the subterms in brackets cancel each other out), 'fp-math' calculates '2' because above 2^53 IEEE 754 is limited to the 'even integers' and some small rounding steps in, calc calculates '64' reg. unneccessary rough rounding and either different effect of it below and above a magnitude range border (2^53), or different effect for '+' vs. '-', 

Steps to Reproduce:
1. calculate '=(2^53+31-2^53)+(2^53-31-2^53)+(2^53+31-2^53)+(2^53-31-2^53)'
2.
3.

Actual Results:
64

Expected Results:
'0' or maximal tolerable '2', 


Reproducible: Always


User Profile Reset: No



Additional Info:
Version: 7.1.0.0.alpha1+ (x64)
Build ID: b61bf7c7cfcf97a5ade6d130873af146670bc2ee
CPU threads: 8; OS: Windows 6.1 Service Pack 1 Build 7601; UI render: default; VCL: win
Locale: de-DE (de_DE); UI: en-US
Calc:
Comment 1 Eike Rathke 2020-11-30 23:19:23 UTC
(In reply to b. from comment #0)
> calc doe's some 'tie to zero' rounding, acc. @erAck intended for integer
> values but imho affecting all calculations, 
I never said the tie to zero was intended for integer values.

Otherwise agree, that should be reworked.
Comment 2 b. 2020-12-02 20:57:30 UTC
hello @erAck, 

sorry, maybe i (or you?) was lost in the nested 'non' and 'not's of: 

'Anyway, you hit an effect of the approxAdd() function used to normally eliminate accuracy limitations when adding/subtracting not non-ambiguous representable integer values with opposite signs of similar magnitudes by scaling to 2^-48 and thus rounding off the least significant 4 bits to yield a result of 0.0, i.e. to let 0.3-0.2-0.1==0.0.', 

cited acc. https://ask.libreoffice.org/en/question/274247/calc-wrong-calculation-would-like-a-recheck/,
Comment 3 QA Administrators 2022-12-03 03:37:29 UTC Comment hidden (obsolete)
Comment 4 QA Administrators 2024-12-03 03:11:24 UTC Comment hidden (obsolete)
Comment 5 b. 2024-12-03 15:15:44 UTC
result still 64 

with ver.:  
Version: 24.2.5.2 (X86_64) / LibreOffice Community  
Build ID: 420(Build:2)  
CPU threads: 8; OS: Linux 6.8; UI render: default; VCL: x11  
Locale: en-US (en_US.UTF-8); UI: en-US  
Debian package version: 4:24.2.5-4  
Calc: threaded