Description: When a call calculates a number that is too large, it shows and uses 0 instead of reporting and error. I have attached a screenshot comparing this with the correct behaviour in MS Excel. I have also attached the Calc ODS and Excel files. The formula is x^((p-1)!) mod p ... where x^((p-1)!) grows too large when x=5 and p=5, that is 5^(24). Steps to Reproduce: See attached spreadsheets and screenshot. Actual Results: The apparent result of the calculation is 0, and the subsequent calculations are false. Expected Results: Instead of 0, an error should be reported. Reproducible: Always User Profile Reset: No Additional Info: We accept that accuracy is limited by floating point representation and calculations. But this is different. This is falsely assigning 0 to an error.
Created attachment 205009 [details] ODS file with error
Created attachment 205010 [details] excel file showing proper error handling
Created attachment 205011 [details] screenshot comparing Calc and Excel
Example: 6^24 results in 4738381338321620000 instead of 4738381338321616896. It is rounded to 15 significant digests. And the rounded value mod 5 is zero. LibreOffice has no special integer arithmetic but uses the precision of "double" data type. And from its 52 significant bits you can get only about 15 significant digits in a decimal representation of the number. Thus it is not an "overflow" problem but a problem of precision. LibreOffice does not consider this as bug. See https://help.libreoffice.org/26.8/en-US/text/scalc/01/calculation_accuracy.html The same precision problem occurs when calculating MOD(1.5^n; 5). It is only accurate for n=1,2,3,...,14. For integers greater than 14, Excel does not display an error message in this case. This is inconsistent in Excel.
Thanks Regina - I can see now this is due to accuracy and not overflow. Ideally the user would be informed but I suspect that is not easy to do, as it might are with a true overflow. Thanks