Bug 90419 - Accuracy of calculation in Fill
Summary: Accuracy of calculation in Fill
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
Inherited From OOo
Hardware: All All
: medium enhancement
Assignee: Eike Rathke
QA Contact:
URL:
Whiteboard: target:5.3.0 target:5.2.0.1 target:5.1.4
Keywords:
Depends on:
Blocks:
 
Reported: 2015-04-02 19:05 UTC by (OPTIONAL) Real Name:
Modified: 2017-01-02 22:34 UTC (History)
3 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description (OPTIONAL) Real Name: 2015-04-02 19:05:45 UTC
If you drag decimal numbers eg 0.12 and 0.11 over a few lines/spans, it starts showing unnessary decimals places at the end like in 0,0900000000000004, which is really odd to anyone who doesn't know what a floating point type is, especially because you can type in eg. 0.09 without the same decimal places beeing shown.

PS: http://www.h-schmidt.net/FloatConverter/IEEE754.html, I had to cringe looking at the floating point representation of 0.14 and 0.13, I doubt that anyone will fix that bug soon, but as it is working for pairs like (10.14, 10.13) and (1.14,1.13) there is still hope. Maybe, IDK.
Comment 1 Eike Rathke 2015-04-02 19:45:35 UTC
While generally all floating point numbers that are not exactly representable in an IEEE 754 double lose precision, this specific case of filling a series of numbers could be improved by using multiplication instead of incrementing (adding delta) values.

Actually this shouldn't be too hard to solve, I'm creating an EasyHack of this.
Code pointer is sc/source/core/data/table4.cxx ScTable::Fill() and subroutines.
Comment 2 Robinson Tryon (qubit) 2015-12-14 05:03:42 UTC
Migrating Whiteboard tags to Keywords: (EasyHack DifficultyInteresting SkillCpp)
[NinjaEdit]
Comment 3 Jaskaran Singh 2016-02-11 18:27:47 UTC
(In reply to Eike Rathke from comment #1)
> While generally all floating point numbers that are not exactly
> representable in an IEEE 754 double lose precision, this specific case of
> filling a series of numbers could be improved by using multiplication
> instead of incrementing (adding delta) values.
> 
> Actually this shouldn't be too hard to solve, I'm creating an EasyHack of
> this.
> Code pointer is sc/source/core/data/table4.cxx ScTable::Fill() and
> subroutines.
Comment 4 Jaskaran Singh 2016-02-12 09:38:22 UTC
(In reply to Eike Rathke from comment #1)
> While generally all floating point numbers that are not exactly
> representable in an IEEE 754 double lose precision, this specific case of
> filling a series of numbers could be improved by using multiplication
> instead of incrementing (adding delta) values.
> 
> Actually this shouldn't be too hard to solve, I'm creating an EasyHack of
> this.
> Code pointer is sc/source/core/data/table4.cxx ScTable::Fill() and
> subroutines.

How would multiplication result in incrementing the value? And how would that result in less error?Any Mathematical clues would be nice. And also , how to reproduce this bug?
Comment 5 Piet van Oostrum 2016-02-12 10:13:56 UTC
(In reply to Eike Rathke from comment #1)
> While generally all floating point numbers that are not exactly
> representable in an IEEE 754 double lose precision, this specific case of
> filling a series of numbers could be improved by using multiplication
> instead of incrementing (adding delta) values.

Actually, this is already how it is done.
In ScTable::FillSeries:

        case FILL_LINEAR:
            {
                //  use multiplication instead of repeated addition
                //  to avoid accumulating rounding errors
                nVal = nStartVal;
                double nAdd = nStepValue;
                if ( !SubTotal::SafeMult( nAdd, (double) ++nIndex ) ||
                        !SubTotal::SafePlus( nVal, nAdd ) )
                    bError = true;
            }

However, depending on the initial values, this may still give rise to "unexpected" values (that is unexpected with regard to the "intuitive" expectations). But that's life with floating point numbers.
Comment 6 Robinson Tryon (qubit) 2016-02-18 14:52:17 UTC
JanI is default CC for Easy Hacks (Add Jan; remove LibreOffice Dev List from CC)
[NinjaEdit]
Comment 7 Eike Rathke 2016-02-23 12:09:33 UTC
(In reply to Piet van Oostrum from comment #5)
> Actually, this is already how it is done.

Indeed. Though in this case (0.12,0.11,...) ScTable::FillSeries() already gets called with nStepValue=-0.009999999999999995 for which at some points the rounding error doesn't get eliminated.
So finding out if and how determining the step value could be improved might help, or if some additional treatment between SafeMult() and SafePlus() would be possible.
Comment 8 jani 2016-04-18 07:34:49 UTC
A polite ping, still working on this issue?
Comment 9 Jaskaran Singh 2016-04-18 13:36:49 UTC
(In reply to jan iversen from comment #8)
> A polite ping, still working on this issue?

Sorry . Wasn't working on this bug. I have unassigned this from myself.
Comment 10 Commit Notification 2016-06-13 11:54:35 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e89c0e4fb783bd36d5f5fea154ee8608e542dae4

Resolves: tdf#90419 diminish precision error in Series Fill

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 11 Commit Notification 2016-06-13 14:28:26 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "libreoffice-5-2":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=c53b0c44e77e0622487cd12ae1fde9acb17f77da&h=libreoffice-5-2

Resolves: tdf#90419 diminish precision error in Series Fill

It will be available in 5.2.0.1.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 12 Eike Rathke 2016-06-13 14:29:22 UTC
Pending review https://gerrit.libreoffice.org/26220 for 5-1
Comment 13 Commit Notification 2016-06-14 19:05:18 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "libreoffice-5-1":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=191100ff3ab38f6487d0da3b23b8544175b61d33&h=libreoffice-5-1

Resolves: tdf#90419 diminish precision error in Series Fill

It will be available in 5.1.5.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 14 Commit Notification 2016-06-14 19:55:51 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "libreoffice-5-1-4":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=c6eb4c3189ff87aa273f172731c0f23e8854a062&h=libreoffice-5-1-4

Resolves: tdf#90419 diminish precision error in Series Fill

It will be available in 5.1.4.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.