Correction on step 4. The value for the Y-axis should be set to 1.4E+026, not 1.4E+020.

While setting it to 1.4E+020 will still show the problem, 1.4E+026 makes it more obvious.

Thank you for reporting the bug. Please attach a sample document, as this makes it easier for us to verify the bug. 
I have set the bug's status to 'NEEDINFO'. Please change it back to 'UNCONFIRMED' once the requested document is provided.
(Please note that the attachment will be public, remove any sensitive information before attaching it. 
See https://wiki.documentfoundation.org/QA/FAQ#How_can_I_eliminate_confidential_data_from_a_sample_document.3F for help on how to do so.)

Created attachment 135058 [details]
Test Case Chart Document

Attached a Calc document with a dataset and 3 charts showing the issue. The issue is present in the 2^n and n! charts, with Y-axis scaling already set to a value that makes the issue (hopefully) more clear.

Confirmed in

Version: 6.0.0.0.alpha0+
Build ID: 50799a721c7ddcf9475a1b79984ed64ddd7cdf57
CPU threads: 4; OS: Linux 4.8; UI render: default; VCL: gtk3; 
Locale: ca-ES (ca_ES.UTF-8); Calc: group

- Version 4.1.0.0.alpha0+ (Build ID: efca6f15609322f62a35619619a6d5fe5c9bd5a)

- LibreOffice 3.3.0 
OOO330m19 (Build:6)
tag libreoffice-3.3.0.4

and also on Win

Are there any updates regarding this issue?

Has this issue been assigned some sort of priority/estimate? I've not seen any movement and don't even have an idea of the amount of effort it would take or if it's really "low-hanging fruit".

I'm responding to confirm that this bug is still present. The LibreOffice installation is from the Kubuntu 18.10 repository.

Version: 6.1.2.1
Build ID: 1:6.1.2-0ubuntu1.1
CPU threads: 8; OS: Linux 4.18; UI render: default; VCL: gtk3_kde5; 
Locale: en-US (en_US.UTF-8); Calc: group threaded

Guys, this bug is not going to magically disappear. There's a problem when creating charts with large numbers.

It has been 1.5+ years since this got reported and, based on bug report activity, there has been NO meaningful movement here.

If I had to take a guess, it looks like there's an overflow problem, based simply on how the graph is behaving...

If I'm wasting my time here with this, please do let me know, because it's not really fun or productive.

I tested this again and the problem is still present. Version info:

Version: 7.0.3.1
Build ID: 00(Build:1)
CPU threads: 24; OS: Linux 5.8; UI render: default; VCL: x11
Locale: en-US (en_US.UTF-8); UI: en-US
Ubuntu package version: 1:7.0.3-0ubuntu0.20.10.1
Calc: threaded

Very strange indeed, but I think this problem got fixed somewhere, because if I recreate the chart from scratch, it works as expected. However, the problem persist if the attached file is used. So imho this was a chart and export problem, where the wrong values now are stored in the attached file.

Version: 7.2.0.0.alpha0+ (x64) / LibreOffice Community
Build ID: 0e812184825adf7bba178de03ddcdced044d2478
CPU threads: 6; OS: Windows 10.0 Build 19042; UI render: Skia/Raster; VCL: win
Locale: de-DE (de_DE); UI: en-US
Calc: CL

Created attachment 171756 [details]
Comparison of charts

Charts created from scratch, which does not show the behaviour even after saving the document.

I cannot reproduce the error in

Version: 6.0.2.1 (x64)
Build-ID: f7f06a8f319e4b62f9bc5095aa112a65d2f3ac89
CPU-Threads: 6; BS: Windows 10.0; UI-Render: Standard; 
Gebietsschema: de-DE (de_DE); Calc: CL

when creating such a chart from scratch. May you remember with which version you created these charts?

I tried to check the xml contents of both charts and this line seems to be the culprit:

WRONG Chart:
<style:chart-properties chart:interpolation="cubic-spline" chart:include-hidden-cells="false" chart:auto-position="true" chart:auto-size="true" chart:treat-empty-cells="ignore" chart:right-angled-axes="true"/>

FRESH RIGHT Chart:
<style:chart-properties chart:symbol-type="automatic" chart:include-hidden-cells="false" chart:treat-empty-cells="ignore" chart:right-angled-axes="true"/>

They only differ in the chart:interpolation="cubic-spline" attribute. Maybe you have added some interpolation?

(In reply to Andreas Heinisch from comment #16)
> May you remember with which version you created these charts?

I don't remember a specific version, other than I was using whatever the latest stable release for Ubuntu was (via apt) at the date/time of reporting. There's a reply from someone confirming the issue the day after my report here on comment 6 (https://bugs.documentfoundation.org/show_bug.cgi?id=110993#c6).

It has version info for an alpha at the time, so, presumably, a stable Ubuntu 17.04 release before that. I hope that helps narrow it down and I apologize for failing to include version info in the initial report. I didn't expect it'd take so long before someone gave some attention to an issue like this.

Also, it's a bit concerning that the error would be stored within the document requiring it to be redone from scratch? (I guess it's not just a rendering issue.) For less trivial documents, this could be a more serious problem.

> Maybe you have added some interpolation?

No, I don't recall adding that manually nor a reason for wanting to do so, either. This is simply a Big-O notation table, so it's not meant to have any fancy effects beyond the basic graph.

Now I could reproduce the problem from scratch.

If you choose the smooth line type in the scatter chart dialog with cubic spline interpolation, you get the wrong result. So there is for sure a problem with this interpolation method. I try to investigate further on this problem. Maybe I can provide a fix.

(In reply to Andreas Heinisch from comment #18)
> Now I could reproduce the problem from scratch.

In what version, the older one closer to the report date or the latest?


> If you choose the smooth line type in the scatter chart dialog with cubic spline interpolation, you get the wrong result.

Maybe there's an option that causes/d that as a consequence/side-effect in my older document? Did your finding produce the same internal XML as my test document or does it seem to be a different way of reproducing the same result (i.e., same destination, but different road)?

> So there is for sure a problem with this interpolation method. I try to investigate further on this problem. Maybe I can provide a fix.

Thanks and good luck. If there's anything else I can provide assistance with, let me know, and I'll try to help out if it's within my means.

The problem arises here: https://opengrok.libreoffice.org/xref/core/chart2/source/view/charttypes/Splines.cxx?r=e871c9cb#398 as a consequence of CalculateCubicSplines in https://opengrok.libreoffice.org/xref/core/chart2/source/view/charttypes/Splines.cxx?r=e871c9cb#534.

The calculation is split into x and y coordinates, and I have to clarify if this can be done for the derivatives of the function, because the x values have always the same size.

Created attachment 171875 [details]
Test Build Image

I have revised the SplineCalculater::CalculateCubicSplines and this is the result. It turned out that the CubicSpline method had some logical as well as an algorithm error. I hope I can send it to gerrit and someone can review it :)

After rechecking the calculation, I committed a patch to gerrit in order to correct a small error in the spline calculation. However, this does not solve the problem of this bug report.

The calculation of the spline should be correct, but the autoclipping to the chart does not work. I tried to recalc the min and max values for the chart in https://opengrok.libreoffice.org/xref/core/chart2/source/view/charttypes/AreaChart.cxx?r=4f6931d6#368 using:

std::vector< ExplicitScaleData > aScales( m_pPosHelper->getScales());
aScales[1].Minimum = *(std::min_element( aPoly.SequenceY[0].begin(), aPoly.SequenceY[0].end()));
aScales[1].Maximum = *(std::max_element(aPoly.SequenceY[0].begin(), aPoly.SequenceY[0].end()));
m_pPosHelper->setScales(aScales, m_pPosHelper->isSwapXAndY());

So the real bug here is that the automatic scaling of charts using cubic splines does not work. Unfortunately, I don't understand enough in order to solve this issue, but maybe someone with more insight in the chart creation process may continue.

Andreas Heinisch committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/e6442b39836f9856aa7b87d1a840158f0cb7d9c4

tdf#110993 - Corrected spline calculation

It will be available in 7.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.

Andreas Heinisch committed a patch related to this issue.
It has been pushed to "libreoffice-7-2":

https://git.libreoffice.org/core/commit/d3280eb3b47472ab3fc6946d475a7b9c6c55c381

tdf#110993 - Corrected spline calculation

It will be available in 7.2.0.0.beta2.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.

Created attachment 176554 [details]
side-by-side LibreOffice and R graphs for 2^n cubic spline

I have a very limited understanding of cubic splines, but I feel like this might be "not a bug".

I have tested building a cubic spline with R and plotting it with ggplot2 for comparison with what LibreOffice does. Attached is the result: both plots look the same.
The R code makes sure similar axis breaks are used. 500 values were interpolated to build the spline curve, using the spline() function and the method "normal".

R's spline() functions has various methods available. Most of them (fmm, periodic, natural) will generate negative values to fit a spline to this data. However, one method (hyman) will only generate values above 0 and be visualised in the hockey stick look (i.e. shooting up only towards the end, after the x = 80 mark).

As I understand it, LibreOffice's cubic spline smoother uses a normal cubic spline, whereas xghost was expecting a different kind. Arguably, the exact kind of interpolation used should be properly documented in our help pages.

xghost, are you able to produce a comparison with a different tool, or a reference that supports that LibreOffice's calculation is fundamentally wrong?

>> xghost, are you able to produce a comparison with a different tool, or a reference that supports that LibreOffice's calculation is fundamentally wrong?

I didn't spend my time trying to replicate the same result in a different tool because:

1. a different tool may have the same problem or may actually work correctly, so its behavior is irrelevant to LibreOffice's;
2. The result itself shows that LibreOffice's behavior is incorrect.

Based on point #2, what other evidence could you possible want beyond a test case chart that shows that the chart is wrong for the numbers given? Whether the error is in the calculation itself or somewhere else (e.g. the displaying of the charts) is not something I can know - or have claimed. The only thing I can say is that we can observe the numbers for the charts and the charts themselves and see that it's not behaving correctly.


PS: I also re-checked the test case document I had originally attached and can confirm that the problem is still present. The version information for this re-test follows:

Version: 7.3.3.2 / LibreOffice Community
Build ID: 30(Build:2)
CPU threads: 24; OS: Linux 5.15; UI render: default; VCL: kf5 (cairo+xcb)
Locale: en-US (en_US.UTF-8); UI: en-US
Ubuntu package version: 1:7.3.3-0ubuntu0.22.04.1
Calc: threaded

The code that defines how splines are calculated mention that it follows the ODF specification 1.2: https://opengrok.libreoffice.org/xref/core/chart2/source/view/charttypes/Splines.cxx?r=e871c9cb#532

The relevant section of the ODF 1.2 standard is section 20.26 chart:interpolation : http://docs.oasis-open.org/office/v1.2/os/OpenDocument-v1.2-os-part1.html#__RefHeading__1419480_253892949
(first dot point is for b-splines, second dot point is for cubic splines)

What needs to be checked is that the calculation is implemented in LibreOffice does indeed follow the standard (or maybe if it needs to be updated to a more recent ODF specification, if the method has changed since 1.2).

As stated in my comment 25, there are different methods to compute a cubic spline. LibreOffice seems to be method called "normal" in R, in which there will be negative values extrapolated to fit the data you are using.

It seems it is expected to have negative values extrapolated in some cases, to fit a smooth line to the data. For example:
- https://www.wavemetrics.com/forum/general/spline-interpolation-introduces-negative-values
- https://stackoverflow.com/questions/56903631/interpolating-using-a-cubic-function-gives-a-negative-value-for-probability#comment100352184_56903631
- natural cubic splines to fit a sequence of number between 0 to 1, negative values created depending on number of degrees of freedom: https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-019-0666-3/figures/4
- https://stackoverflow.com/a/51426181/1494531

I know Regina commented on that section of the specification. Would you be interested in this particular issue, Regina?

(In reply to Stéphane Guillou (stragu) from comment #28)
> 
> I know Regina commented on that section of the specification. Would you be
> interested in this particular issue, Regina?

No, sorry. It is long ago that I have worked on interpolation. I have the needed mathematics no longer present and will not dig in it again. I want to focus on problems in Draw.

Clarified the summary to include "cubic spline".

But I stand by my comment 25 and comment 28: there are various ways to calculate cubic splines, and the method LibreOffice uses is consistent with the "Normal" method. Other alternative methods will also extrapolate this kind of data to negative values.
The use of R for comparison is not a random choice: it's a statistical programming language used very widely.

If something needs to be checked, it's the consistency between the LibreOffice code and the ODF standard (i.e. are we using the method that the standard defines?).

In summary, my opinion is "resolved - not a bug" because negative values are expected in some (most?) cubic spline methods applied to this data.

(In reply to Stéphane Guillou (stragu) from comment #30)
> In summary, my opinion is "resolved - not a bug" because negative values are
> expected in some (most?) cubic spline methods applied to this data.

So, in summary, the wrong behavior is "not a bug" because the wrong behavior is expected? How does knowing that the behavior is wrong make it "not wrong"? As a user, that does not make sense to me.

(In reply to xghost from comment #31)
> (In reply to Stéphane Guillou (stragu) from comment #30)
> > In summary, my opinion is "resolved - not a bug" because negative values are
> > expected in some (most?) cubic spline methods applied to this data.
> 
> So, in summary, the wrong behavior is "not a bug" because the wrong behavior
> is expected? How does knowing that the behavior is wrong make it "not
> wrong"? As a user, that does not make sense to me.

To put it clearly: my understanding is that the behaviour is *not* wrong.

(In reply to Stéphane Guillou (stragu) from comment #32)
> To put it clearly: my understanding is that the behaviour is *not* wrong.

But the result is clearly wrong though. How does it make sense to claim that the behavior producing that result isn't?

That is no error in the implementation but a property of cubic splines. You see negative values and not being monotonic already with much smaller values. Try
(0|1) (3|8) (6|64) (9|512) (12|4096)
The site https://tools.timodenk.com/?p=cubic-spline-interpolation has an online calculator. Try it there and compare with LibreOffice.

You get a more appealing result, when you use smaller x-distances if the y-differences change too rapidly otherwise. For 2^x for example
(0|1) (5|32) (9|512) (11|2048) (12|4096)

For me it is a resolve as "Not a bug".

(In reply to Regina Henschel from comment #34)
> That is no error in the implementation but a property of cubic splines. [...] For me it is a resolve as "Not a bug".

Objectively speaking, the behavior is wrong. Any user can see this. If the issue is caused by the fact that "cubic splines" are being used, then doesn't that imply that, maybe, something more robust should be used instead?

That would be an enhancement request. Solving it requires not only implementing a new method of interpolating but in addition extending the ODF standard.

When writing an enhancement request it would be useful if it points to a suitable method and to a proof, that the method solves the problem.

(In reply to Regina Henschel from comment #36)
> That would be an enhancement request. Solving it requires not only
> implementing a new method of interpolating but in addition extending the ODF
> standard.
> 
> When writing an enhancement request it would be useful if it points to a
> suitable method and to a proof, that the method solves the problem.

I don't think that kind of expectation is reasonable for the application user. It might be for reasonable for the subject matter experts, i.e. app developers, but not the regular users.

> That would be an enhancement request. Solving it requires not only implementing a new method of interpolating but in addition extending the ODF standard.

Also, I disagree that this should be considered an enhancement request. Even if there's an argument that the standard is implemented correctly, which it very well might be, the correct implementation of a specification/standard that is itself fundamentally flawed is still a bug --it's just one that exists at the design level rather than the implementation level.

Please don't reopen this.

> Please don't reopen this.

Perhaps not closing this under a bad pretense would've been a better course of action. I'm done wasting my time here and the system in general. You can pat yourselves on the back.

xghost, both Regina and I have provided evidence (tested other tools, linked to documentation on how a cubic spline interpolation can result in negative values even if dataset only has positive values...) that suggest that this works as expected. Your response to the evidence we provide is "it's obviously wrong".

If the interpolation method is not the right one for you, please use another one that is already available in LibreOffice, or open a new ticket suggesting the addition of a new interpolation method, as kindly suggested by Regina.

I do however take this ticket as a need to improve our documentation to be explicit about which method is used for cubic spline interpolation.

All the best.

(In reply to Stéphane Guillou (stragu) from comment #41)
> both Regina and I have provided evidence (tested other tools, linked to documentation on how a cubic spline interpolation can result in negative values even if dataset only has positive values...) that suggest that this works as expected.

Stéphane, the only things the evidence you've both provided shows is:

1. that other systems that chose to implement the same method have the same problems, b/c the problem is the chosen method itself --even when the implementation itself is correct;
2. that you already had prior knowledge of the fact that, even when implemented correctly, the method itself behaves incorrectly in some cases.

I hope it's not asking too much for you to understand that having prior knowledge of something behaving incorrectly does not magically make the incorrect behavior "correct" or "not a bug".

At the expense (re)stating what should be plainly obvious, imagine, as an example, if car manufacturers attempted to use your own line of reasoning for safety recalls: "Your honor, we already knew that our wiring methods were a fire hazard, and some cars actually did catch on fire; however, other car manufacturers have the same problem and we expected that to be a fire hazard. Therefore, there's no fire hazard." TL;DR: *Insert this-is-fine-dog meme*

It's a word game where the meaning of "correct behavior" is redefined so that it includes incorrect behavior as part of it. Can't go wrong with that approach ¯\_(ツ)_/¯

> Your response to the evidence we provide is "it's obviously wrong".

Because it is. I'm not going to publish an academic research paper or an elaborate argument on why something that gives a wrong result is "obviously wrong". That's self-evidently true and I'd rather not resort to having to use simpler language if I can avoid it.

Your prior knowledge of the incorrect behavior does not make it correct. If you were to be given a calculator that's expected to say `2 + 2 != 4`, which is a result that's "Obviously Wrong"(tm), claiming that it's "correct" simply b/c it was already known, and expected, to give a wrong results would be not only unreasonable, but absurd --and yet, that's precisely the kind of "argument" that you've both provided. I hope you can understand my level of skepticism at your line of reasoning in this thread.

> If the interpolation method is not the right one for you, please use another one that is already available in LibreOffice

I guess I will have to look into that, b/c I was not aware that there were other interpolation methods in there. I can only cross my fingers that it won't be a repeat of this thread for some other method X.

> or open a new ticket suggesting the addition of a new interpolation method, as kindly suggested by Regina.

The "kind suggestion" basically asks for a new method and a proof[1]. I'm not a mathematician, and even if I were one, you'd still have to forgive me for not being able to come up with a new mathematical method for line interpolation, a theoretical proof for it, and then another proof that it actually works within a computer on-demand.

I'd be more understanding of your position if your response had been something like "You know, it's a known limitation. We really don't know how to solve that and, to the best of our knowledge, there's no other known line interpolation method that could be implemented to address this issue [...]", or something communicating the a similar idea. However, that was never your response; rather, it was "this is correct and there's no problem here".


[1] "When writing an enhancement request it would be useful if it points to a suitable method and to a proof, that the method solves the problem."


> I do however take this ticket as a need to improve our documentation to be explicit about which method is used for cubic spline interpolation.

Of course... ¯\_(ツ)_/¯

Regards.