Bug 144052 - Calc hangs when I try insert a chart for 800000 cells
Summary: Calc hangs when I try insert a chart for 800000 cells
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
7.1.0.3 release
Hardware: x86-64 (AMD64) All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard: target:7.3.0
Keywords: perf
Depends on:
Blocks: Chart
  Show dependency treegraph
 
Reported: 2021-08-24 11:19 UTC by Mark Dunn
Modified: 2021-09-12 17:33 UTC (History)
5 users (show)

See Also:
Crash report or crash signature:


Attachments
Flamegraph (276.80 KB, application/x-bzip)
2021-08-24 21:21 UTC, Julien Nabet
Details
Example file (3.03 MB, application/vnd.ms-excel)
2021-08-31 11:35 UTC, Telesto
Details
Example file (2.38 MB, application/vnd.oasis.opendocument.spreadsheet)
2021-08-31 11:36 UTC, Telesto
Details
perf flamegraph (471.47 KB, application/x-bzip)
2021-09-09 19:57 UTC, Julien Nabet
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mark Dunn 2021-08-24 11:19:55 UTC
Description:
I selected "insert/chart" instead of "insert/pivot table" on a spreadsheet of 100,000 rows by 8 columns. Calc locked up (two threads taking turns at 100%) for over three minutes. I wanted to cancel the operation but the UI was unresponsive.

terminating the app was the only way out

Steps to Reproduce:
1.use a large spreadsheet 100,000 by 8
2.Select all
3.Select "insert/chart"


Actual Results:
UI was locked so I was unable to cancel my mistake

Expected Results:
UI should allow me to cancel a long running job


Reproducible: Always


User Profile Reset: No



Additional Info:
UI should always available.
Comment 1 Roman Kuznetsov 2021-08-24 14:15:01 UTC
Confirm in

Version: 7.3.0.0.alpha0+ (x64) / LibreOffice Community
Build ID: ac80ec817eb07c77a51bc0729985a473c734182e
CPU threads: 8; OS: Windows 10.0 Build 19043; UI render: Skia/Vulkan; VCL: win
Locale: ru-RU (ru_RU); UI: ru-RU
Calc: threaded

and 7.1 too


Julien, could you please create a perfgraph for this problem?
Comment 2 m.a.riosv 2021-08-24 20:40:21 UTC
What can we expect?. A graph with 100000 points and seven data series.

Having an option to cancel the actual operation, looks fine, but maybe with a high performance cost.
Comment 3 Julien Nabet 2021-08-24 21:21:21 UTC
Created attachment 174525 [details]
Flamegraph

On pc Debian x86-64 with master sources updated today, I selected 8 columns and 100 000 rows, then did "Insert/chart" then waited until the dialog box appeared.
Here's the resulting Flamegraph
Comment 4 Telesto 2021-08-30 12:34:14 UTC
@Noel, 
Perf, so you might be interested..
Comment 5 Mark Dunn 2021-08-31 10:27:30 UTC
So to be clear.

I download: 
    https://api.coronavirus.data.gov.uk/v2/data?areaType=region&metric=newCasesBySpecimenDateAgeDemographics&format=csv

every day, which as you say is unwieldy

I then select the sheet and "insert/pivot" to obtain a much more wieldy table of cases (Area Name by Date by Age Group if you are interested) 

Currently that is 20 * 578 cells

It is a reasonable thing for calc to do quickly.

But sometimes I have missed "insert/pivot table" and hit "insert/chart" by mistake. (I use "insert/chart" a lot in my work.)

It is the consequence of this very easy mistake which is the problem. Forcing me to crash out of LibreOffice loses all the work in ALL documents for up to 10 minutes the last auto-save. In my case this is three spreadsheets and a document.

It seems that miguelangelrv already knows that this will cause calc problems with his developer-esque comment 
    "What can we expect?. A graph with 100000 points and seven data series."

If you do not know how to 
    a) keep the UI responsive and
    b) insert an asynchronous cancel
without affecting performance, then maybe you should query the user's intent before embarking on an endless?! task. 

NOTE: Performing this same task on a slightly less powerful machine in Excel 2010 produced a chart in 4 seconds.
Comment 6 Telesto 2021-08-31 11:35:46 UTC
Created attachment 174666 [details]
Example file

1. open the attached file
2. Select both columns & press Chart.. 

Aside from the whole speed issue. Why does the wizard start rendering a chart before configuration? If select some weird range, you have the cope with lots of waiting time.. 

Why does the chart recalculate.. again and again and again.. Is there no cache? Its recalculating the chart. On selecting the chart, on on resize of the chart

Even dragging the Calc Window around with chart visible causes strain..
Comment 7 Telesto 2021-08-31 11:36:36 UTC
Created attachment 174667 [details]
Example file
Comment 8 Telesto 2021-08-31 11:38:56 UTC
The drawing problem is apparently related to B2DPolyPolygon

RtlAllocateHeap
RtlAllocateHeap
malloc_base
package_OStorageFactory_get_implementation
basegfx::B2DPolyPolygon::B2DPolyPolygon
basegfx::B2DPolyPolygon::makeUnique
basegfx::B2DPolyPolygon::transform
com_sun_star_comp_graphic_PrimitiveFactory2D_get_implementation
drawinglayer::primitive2d::BufferedDecompositionPrimitive2D::get2DDecomposition
drawinglayer::primitive2d::BasePrimitive2D::getB2DRange
drawinglayer::primitive2d::getB2DRangeFromPrimitive2DReference
drawinglayer::primitive2d::Primitive2DContainer::getB2DRange
drawinglayer::primitive2d::BasePrimitive2D::operator==
drawinglayer::primitive2d::BasePrimitive2D::getB2DRange
drawinglayer::primitive2d::getB2DRangeFromPrimitive2DReference
drawinglayer::primitive2d::Primitive2DContainer::getB2DRange
drawinglayer::primitive2d::BasePrimitive2D::operator==
drawinglayer::primitive2d::BasePrimitive2D::getB2DRange
drawinglayer::primitive2d::getB2DRangeFromPrimitive2DReference
drawinglayer::primitive2d::Primitive2DContainer::getB2DRange
drawinglayer::primitive2d::BasePrimitive2D::operator==
drawinglayer::primitive2d::BasePrimitive2D::getB2DRange
drawinglayer::primitive2d::getB2DRangeFromPrimitive2DReference
drawinglayer::primitive2d::Primitive2DContainer::getB2DRange
drawinglayer::primitive2d::BasePrimitive2D::operator==
drawinglayer::primitive2d::BasePrimitive2D::getB2DRange
drawinglayer::primitive2d::getB2DRangeFromPrimitive2DReference
drawinglayer::primitive2d::Primitive2DContainer::getB2DRange
drawinglayer::primitive2d::BasePrimitive2D::operator==
drawinglayer::primitive2d::BasePrimitive2D::getB2DRange
drawinglayer::primitive2d::getB2DRangeFromPrimitive2DReference
drawinglayer::primitive2d::Primitive2DContainer::getB2DRange
ChartHelper::tryToGetChartContentAsPrimitive2DSequence
ChartHelper::operator=
SdrObject::IsResizeProtect
sdr::contact::ViewObjectContact::getPrimitive2DSequence
SdrObjectPrimitiveHit
SdrMarkView::CheckSingleSdrObjectHit
Comment 9 Commit Notification 2021-09-08 08:54:18 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/18d6373d3ae767d3f06d53acd3b1f88f008c4bf4

reduce cost of creating large charts (tdf#144052)

It will be available in 7.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 10 Commit Notification 2021-09-09 11:02:11 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/7b4b032daa2381e5c2f102d4e9619adf38968f94

reduce time in ResetEvent (tdf#144052)

It will be available in 7.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 11 Commit Notification 2021-09-09 11:02:30 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/a9cd44f8deeccbb8afc7209fdf86fbc26ea7a464

tdf#144052 speedup inserting large charts

It will be available in 7.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 12 Commit Notification 2021-09-09 13:23:26 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/8104d771b10a5c8b15eae4b67aa112ae2ef77b5b

tdf#144052 reduce Primitive2D copying

It will be available in 7.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 13 Julien Nabet 2021-09-09 19:57:52 UTC
Created attachment 174933 [details]
perf flamegraph

Here's a Flamegraph retrieved on pc Debian x86-64 with master sources updated today (8104d771b10a5c8b15eae4b67aa112ae2ef77b5b) + gen rendering on Telesto's file.
Comment 14 Mark Dunn 2021-09-12 17:33:15 UTC
So I removed libreoffice
    sudo apt-get remove LibreOffice*
    sudo apt autoremove

I then downloaded
    https://dev-builds.libreoffice.org/daily/master/Linux-rpm_deb-x86_64@86-TDF/2021-09-10_21.28.22/master~2021-09-10_21.28.22_LibreOfficeDev_7.3.0.0.alpha0_Linux_x86-64_deb.tar.gz

and Installed with 
    sudo dpkg -i *.deb

Then ran the app with 
    libreofficedev7.3

And checked the verson
    Version: 7.3.0.0.alpha0+
    (https://gerrit.libreoffice.org/gitweb?p=core.git;a=log;h=84e1c79dd7394168459a3bbdea8bd94d765708e0)
    Environment: CPUThreads:8 OS:Linux 5.11

Then downloaded the file 
    https://api.coronavirus.data.gov.uk/v2/data?areaType=region&metric=newCasesBySpecimenDateAgeDemographics&format=csv

Imported the csv file (with commas and YMD type on the appropriate column)

Selected the Sheet (left clicked between '1' and 'a')
Selected Insert/Chart from the menu
    The app was switching between two CPUs and using memory
    after about 2 minuted, it stopped using up memory, but continued switching
    at 5 minutes cacelled the app

Let me know if I tested it incorrectly or you need further info.