Bug 125254 - Performance: A spreadsheet opens too slow on 64 bit OS and causes crash on 32 bit OS
Summary: Performance: A spreadsheet opens too slow on 64 bit OS and causes crash on 32...
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
6.2.4.1 rc
Hardware: All All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard: target:6.3.0
Keywords: perf
Depends on:
Blocks: File-Opening Memory
  Show dependency treegraph
 
Reported: 2019-05-13 08:03 UTC by Mike Kaganski
Modified: 2022-09-08 06:43 UTC (History)
6 users (show)

See Also:
Crash report or crash signature:
Regression By:


Attachments
A sample spreadsheet from https://ask.libreoffice.org/en/question/193663 (18.88 MB, application/vnd.oasis.opendocument.spreadsheet)
2019-05-13 08:03 UTC, Mike Kaganski
Details
massif allocation report (118.60 KB, image/png)
2019-05-13 18:52 UTC, Noel Grandin
Details
perf flamegraph (259.12 KB, application/x-bzip)
2020-09-07 14:35 UTC, Julien Nabet
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mike Kaganski 2019-05-13 08:03:52 UTC
Created attachment 151348 [details]
A sample spreadsheet from https://ask.libreoffice.org/en/question/193663

The attachment opens longer than in 220 s on my system. Hard recalc takes only ~4 s - so it looks like it makes multiple unnecessary recalcs during load, instead of a single one after everything loaded?
Comment 1 Roman Kuznetsov 2019-05-13 11:10:10 UTC
I got a crash when try open file from attach in

Version: 6.3.0.0.alpha0+
Build ID: 773ac3abbf8ab1343367e51b1774d2ee1f8c4f49
CPU threads: 4; OS: Windows 6.1; UI render: default; VCL: win; 
Locale: ru-RU (ru_RU); UI-Language: en-US
Calc: threaded

and LO takes over 1,5Gb of memory before crash

Noel, may be you'll can look at this problem?
Comment 2 Drew Jensen 2019-05-13 13:52:23 UTC
Is it possible that the links to non-existent external file data is part of the problem. The number of cells with a formula referencing that missing external file is huge, over one million.
Comment 3 Julien Nabet 2019-05-13 15:35:07 UTC
Mike: Would a perf/Flamegraph help here or do you have already an idea how to fix this?
Comment 4 Noel Grandin 2019-05-13 18:52:42 UTC
Created attachment 151376 [details]
massif allocation report
Comment 5 Noel Grandin 2019-05-13 18:53:37 UTC
So I can see that we are allocating 1G of string data to store formulas, and I can see where we are allocating them, but I don't know who is owning this memory and keeping it alive (since it is ref-counted stuff and passed around a lot)
Comment 6 Mike Kaganski 2019-05-14 08:05:05 UTC
(In reply to Julien Nabet from comment #3)
> Mike: Would a perf/Flamegraph help here or do you have already an idea how
> to fix this?

I'm sorry for delay answering. I just found a performance problem, and reported it; I haven't looked (and have no plans to look) into this - acting as just a user here :-)
Comment 7 Roman Kuznetsov 2019-05-14 08:13:23 UTC
Eike, Markus, may be you'll can look at it?
Comment 8 Eike Rathke 2019-05-14 10:26:08 UTC
In column G up to G5000 there are formulas like

=IF($D5000="","",SUMPRODUCT(($A$2:$A$5000)=$D5000,$B$2:$B$5000))

Starting from G5001 for the rest of the entire (!) column there are formulas like

=IF($E5001="","",SUMPRODUCT(('file:///media/u/Seagate Expansion Drive/files/Delivery Driving/Damgoode Pies/Damgoode_Spreadsheet_Preliminary005.ods'#$DG_DailyLog.$A$2:$A$5000)=$E5001,'file:///media/u/Seagate Expansion Drive/files/Delivery Driving/Damgoode Pies/Damgoode_Spreadsheet_Preliminary005.ods'#$DG_DailyLog.$AB$2:$AB$5000))

The data is only up to row 151 anyway..

If a 32 build crashes it's probably simply because it's out of memory.
Comment 9 Commit Notification 2019-05-16 09:01:54 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/57df32e4bc12a3d67c44a1c544f7df21e1c8861e%5E%21

tdf#125254 Performance: A spreadsheet opens too slow, optimise listener

It will be available in 6.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 10 Commit Notification 2019-05-16 09:50:22 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/b27755b704fa4866025f1ed6d26c0d43aafa3407%5E%21

tdf#125254 Performance: A spreadsheet opens too slow, part1

It will be available in 6.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 11 Commit Notification 2019-05-18 08:47:22 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/3c3a371c799d00475deb13b4c3e0a8860c7e4fb3%5E%21

tdf#125254 Performance: A spreadsheet opens too slow, part2

It will be available in 6.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 12 Commit Notification 2019-05-18 12:19:40 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/ecf456d067346d49d7e2a366434477fa7de06d8a%5E%21

tdf#125254 Performance: A spreadsheet opens too slow, part3

It will be available in 6.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 13 Noel Grandin 2019-05-18 12:46:15 UTC
This is about as good as I can get it, CPU-wise.

The memory usage is a different storry - interning doesn't work, the formulas are all different, so the only way to improve it would be to change the import process to compile the formulas as it goes, which is a bigger change than I feel comfortable making.
Comment 14 Commit Notification 2019-05-18 13:11:29 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/787525a314de9c8178d0fedcd71ddbd08ec41a55%5E%21

tdf#125254 Performance: A spreadsheet opens too slow, NameSpaceEntry

It will be available in 6.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 15 Xisco Faulí 2019-06-26 10:39:14 UTC
in

Version: 6.4.0.0.alpha0+
Build ID: fe855eda54faf6196ad9dea12d8dc090b6d6c1da
CPU threads: 4; OS: Linux 4.15; UI render: default; VCL: gtk3; 
Locale: ca-ES (ca_ES.UTF-8); UI-Language: en-US
Calc: threaded

it takes

real	1m17,324s
user	1m35,619s
sys	0m5,378s

while in

Version: 6.2.6.0.0+
Build ID: 1914932e2fd0598508823464765f3b1ac31236a1
CPU threads: 4; OS: Linux 4.15; UI render: default; VCL: gtk3; 
Locale: ca-ES (ca_ES.UTF-8); UI-Language: en-US
Calc: threaded

it takes

real	4m45,230s
user	5m3,957s
sys	0m6,401s

so there is a nice improvement here...
Comment 16 Julien Nabet 2020-09-07 14:35:05 UTC
Created attachment 165238 [details]
perf flamegraph

Here's a Flamegraph retrieved on pc Debian x86-64 with master sources updated today (49c8656eaf18dbbcfdcfb178c9407ea61c206ec4) + gen rendering

I don't know if there's an impact but because of the bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965247, I had lots of:
addr2line: DWARF error: could not find variable specification at offset 33966

So hope Flamegraph is correct.
Comment 17 QA Administrators 2022-09-08 05:06:20 UTC Comment hidden (obsolete)
Comment 18 Roman Kuznetsov 2022-09-08 06:37:48 UTC
I'm not sure about "too slow" now in

Version: 7.5.0.0.alpha0+ (x64) / LibreOffice Community
Build ID: 3e544b6938ee509a4f6df4c2e2996d71ce072506
CPU threads: 8; OS: Windows 10.0 Build 19043; UI render: Skia/Raster; VCL: win
Locale: ru-RU (ru_RU); UI: en-US
Calc: CL threaded

It opens the file for around 30 sec

And LO still allocate ~1,7Gb of memory while opens the file
Comment 19 Mike Kaganski 2022-09-08 06:42:15 UTC
(In reply to Roman Kuznetsov from comment #18)
> I'm not sure about "too slow" now in
> 
> Version: 7.5.0.0.alpha0+ (x64) / LibreOffice Community
> Build ID: 3e544b6938ee509a4f6df4c2e2996d71ce072506
> CPU threads: 8; OS: Windows 10.0 Build 19043; UI render: Skia/Raster; VCL:
> win
> Locale: ru-RU (ru_RU); UI: en-US
> Calc: CL threaded
> 
> It opens the file for around 30 sec

Agreed. Closing WORKDFORME.
The memory consumption is unrelated, and wasn't initially the topic.