Bug 148094 - Calc Hangs and uses ever increasing amount of memory with lots of MATCH(1, ("X"=range) * ("Y"=range), 0) EDITING/OPENING
Summary: Calc Hangs and uses ever increasing amount of memory with lots of MATCH(1, ("...
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
7.1.8.1 release
Hardware: x86-64 (AMD64) All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard:
Keywords: perf
Depends on:
Blocks: Calc-Function
  Show dependency treegraph
 
Reported: 2022-03-20 08:59 UTC by Peter
Modified: 2024-04-01 10:03 UTC (History)
5 users (show)

See Also:
Crash report or crash signature:


Attachments
Demonstration Spreadsheet (1.50 MB, text/plain)
2022-03-20 09:01 UTC, Peter
Details
Demonstration Spreadsheet (877.04 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
2022-03-20 09:03 UTC, Peter
Details
Apple Crash Report - Demo Spreadsheet Memory Exponential increase (1.50 MB, text/plain)
2022-03-20 09:04 UTC, Peter
Details
Apple Crash Report - 2.5 MB Spreadsheet (2.23 MB, text/plain)
2022-03-20 09:06 UTC, Peter
Details
Screen Shot of DEMO spreadsheet Saving (516.19 KB, image/png)
2022-03-20 09:07 UTC, Peter
Details
Screenshot of hang while calculating lots of INDEX(MATCH) formulas on fill (274.96 KB, image/png)
2022-03-20 09:09 UTC, Peter
Details
Screenshot of RAM stats during LibreOffice Hang + Memory Leak (61.46 KB, image/png)
2022-03-20 09:10 UTC, Peter
Details
When I tried to archive user profile to upload (111.53 KB, image/png)
2022-03-20 09:11 UTC, Peter
Details
RAM Statistics -- After 2.5MB spreadsheet memory leak left for a while. The Inverse spike is the moment I force quit Libreoffice. (47.75 KB, image/png)
2022-03-20 09:12 UTC, Peter
Details
RAM Statistc - A few force quits in regular succession (61.24 KB, image/png)
2022-03-20 09:15 UTC, Peter
Details
Flamegraph (415.91 KB, image/svg+xml)
2022-04-01 07:13 UTC, Julien Nabet
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Peter 2022-03-20 08:59:21 UTC
Description:
I have a 2.5MB spreadsheet with which uses lots of INDEX(range,MATCH(1, ("X"=range) * ("Y"=range), 0),1) formulas (to do a lookup based on two criteria). 

If you open this spreadsheet, or randomly afterwards LibreOffice hangs and starts using experiential amounts of RAM. I have 8GM ram and if I don't force quite before it gets to 5/6GB just for libreoffice the whole complete locks up.

Open with autocalculate off isn't possible (and having auto calculate on/off doesn't seem to matter). If you open the same file in MS Excel 2011 it opens fine.

I've created a simplified version of this spreadsheet which is smaller, and only contains the formulas above (see attached). Although this doesn't crash at open very often. It does fairly regularly exhibit the same behaviour where it hangs and start using ever more RAM as the bigger spreadsheet does. I've not been able to deduce what causes this.

There is also performance issue with calculating spreadsheet with such formulas. A sheet with 400 such formulas (with Multithreading enabler take a good few minutes too calculate (even with multithreading on it still only seems to use one core's worth of CPU). 

And odd other performances issues, like it taking 10 seconds to add a new sheet (which is doesn't happen on other spreadsheets). Saving taking 30secs sometimes, but weirdly not always (that's on MATCH()_CRASH_DEMO.xlsx with files size of 200Kb or bigger).

OS Version:      Mac OS X 10.14.6 (Build 18G9323)
Architecture:    x86_64

Command:         soffice
Path:            /Applications/Downloaded/LibreOffice.app/Contents/MacOS/soffice
Identifier:      org.libreoffice.script
Version:         7.1.8.1 (7.1.8.1)
PID:             60566

Event:           hang

Hardware model:  MacBookPro9,2
Active cpus:     4
RAM:             8GB
Hard Drive:      960GB ScanDisk Pro SSD (SanDisk SDSSDXPS960G)
Graphics:        Intel HD Graphics 4000

Tried turning on OpenCL --> Doesn't seem to be available with computer hardware.
Tried turning off Multithreading --> Doesn't seem to make any difference

I've run out of time to keep compiling this bug report. I think I've produced replicable example I could upload, and then I re-open libreoffice in safe mode and struggled to repeat the expolnencail increase in memory use with DEMO spreadsheet. I've definetly crashed full 2.5MB spreadsheet in safe mode, but so far not the DEMO on I've attached. I'll keep using it in safe mode and see how long that last. I can upload my user profile if that's helpful.



Steps to Reproduce:
Reproducing the slow performance on INDEX(MATCH) formulas is very easy:
1. Open spreadsheet attached
2. Go to first sheet 'INDEX(MATCH)'
3. Drag down another 100 rows.

Reproducing the memory fault is harder to pin down.
1. Open spreadsheet attached
2. If that's not enough then trying to work on it for a few minutes is usually sufficient
3.

Actual Results:
Hang + Memory Leak a seemingly random intervals. 

Hang without memory increase for long time whilst calculating.


Expected Results:
Calculations that take significant amount of time should not lock up UI, or cause application to be un-responsive. They should just stop use editing spreadsheet and indicate some how that recalculation is in progress.

Under no circumstance should LibreOffice start using infinitely more RAM until computer locks up.


Reproducible: Sometimes


User Profile Reset: No



Additional Info:
Not sure if either of these bugs relate to issues I'm having:
https://bugs.documentfoundation.org/show_bug.cgi?id=139444
https://bugs.documentfoundation.org/show_bug.cgi?id=144142


Due to this bug I can't run latest version:
https://bugs.documentfoundation.org/show_bug.cgi?id=148069
Comment 1 Peter 2022-03-20 09:01:17 UTC
Created attachment 178981 [details]
Demonstration Spreadsheet

This spreadsheet contains the INDEX(range,MATCH(1, ("X"=range) * ("Y"=range), 0),1) formulas (to do a lookup based on two criteria).
Comment 2 Peter 2022-03-20 09:03:11 UTC
Created attachment 178982 [details]
Demonstration Spreadsheet

Spreadsheet with INDEX(range,MATCH(1, ("X"=range) * ("Y"=range), 0),1) formulas (to do a lookup based on two criteria).
Comment 3 Peter 2022-03-20 09:04:49 UTC
Created attachment 178983 [details]
Apple Crash Report - Demo Spreadsheet Memory Exponential increase

In normal (i.e. not safe mode) after DEMO spreadsheet (only think open) hang and start using ever increasing amounts of RAM
Comment 4 Peter 2022-03-20 09:06:12 UTC
Created attachment 178984 [details]
Apple Crash Report - 2.5 MB Spreadsheet

In normal (i.e. not safe mode) after 2.5MB spreadsheet hang and started using ever increasing amounts of RAM
Comment 5 Peter 2022-03-20 09:07:33 UTC
Created attachment 178985 [details]
Screen Shot of DEMO spreadsheet Saving

Sometimes saving task 30 seconds and libreoffice stops responding.
Comment 6 Peter 2022-03-20 09:09:02 UTC
Created attachment 178986 [details]
Screenshot of hang while calculating lots of INDEX(MATCH) formulas on fill
Comment 7 Peter 2022-03-20 09:10:55 UTC
Created attachment 178987 [details]
Screenshot of RAM stats during LibreOffice Hang + Memory Leak

This was probably 10-20 seconds after it stopped responding, and if I'd have left another 30 computer would have locked up.
Comment 8 Peter 2022-03-20 09:11:45 UTC
Created attachment 178988 [details]
When I tried to archive user profile to upload
Comment 9 Peter 2022-03-20 09:12:40 UTC
Created attachment 178989 [details]
RAM Statistics -- After 2.5MB spreadsheet memory leak left for a while.

The Inverse spike is the moment I force quit Libreoffice.
Comment 10 Peter 2022-03-20 09:15:03 UTC
Created attachment 178990 [details]
RAM Statistc - A few force quits in regular succession

Note: compared of other screenshot I'd closed a few other applications so more Ram was available.
Comment 11 Xisco Faulí 2022-03-21 11:07:07 UTC
Not reproduced in

Version: 7.4.0.0.alpha0+ / LibreOffice Community
Build ID: f461c889ac4bc053c306537f644ec4bf3e0d5128
CPU threads: 8; OS: Linux 5.10; UI render: default; VCL: gtk3
Locale: es-ES (es_ES.UTF-8); UI: en-US
Calc: threaded
Comment 12 Xisco Faulí 2022-03-21 11:07:20 UTC
Could you please try to reproduce it with the latest version of LibreOffice from https://www.libreoffice.org/download/libreoffice-fresh/ ?
I have set the bug's status to 'NEEDINFO'. Please change it back to 'UNCONFIRMED' if the bug is still present in the latest version.
Comment 13 Peter 2022-03-21 11:31:14 UTC
(In reply to Xisco Faulí from comment #12)
> Could you please try to reproduce it with the latest version of LibreOffice
> from https://www.libreoffice.org/download/libreoffice-fresh/ ?
> I have set the bug's status to 'NEEDINFO'. Please change it back to
> 'UNCONFIRMED' if the bug is still present in the latest version.

That is difficult for me to do because the latest version of Libreoffice crashes with a blank spreadsheet from template for other reason reported in this bug:
https://bugs.documentfoundation.org/show_bug.cgi?id=148069
Comment 14 QA Administrators 2022-03-22 03:57:42 UTC Comment hidden (obsolete)
Comment 15 Roman Kuznetsov 2022-03-31 20:13:28 UTC
I didn't have any memory leaks but I got the 100% CPU loading in

Version: 7.4.0.0.alpha0+ / LibreOffice Community
Build ID: 0a05b1f46263a16c6d40c841a317c3ba9f4d31d6
CPU threads: 4; OS: Mac OS X 12.3.1; UI render: default; VCL: osx
Locale: ru-RU (ru_RU.UTF-8); UI: en-US
Calc: threaded

My steps:

1. Open the example
2. Go to sheet 'INDEX(MATCH)'
3. Select a last row with data on the sheet
4. Try expand/drag it to over 100 rows to down => LO hangs with 100% CPU loading

Julien could you please test it on your machine and if you'll get the same please will create the perfgraph? Thank you
Comment 16 Julien Nabet 2022-04-01 07:13:45 UTC
Created attachment 179245 [details]
Flamegraph

Indeed, I reproduced the hang.
Here's a Flamegraph retrieved on pc Debian x86-64 with master sources updated today (+gen rendering).
Comment 17 Julien Nabet 2022-04-01 07:15:13 UTC
Noel/Eike: thought you might be interested in this perf pb in Calc.
I attached a Flamegraph in a previous comment.
Comment 18 QA Administrators 2024-04-01 03:12:06 UTC Comment hidden (obsolete)
Comment 19 Peter 2024-04-01 10:03:53 UTC
Hi QA Administrators,

As requested I've just run a further tests on new version of LibreOffice. I can't run LibreOffice 24 on my main machine as I'm still running MacOS10.14 which isn't supported, however I'm currently using LibreOffice 7.5.8.2 which is relatively stable in other use cases on my system.

Test Steps:
- I download the Demonstration Spreadsheet attached above.
- I opened it in LibreOffice 7.5.8.2 and waited for a few minutes.
- I noted that LibreOffice was using 2.42GB of ram (I had on other .docx open).
- I clicked around a bit and changed worksheets for 5mins.
- I check ram usage and it had started increasing continuously (3.2, 3.4, 3.6GBs....).
- I force quit LibreOffice

When I've got another machine up and running later OS, I'll try and reproduce in version 24.