Bug 79654 - Save XLSX consumes 100% memory and hangs with "bad allocation" (scfiltlo.dll)
Summary: Save XLSX consumes 100% memory and hangs with "bad allocation" (scfiltlo.dll)
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
4.2.3.3 release
Hardware: All All
: medium critical
Assignee: Not Assigned
URL:
Whiteboard:
Keywords: filter:xlsx, haveBacktrace
Depends on:
Blocks: Memory Performance CPU-AT-100%
  Show dependency treegraph
 
Reported: 2014-06-04 22:21 UTC by Michael Goodson
Modified: 2024-04-25 06:39 UTC (History)
5 users (show)

See Also:
Crash report or crash signature:


Attachments
Sample xlsx format file. (427.39 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
2014-06-04 22:47 UTC, Michael Goodson
Details
a backtrace (9.02 KB, text/plain)
2016-12-11 12:15 UTC, fiftyigfuci_f_mi
Details
perf flamegraph (63.42 KB, application/x-bzip)
2020-12-22 10:17 UTC, Julien Nabet
Details
Flamegraph (428.47 KB, image/svg+xml)
2022-09-16 19:21 UTC, Julien Nabet
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Goodson 2014-06-04 22:21:51 UTC
On some of my Libreoffice Calc files (in xlsx) format, when I save, it uses up 100% of available memory, causes the HD to read/write continuously and hangs.  After some period of time (typicaly 1-5 minutes) it will finish successfully.  Only when I close all Libreoffice windows will the memory be released again. Not sure why this is happening (memory leak somewhere?).  Interesting the the programs runs just fine and uses less than 1GB of RAM until I tell it to save.

The file is large (11.7Mb--there are a couple of scanned files in it, but this happens even without those) with quite a few formulae. 

 I am running Ubuntu 14.04 on a Intel® Core™ i7-3770 CPU @ 3.40GHz × 8 processor with 32 GB of RAM.  I have attached one file that typically gives me this problem.
Comment 1 Michael Goodson 2014-06-04 22:22:57 UTC Comment hidden (obsolete)
Comment 2 m_a_riosv 2014-06-04 22:33:04 UTC Comment hidden (obsolete)
Comment 3 Michael Goodson 2014-06-04 22:47:27 UTC
Created attachment 100424 [details]
Sample xlsx format file.

This is only 437Kb, still does it and took 15 minutes to save.
Comment 4 Michael Goodson 2014-06-04 22:55:47 UTC Comment hidden (obsolete)
Comment 5 m_a_riosv 2014-06-04 23:51:28 UTC
I can reproduce the issue, in:
Win7x64
Version: 4.2.3.3 Build ID: 882f8a0a489bc99a9e60c7905a60226254cb6ff0,
also with the last versions and master.

Maybe it is in relation with a lot of formulas in the last row/column on much of the sheets, deleting those last rows saves is fast.

Saving in the native file format (ods), is really fast.
Comment 6 Michael Goodson 2014-06-05 22:28:46 UTC
Thanks for catching those formulae in the last rows/columns.  Most of the sheets were copied from that 1 template sheet, so most had that. I deleted those rows/columns and it nows saves very quickly.  I wish I could use ODS format, but I have to work with Excel users as well. 

Thanks again.

PS I changed this to Resolved/Not a bug.  Please change it to the appropriate status if that isn't correct.
Comment 7 m_a_riosv 2014-06-05 23:18:19 UTC
Fortunately there is a solution for your issue, but I think there is some kind of bug, why it must take so long with xlsx while saves quickly with ods format?.

So let's reopen the bug.
Comment 8 QA Administrators 2015-07-18 17:43:38 UTC Comment hidden (obsolete)
Comment 9 m_a_riosv 2015-10-09 21:17:46 UTC
Issue persist with:
Version: 5.0.3.1 (x64) Build ID: fd8cfc22f7f58033351fcb8a83b92acbadb0749e
Comment 10 QA Administrators 2016-11-08 11:03:01 UTC Comment hidden (obsolete)
Comment 11 fiftyigfuci_f_mi 2016-12-11 12:15:23 UTC
Created attachment 129488 [details]
a backtrace

I reproduced on both:
Version: 5.2.2.2
Build ID: 1:5.2.2-0ubuntu2
CPU Threads: 2; OS Version: Linux 4.9; UI Render: default; 
Locale: en-US (en_US.UTF-8); Calc: group

(My debug dev build)
Version: 5.4.0.0.alpha0+
Build ID: f35d29c8388744be1f95ec4acfca12eec706911a
CPU Threads: 2; OS Version: Linux 4.9; UI Render: default; VCL: gtk3; 
Locale: en-US (en_US.UTF-8); Calc: group
Comment 12 QA Administrators 2017-12-13 09:30:05 UTC Comment hidden (obsolete)
Comment 13 Timur 2017-12-14 18:34:50 UTC
Also in 6.1+
Comment 14 QA Administrators 2018-12-15 03:56:43 UTC Comment hidden (obsolete)
Comment 15 Roman Kuznetsov 2020-07-20 10:07:35 UTC
still repro in

Version: 7.1.0.0.alpha0+
Build ID: 27cf6e73d05ac803d5fc12c53aea20ed53007234
CPU threads: 4; OS: Linux 5.3; UI render: default; VCL: gtk3
Locale: ru-RU (ru_RU.UTF-8); UI: en-US
TinderBox: Linux-rpm_deb-x86_64@86-TDF, Branch:master, Time: 2020-07-19_17:34:48
Calc: threaded

Julien, can you create your perfgraph for this? Just open file from attach and "Save as" it as XLSX
Comment 16 Julien Nabet 2020-07-21 16:43:11 UTC
(In reply to Roman Kuznetsov from comment #15)
> ...
> Julien, can you create your perfgraph for this? Just open file from attach
> and "Save as" it as XLSX

perfgraphs are useful only if there's someone to interpret it and to try to fix it. If there's nobody, they just going to rot and become useless because code would have changed a lot.
Comment 17 Roman Kuznetsov 2020-12-21 22:00:14 UTC
(In reply to Julien Nabet from comment #16)
> (In reply to Roman Kuznetsov from comment #15)
> > ...
> > Julien, can you create your perfgraph for this? Just open file from attach
> > and "Save as" it as XLSX
> 
> perfgraphs are useful only if there's someone to interpret it and to try to
> fix it. If there's nobody, they just going to rot and become useless because
> code would have changed a lot.

But without the perfgraph nobody from devs will not look at it at all =( 
Of course if you don't have any free time for it then I'm sorry for noise
Comment 18 Julien Nabet 2020-12-22 10:17:27 UTC
Created attachment 168408 [details]
perf flamegraph

Wait and see now...

(I think it'd be interesting to create new tags like wantperftrace/haveperftrace)
Comment 19 Julien Nabet 2022-09-16 19:21:03 UTC
Created attachment 182504 [details]
Flamegraph

Here's an update of the Flamegraph on pc Debian x86-64 with master sources updated today + gen rendering.
Comment 20 Julien Nabet 2022-09-16 19:39:18 UTC
Taking a look at the Flamegraph, I noticed
mxCellTable = new XclExpCellTable( GetRoot() );
(see https://opengrok.libreoffice.org/xref/core/sc/source/filter/excel/excdoc.cxx?r=1d25d32b#564).

The ctr of XclExpCellTable does quite a lot of things.

First I thought using a mere singleton but since the finalization seems to modify the state of the object, I think we rather need to make copies of the singleton (which corresponds to the original state).

Any thoughts here?

(BTW, there's the equivalent in ExcTable::FillAsTableBinary).