Bug 73049 - FILEOPEN Huge memory use on opening specific complex ods spreadsheets with array functions
Summary: FILEOPEN Huge memory use on opening specific complex ods spreadsheets with ar...
Status: RESOLVED WORKSFORME
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
4.1.1.1 rc
Hardware: Other Linux (All)
: medium critical
Assignee: Not Assigned
URL:
Whiteboard:
Keywords: bibisected, perf, regression
Depends on:
Blocks:
 
Reported: 2013-12-26 20:46 UTC by andis.lazdins
Modified: 2016-02-22 16:26 UTC (History)
10 users (show)

See Also:
Crash report or crash signature:


Attachments
valgrind trace (24.70 KB, application/x-bzip)
2013-12-31 15:35 UTC, Julien Nabet
Details

Note You need to log in before you can comment on or make changes to this bug.
Description andis.lazdins 2013-12-26 20:46:50 UTC
Libreoffice Calc module constantly crashes on opening particular complex ods file containing array formulas.

Program version Libreoffice 4.1.4.2, Linux Ubuntu 13.04., 32 bit version. 

The problematic file is available here https://drive.google.com/file/d/0Bxv4jQ_04jXZaVR2ZHZGR0JLVFE/edit?usp=sharing

Symptoms:

Try to open the file from file manager or the program Open dialogue. Process starts, use of CPU reach 98 %, then use of memory grows to 2,3 GB or something similar and after about 5 minutes program crashes. If started from terminal it crashes with message 
terminate called after throwing an instance of 'std::bad_alloc'
what():  std::bad_alloc

The problem is 100 % repeatable on this computer.

Libreoffice 4.2.0.0.beta2 can open file, but daverage () function do not shows reasonable results. Apache OpenOffice.org 4.0.1 opens file easily, all formulas and charts are in place and results are correct. 

I have quite many such files, the most of them are much bigger, so I think it's quite serious regression in Calc module.
Comment 1 Cor Nouws 2013-12-26 22:53:34 UTC
HI Andis,

Can confirm this. Thanks.

Tested with 32 bits Ubuntu 
4.1.3.2 and 4.2.0.rc1, and - by the way - 4.2.0beta2 show the same problem.

4.0.6.2 does open it (takes a while)
4.1.0. beta1 opens it fast
4.1.0. beta2 opens it slowwww
4.1.0. rc1 opens it slowwww
4.1.0. rc3 opens it slowwww

4.1.1 rc2 is the first one that fails for me. 
I do not have 4.1.1.1 to test, but set to that version...
Comment 2 Markus Mohrhard 2013-12-27 19:22:21 UTC
This is an out of memory problem and not a normal crash.
Comment 3 Julien Nabet 2013-12-31 12:51:17 UTC
On pc Debian x86-64 with master sources updated today (debug mode also)
PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM     TIME+ COMMAND
18575 julien    20   0 4932m 2,3g 119m S   1,7 40,8   1:43.49 soffice.bin
Markus: Indeed about the memory pb! :-)
Comment 4 Julien Nabet 2013-12-31 15:35:43 UTC
Created attachment 91370 [details]
valgrind trace

Valgrind trace from file opening to its closing.
Comment 5 Maxim Monastirsky 2013-12-31 15:58:05 UTC
Added similar reports to 'See Also'.
Comment 6 andis.lazdins 2014-08-09 07:33:49 UTC
The problem persists in Libreoffice 4.2.6 It's really hopeless to open 10 MB Calc file (Ubuntu 14.04, 32 bit), in spite it can be opened and manipulated in Openoffice.org on the same computer.
Comment 7 andis.lazdins 2014-08-09 11:09:20 UTC
When I changed memory settings to default values in Openoffice.org, I was able to lead files and even do changes in Pivot tables. The default values are:
Undo 
Number of steps - 100
Graphics cache
Use for Libreoffice 20
Memory per object 5.2
Remove from memory after 00:10
Cache for inserted objects
Number of objects 20

Probably the first step would be to test different memory options.
Comment 8 Julien Nabet 2014-08-09 12:30:39 UTC
With 4.2.6 LO Debian package with i5 + 6GB, it gives: 
real	0m14.427s
user	0m12.560s
sys	0m0.820s

with this command:
time soffice --calc MPS\ Augsnes\ pretestiba\ kopsanas\ parauglaukumos.ods 

For the test, could you rename your LO directory profile (see https://wiki.documentfoundation.org/UserProfile#GNU.2FLinux) and give it a new try?
Comment 9 andis.lazdins 2014-08-09 15:05:03 UTC
(In reply to comment #8)
> With 4.2.6 LO Debian package with i5 + 6GB, it gives: 
> real	0m14.427s
> user	0m12.560s
> sys	0m0.820s
> 
> with this command:
> time soffice --calc MPS\ Augsnes\ pretestiba\ kopsanas\ parauglaukumos.ods 
> 
> For the test, could you rename your LO directory profile (see
> https://wiki.documentfoundation.org/UserProfile#GNU.2FLinux) and give it a
> new try?

Libreoffice 4.2.6 crashed with these messages

(soffice:28742): Gtk-WARNING **: Error loading theme icon 'dialog-warning' for stock: Unable to load image-loading module: /usr/lib/i386-linux-gnu/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-png.so: /usr/lib/i386-linux-gnu/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-png.so: failed to map segment from shared object: Cannot allocate memory

(soffice:28742): Gdk-WARNING **: shmat failed: error 12 (Cannot allocate memory)

real	14m12.912s
user	0m50.132s
sys	0m3.564s

Libreoffice 4.3.0.4 crashed with this message

real	5m53.196s
user	0m0.340s
sys	0m0.208s

When I closed most of the applications, Libreoffice 4.3 was able to load document

real	5m36.688s
user	0m0.024s
sys	0m0.096s

System: Ubuntu 14.04, 32 bit, Kernel 3.13.0-32-generic, Memory 3.0 GiB, Processor 2 x Pentium(R) Dual-Core CPU T4500 @ 2,3 GHz
Comment 10 Julien Nabet 2014-08-14 20:35:12 UTC
Thank you Andis for your feedback, I put it back to NEW.
Comment 11 andis.lazdins 2014-09-08 21:00:00 UTC
Hello!

I tried in Libreoffice 4.3.1.2. It's impossible to open 7,9 MB calc file with libreoffice calc, if the file contains array or complex formula. The program just crash with:
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

I guess it became worse since 4.1, and in Openoffice.org 4.1 the same file opens rather easy.

Together with bug on slow scrolling the situation became really terrible for research use of libreoffice.
Comment 12 Joel Madero 2014-09-08 21:02:42 UTC
Do you think you could bibisect the bug? https://wiki.documentfoundation.org/QA/HowToBibisect

That would help quite a bit.
Comment 13 Julien Nabet 2014-09-08 21:04:55 UTC
Andis: just to be sure, did you rename your LO directory profile (as indicated in my previous comment)?
You gave results but didn't confirm you had renamed it.
Comment 14 andis.lazdins 2014-09-08 21:22:17 UTC
Hi!

I tried it without installation by extraction using command 

for i in ../*.deb; do dpkg-deb -x $i . ; done

and with it's own user directory

[Bootstrap]
BaseInstallation=${OOO_BASE_DIR}
InstallMode=<installmode>
ProductKey=OOo-dev 3.0
UserInstallation=$ORIGIN/..

I have one desktop PC with 8 GB of RAM and it is not better.
Comment 15 andis.lazdins 2014-09-08 21:25:06 UTC
(In reply to comment #12)
> Do you think you could bibisect the bug?
> https://wiki.documentfoundation.org/QA/HowToBibisect
> 
> That would help quite a bit.

I have 32 bit OS, not an option to me
Comment 16 Joel Madero 2014-09-08 21:31:55 UTC
@Andis - can you provide the 7.9 meg file?
Comment 17 andis.lazdins 2014-09-09 19:05:55 UTC
(In reply to comment #16)
> @Andis - can you provide the 7.9 meg file?

Hello!

Unfortunately that particular file contains classified information. I can send to you link to that file by e-mail (mine is andis.lazdins@gmail.com).

You can try this one below. I was able to open this (initially xls) file and even do some edits at the beginning of 2013 with libreoffice 3.X. Now it can't be opened either with libreoffice and openoffice. However, I'm not sure if there are some integrity errors in the file, because I don't have possibility to check it. The original xls file was also very hard to open for Excel.

https://drive.google.com/file/d/0Bxv4jQ_04jXZcXVIazkyMEw0R2s/edit?usp=sharing

Here is another example. The file initially created in libreoffice 3.X, now I can open it only with openoffice. With libreoffice it constantly crashes. The same happens also with much smaller files. And it is impossible to work even with smaller files because auto saving is very common status of the files.

https://drive.google.com/file/d/0Bxv4jQ_04jXZUk5qeEZBVnZXb1E/edit?usp=sharing

Here is another example. It can be opened with libreoffice 4.2.6.3, but last time it took about 15 minutes and any changes in formulas are really painful in terms of time consuming.

https://drive.google.com/file/d/0Bxv4jQ_04jXZV1J6bHpZZkstVTA/edit?usp=sharing

I hope it will help, because performance of calc, in spite of excellent improvements, becomes worse with every major release.
Comment 18 andis.lazdins 2014-09-21 07:39:22 UTC
If there are any hope to solve this and similar terrible Calc CPU use regressions or the only way is to move back to OpenOffice.org? Unfortunately it becomes worse with every next major releases and gap between Libreoffice and OpenOffice.org Calc performance increases. There are nice new features in Libreoffice Calc, like improved pivot tables, but they cost nothing, if the most of files, which could be easily operated with 3rd version branch now can't be opened at all or it takes minutes to do any changes in documents. I'm not mentioning Excel, which is much worse in terms of functionality, but much faster with those big files.

One example https://drive.google.com/file/d/0Bxv4jQ_04jXZRHZUeDlxSDRHNTQ/edit?usp=sharing

With 4.2.6.3 it can be opened in 54 seconds (production version, extensions etc.)
With 4.3.2.2 - 55 seconds
With 4.4.0.0 - 49 seconds
With Openoffice.org 4.1.1. - 18 seconds (production version, only 50% of CPU load in contrast to 100 % of all Libreoffice versions)
Excel 2007 through wine (file exported to xlsx format - 4 seconds (converted file https://drive.google.com/file/d/0Bxv4jQ_04jXZWGxOLUlIRnU2cUU/edit?usp=sharing)

Something really should be done here.
Comment 19 Julien Nabet 2014-09-21 07:42:35 UTC
Kohei/Markus/Eike: thought you might be interested in this one.
Comment 20 Jean-Baptiste Faure 2014-09-21 16:22:11 UTC
(In reply to comment #18)
> [...]
> One example
> https://drive.google.com/file/d/0Bxv4jQ_04jXZRHZUeDlxSDRHNTQ/edit?usp=sharing
> 
> With 4.2.6.3 it can be opened in 54 seconds (production version, extensions
> etc.)
> With 4.3.2.2 - 55 seconds

Hmmm, 8 seconds for me with version 4.3.3.0.0+ x86-64 build at home under Ubuntu 14.04 (LO being closed, select the file on the desktop, press enter key at hh:mm:10, file ready to work with at hh:mm:18, sometimes hh:mm:16)

Best regards. JBF
Comment 21 andis.lazdins 2014-09-21 17:28:38 UTC
(In reply to comment #20)
> (In reply to comment #18)
> > [...]
> > One example
> > https://drive.google.com/file/d/0Bxv4jQ_04jXZRHZUeDlxSDRHNTQ/edit?usp=sharing
> > 
> > With 4.2.6.3 it can be opened in 54 seconds (production version, extensions
> > etc.)
> > With 4.3.2.2 - 55 seconds
> 
> Hmmm, 8 seconds for me with version 4.3.3.0.0+ x86-64 build at home under
> Ubuntu 14.04 (LO being closed, select the file on the desktop, press enter
> key at hh:mm:10, file ready to work with at hh:mm:18, sometimes hh:mm:16)
> 
> Best regards. JBF

The results mentioned in comment 18 are from Ubuntu 14.04, 32 bit, Kernel 3.13.0-32-generic, Memory 3.0 GiB, Processor 2 x Pentium(R) Dual-Core CPU T4500 @ 2,3 GHz

It is the best result from computers we are using in our office. It's actually very complicated to say to employees, that the problem will be solved in some time (repeat it with every upgrade of Libreoffice).

I think it is important sometimes to imagine, that people are not laying just to get more attention, but to put yourself in place of those trying to advocate open-source solutions, donating personal money and having to negotiate with employees, which are really unhappy with the software and have good reason to sit in the office and look to the monitor doing nothing, because the program is hanged again in spite the problematic file can be opened on another nearly 10 years old celeron with MS Office 2003, but our policy is not to use MS Office.
Comment 22 Jean-Baptiste Faure 2014-09-21 20:07:03 UTC
(In reply to comment #21)
> [...]
> The results mentioned in comment 18 are from Ubuntu 14.04, 32 bit, Kernel
> 3.13.0-32-generic, Memory 3.0 GiB, Processor 2 x Pentium(R) Dual-Core CPU
> T4500 @ 2,3 GHz

It would be interesting if you could try with 64 bits versions of Ubuntu 14.04 and LibreOffice.
Did you observe an heavy use of the swap when using LibreOffice ? Ubuntu can be tweaked to reduce the use of the swap.

Note: I only try to find what is the cause of the difference in performance.

Best regards. JBF
Comment 23 andis.lazdins 2014-09-22 05:06:03 UTC
(In reply to comment #22)

> 
> It would be interesting if you could try with 64 bits versions of Ubuntu
> 14.04 and LibreOffice.
> Did you observe an heavy use of the swap when using LibreOffice ? Ubuntu can
> be tweaked to reduce the use of the swap.
> 
> Note: I only try to find what is the cause of the difference in performance.
> 
> Best regards. JBF

We are using here only 32 bit OS, all of them are different kind of Ubuntu versions, mostly 12.04 and 14.04.

Swap is not used, on laptop with 3GB RAM is utilized to about 1-2 GB, the same on PC with 8 GB RAM. Only processor is loaded to 100 %. I'll try to test this evening on laptop with 64 bit windows.
Comment 24 andis.lazdins 2014-09-22 17:55:04 UTC
Ubuntu 13.10, Kernel 3.11-0-26-generic 64 bit, Mate 1.6.0
RAM 7.7 GiB
Processor 4 x Intel(R) Core(TM) i3-3110M CPU @ 2.40 GHz
Libreoffice 4.1.6.2 (used in production)
7 sec
Libreoffice 4.3.1.2 (fresh installation)
16 sec
For testing first open libreoffice dashboard, then file - open the specified document.
Comment 25 andis.lazdins 2014-09-24 14:49:50 UTC
I tried to open the problematic file (https://drive.google.com/file/d/0Bxv4jQ_04jXZaVR2ZHZGR0JLVFE/edit?usp=sharing) on 64 bit Lubuntu 14.04 loaded from USB stick. Loading time is a bit longer on 64 bit system (about 70 sec. and 50 sec with Libreoffice 4.3.2.2, 8 GB RAM and 2 x 1,6 GHz CPU), but probably it is because the system is loaded from USB stick.
Comment 26 andis.lazdins 2014-09-30 10:36:52 UTC
(In reply to comment #25)
> I tried to open the problematic file
> (https://drive.google.com/file/d/0Bxv4jQ_04jXZaVR2ZHZGR0JLVFE/
> edit?usp=sharing) on 64 bit Lubuntu 14.04 loaded from USB stick. Loading
> time is a bit longer on 64 bit system (about 70 sec. and 50 sec with
> Libreoffice 4.3.2.2, 8 GB RAM and 2 x 1,6 GHz CPU), but probably it is
> because the system is loaded from USB stick.

Installed fresh 64 bit Ubuntu 14.04 on that machine and got the same results, about 1 minute to open complex calc file with 4.2.6, respectively there are no difference between 32 and 64 bit versions on that particular machine.
Comment 27 raal 2014-10-16 18:22:10 UTC
unable to bibisect with bibisect-43all (git checkout oldest - LO3.5 freeze )
Comment 28 Matthew Francis 2014-12-06 14:01:11 UTC
Results from bibisect-43all:
(Unsure if this is the only relevant performance regression, but the below is one significant transition from acceptable speed/memory use to extreme slowness and high memory use)
 89740762f0af849e492932bd71e59149cdcd5a00 is the first bad commit
commit 89740762f0af849e492932bd71e59149cdcd5a00
Author: Bjoern Michaelsen <bjoern.michaelsen@canonical.com>
Date:   Mon Dec 10 01:57:45 2012 +0000

    source-hash-06f20d73da21342046a480a6b22af69901351328
    
    commit 06f20d73da21342046a480a6b22af69901351328
    Author:     Stephan Bergmann <sbergman@redhat.com>
    AuthorDate: Fri Jul 20 11:10:05 2012 +0200
    Commit:     Stephan Bergmann <sbergman@redhat.com>
    CommitDate: Fri Jul 20 11:10:05 2012 +0200
    
        Fix SAL_LOG area usage
    
        Change-Id: If8acc5e9fee2730796637dfb505e0c514f96f1a3


# bad: [2e0fa432485d1db6abd355dad8ccb06f0b97e4fb] source-hash-ce90f99a2d66c2b998ad3f9f028e2ea623a757f5
# good: [a71a4447320f177181c9cff9f7c6fd93802cbd8e] source-hash-9afb6e1e38c362a768e8e981f7b03cf8bcaf22cf
git bisect start 'last40onmaster' 'last36onmaster'
# bad: [bf9969effb2f759d95ecbb1a688e25f75a78da16] source-hash-8638f1e72a3fe830c0e8dcc1bd847d4fb9e599ee
git bisect bad bf9969effb2f759d95ecbb1a688e25f75a78da16
# good: [7742d3f486b624b5467b51abdccf658dbbafc505] source-hash-836822522a2e9f009c0870cbbcd48d45bbd3c622
git bisect good 7742d3f486b624b5467b51abdccf658dbbafc505
# bad: [2e349599ef946cf01cfe40929509254c596fdca3] source-hash-cf2bdd65945d2a02af44db535cf1964d4d09ae20
git bisect bad 2e349599ef946cf01cfe40929509254c596fdca3
# bad: [173f32b96a0224f28f311adf21d65f4d4e98dfa1] source-hash-22cf0759547aa1803f77dbd3ee91774600dadc6f
git bisect bad 173f32b96a0224f28f311adf21d65f4d4e98dfa1
# good: [33fcd09137f9589d4f5f619e1b64347aa0beaef5] source-hash-36170cd0dbc3409270cf3cf998805a790ee6228f
git bisect good 33fcd09137f9589d4f5f619e1b64347aa0beaef5
# bad: [89740762f0af849e492932bd71e59149cdcd5a00] source-hash-06f20d73da21342046a480a6b22af69901351328
git bisect bad 89740762f0af849e492932bd71e59149cdcd5a00
# good: [ce3917fd9ea98ae2089cf4dca9cc742c10935e7f] source-hash-b0f170d7df9cff12535d2ecfd146b32b745a8ef8
git bisect good ce3917fd9ea98ae2089cf4dca9cc742c10935e7f
# first bad commit: [89740762f0af849e492932bd71e59149cdcd5a00] source-hash-06f20d73da21342046a480a6b22af69901351328
Comment 29 andis.lazdins 2014-12-06 15:05:17 UTC
(In reply to Matthew Francis from comment #28)
> Results from bibisect-43all:
> (Unsure if this is the only relevant performance regression, but the below
> is one significant transition from acceptable speed/memory use to extreme
> slowness and high memory use)

In our group we can use Libreoffice versions above 4.1 only on computers with fast CPU, initially obtained for GIS data processing. On other computers we are using Libreoffice 4.1.x or 4.1.x in combination with Openoffice.org 4.1 for calc data processing, because performance of Openoffice in such operations is still better. 

Increase of RAM on general purpose notebooks doesn't change anything. The problem is load of CPU.

I hope it will be solved somehow.
Comment 30 andis.lazdins 2015-02-14 06:45:05 UTC
Unfortunately the problem is still there in Libreoffice 4.4.0.3. Of course, we can use side by side libreoffice and openoffice for complex calculations, but it is not very convenient way of work and not everybody understand and accept, why to use different program in certain cases.

Libreoffice made huge progress in terms of functionality in compare to Microsoft Excel and lack of ability to work with larger documents on computers with, let's say, average CPU is actually the biggest drawback of the program.
Comment 31 Noel Grandin 2015-11-27 10:14:29 UTC
Running massif over this, points to the ScMatrix stuff (trimmed the MDDS stuff from the output):

99.29% (2,300,487,554B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
|     |         ->95.10% (2,203,326,856B) 0x3D2EF7F2: mdds::mtv::element_block<mdds::mtv::default_element_block<0, double>, 0, 
|     |               |         ->94.99% (2,200,882,248B) 0x3D9DAD41: ScMatrixImpl::PutStringVector(std::__debug::vector<svl::SharedString, std::allocator<svl::SharedString> > const&, unsigned long, unsigned long) (scmatrix.cxx:820)
|     |               |           ->94.99% (2,200,882,248B) 0x3D9DF701: ScFullMatrix::PutStringVector(std::__debug::vector<svl::SharedString, std::allocator<svl::SharedString> > const&, unsigned long, unsigned long) (scmatrix.cxx:2492)
|     |               |             ->94.99% (2,200,882,248B) 0x3D98EC1B: ScJumpMatrix::FlushBufferOtherThan(ScJumpMatrix::BufferType, unsigned long, unsigned long) (jumpmatrix.cxx:191)
|     |               |               ->94.99% (2,200,882,248B) 0x3D98EE2C: ScJumpMatrix::PutResultDouble(double, unsigned long, unsigned long) (jumpmatrix.cxx:221)
|     |               |               | ->94.99% (2,200,882,248B) 0x3D8FB4E6: lcl_storeJumpMatResult(ScMatrix const*, ScJumpMatrix*, unsigned long, unsigned long) (interpr1.cxx:228)
|     |               |               |   ->94.99% (2,200,882,248B) 0x3D8FDAC0: ScInterpreter::JumpMatrix(short) (interpr1.cxx:723)
|     |               |               |     ->94.99% (2,200,882,248B) 0x3D968BDA: ScInterpreter::Interpret() (interpr4.cxx:4091)
|     |               |               |       ->94.99% (2,200,882,248B) 0x3D6E91F6: ScFormulaCell::InterpretTail(ScFormulaCell::ScInterpretTailParameter) (formulacell.cxx:1805)
|     |               |               |         ->94.99% (2,200,882,248B) 0x3D6E7FBF: ScFormulaCell::Interpret() (formulacell.cxx:1531)
|     |               |               |           ->94.99% (2,200,882,248B) 0x3D6EBE94: ScFormulaCell::GetMatrix() (formulacell.cxx:2544)
|     |               |               |             ->94.99% (2,200,882,248B) 0x3D982B1A: ScInterpreter::ScMatRef() (interpr5.cxx:3181)
|     |               |               |               ->94.99% (2,200,882,248B) 0x3D967EA2: ScInterpreter::Interpret() (interpr4.cxx:3886)
|     |               |               |                 ->94.99% (2,200,882,248B) 0x3D6E91F6: ScFormulaCell::InterpretTail(ScFormulaCell::ScInterpretTailParameter) (formulacell.cxx:1805)
|     |               |               |                   ->94.99% (2,200,882,248B) 0x3D6E7FBF: ScFormulaCell::Interpret() (formulacell.cxx:1531)
|     |               |               |
Comment 32 Robinson Tryon (qubit) 2015-12-10 09:46:19 UTC
Migrating Whiteboard tags to Keywords: (bibisected, perf)
Comment 33 andis.lazdins 2015-12-12 08:01:48 UTC
I got feeling that in 5.0.4 this problem is even worse. Calc stops responding for more than 5 minutes or crashes on array formula with 46000 records.Openoffice.org do it in few seconds. I can provide test file on email, not for public use.
Comment 34 Markus Mohrhard 2016-02-15 10:11:36 UTC
Does this still happen in master. There were a few fixes to get the memory consumption in mdds under control and they are at least in current master, maybe also in 5.1.
Comment 35 andis.lazdins 2016-02-15 10:35:13 UTC
(In reply to Markus Mohrhard from comment #34)
> Does this still happen in master. There were a few fixes to get the memory
> consumption in mdds under control and they are at least in current master,
> maybe also in 5.1.

Hi!

In 5.1.0.2 I'm able to open some of problematic files, which can't be opened with 5.0.5.2, but it is still much less responsible than OpenOffice.org. The workaround for those files, which can be opened, is to switch off Auto calculate.
Comment 36 Julien Nabet 2016-02-15 16:25:00 UTC
(In reply to Markus Mohrhard from comment #34)
> Does this still happen in master. There were a few fixes to get the memory
> consumption in mdds under control and they are at least in current master,
> maybe also in 5.1.

On pc Debian x86-64 with master sources updated today (+enable-dbgutil), it took about 1min40 to load the file indicated (i5 with 6GB).
Comment 37 Markus Mohrhard 2016-02-18 09:26:57 UTC
(In reply to Julien Nabet from comment #36)
> (In reply to Markus Mohrhard from comment #34)
> > Does this still happen in master. There were a few fixes to get the memory
> > consumption in mdds under control and they are at least in current master,
> > maybe also in 5.1.
> 
> On pc Debian x86-64 with master sources updated today (+enable-dbgutil), it
> took about 1min40 to load the file indicated (i5 with 6GB).

A enable-dbgutil build should never used to provide any performance information.

From all the comments it seems that 5.1 is at least much better than older versions. Maybe not perfect but still better.
Comment 38 andis.lazdins 2016-02-18 12:05:55 UTC
(In reply to Markus Mohrhard from comment #37)

> From all the comments it seems that 5.1 is at least much better than older
> versions. Maybe not perfect but still better.

I would not say "much better", but there are some improvements. I just tried to work with 8 MB calc while without any formulas, which takes about 30 sec to open for OpenOffice.org. In 5.2 it took about 5 min. and it is not really possible to work with the file.
Comment 39 Markus Mohrhard 2016-02-20 07:30:47 UTC
(In reply to andis.lazdins from comment #38)
> (In reply to Markus Mohrhard from comment #37)
> 
> > From all the comments it seems that 5.1 is at least much better than older
> > versions. Maybe not perfect but still better.
> 
> I would not say "much better", but there are some improvements. I just tried
> to work with 8 MB calc while without any formulas, which takes about 30 sec
> to open for OpenOffice.org. In 5.2 it took about 5 min. and it is not really
> possible to work with the file.

You are mixing many different problems into this bug report. I'm loosing the overview what you are complaining about. The original report was about huge memory usage with array formulas and the last comment is about something completely different. If you still want something fixed please open new bug reports for each issue. In its current form this report has become unfixable as it mixes too many unrelated problems.

@QA team: IMHO we can close this issue.
Comment 40 andis.lazdins 2016-02-20 08:07:45 UTC
(In reply to Markus Mohrhard from comment #39)

> You are mixing many different problems into this bug report. I'm loosing the
> overview what you are complaining about. The original report was about huge
> memory usage with array formulas and the last comment is about something
> completely different. If you still want something fixed please open new bug
> reports for each issue. In its current form this report has become unfixable
> as it mixes too many unrelated problems.
> 
> @QA team: IMHO we can close this issue.

I'm not mixing anything, I don't know if there is one problem or several issues behind. What I can see is huge memory consumption, which sometimes results in crash, sometimes in opened but not useful file. There is clearly visible, terrible regression since 3.0 version, which makes calc useless for something more than school exercises.

If somebody don't want to recognize the problem, it doesn't mean that it is not existing any more. I guess libreoffice is supposed to be used by people, which are not programmers by definition, and I'm one of those persons not very interested in "how things work", but to use program products. To get things improved or at least working I'm reporting problem, which I have in my work, and I'm sending donations every year to libreoffice, which are equal to cost of licence of Microsoft office for our researchers team.

It is very complicated to persuade people to use libreoffice, especially if something is broken form more than one year and no solution is provided or at least promised. I don't know what arguments I can use, when I receive response like in previous comment. 

This issue can be closed when libreoffice calc will operate large files like openoffice.org or at least version 3.X of libreoffice
Comment 41 Joel Madero 2016-02-20 08:21:28 UTC
Markus is one of our most experienced developers - closing.

His point was that you were describing a separate issue from how this bug starts: 

"Libreoffice Calc module constantly crashes on opening particular complex ods file containing array formulas." is not the same as "just tried to work with 8 MB calc while without any formulas, which takes about 30 sec to open for OpenOffice.org."

As for the rest of your rant, I'll avoid commenting beyond saying...I do a lot more with Calc than just school exercises....

Feel free to open a new clean bug that is specific to the issue that you describe in your latest (prior to rant) comment.
Comment 42 andis.lazdins 2016-02-20 08:59:39 UTC
(In reply to Joel Madero from comment #41)
> Markus is one of our most experienced developers - closing.
> 
> His point was that you were describing a separate issue from how this bug
> starts: 
> 
> "Libreoffice Calc module constantly crashes on opening particular complex
> ods file containing array formulas." is not the same as "just tried to work
> with 8 MB calc while without any formulas, which takes about 30 sec to open
> for OpenOffice.org."
> 
> As for the rest of your rant, I'll avoid commenting beyond saying...I do a
> lot more with Calc than just school exercises....
> 
> Feel free to open a new clean bug that is specific to the issue that you
> describe in your latest (prior to rant) comment.

It is very sad that you cannot understand, that for many people it can be the same that "Libreoffice Calc module constantly crashes on opening particular complex ods file containing array formulas" and "just tried to work with 8 MB calc". It is very sad that formulation of comment is used as an argument to close report on serious regression. If it is so important I can repeat that I have the same issue: 
terminate called after throwing an instance of 'std::bad_alloc'
what():  std::bad_alloc

with the same files I had initially, but it is not any more 100% "success". I don't have time to waste for testing if the "improvement" works on other computers. Luckily openoffice.org still exist and we can use it for production. Will try one more time, when we will change computers or libreoffice will change something in attitude.

Good luck!
Comment 43 Markus Mohrhard 2016-02-22 16:26:49 UTC
(In reply to andis.lazdins from comment #42)
> (In reply to Joel Madero from comment #41)
> > Markus is one of our most experienced developers - closing.
> > 
> > His point was that you were describing a separate issue from how this bug
> > starts: 
> > 
> > "Libreoffice Calc module constantly crashes on opening particular complex
> > ods file containing array formulas." is not the same as "just tried to work
> > with 8 MB calc while without any formulas, which takes about 30 sec to open
> > for OpenOffice.org."
> > 
> > As for the rest of your rant, I'll avoid commenting beyond saying...I do a
> > lot more with Calc than just school exercises....
> > 
> > Feel free to open a new clean bug that is specific to the issue that you
> > describe in your latest (prior to rant) comment.
> 
> It is very sad that you cannot understand, that for many people it can be
> the same that "Libreoffice Calc module constantly crashes on opening
> particular complex ods file containing array formulas" and "just tried to
> work with 8 MB calc". It is very sad that formulation of comment is used as
> an argument to close report on serious regression. If it is so important I
> can repeat that I have the same issue: 
> terminate called after throwing an instance of 'std::bad_alloc'
> what():  std::bad_alloc
> 
> with the same files I had initially, but it is not any more 100% "success".
> I don't have time to waste for testing if the "improvement" works on other
> computers. Luckily openoffice.org still exist and we can use it for
> production. Will try one more time, when we will change computers or
> libreoffice will change something in attitude.
> 
> Good luck!

As I mentioned you should open a new bug report for each of the issues. We have a one issue, one bug report policy and as soon as you start using the same bug report for unrelated issues it is better to close the report and open new ones.

This policy helps developers and QA person as it is otherwise never clear what a bug fix actually tried to fix. I did not say anywhere that your reports are invalid just that this report in its current form with many unrelated issues has become too confusing. We need to start with a clean report for each issue and let a developer decide if they might be duplicates.