Bug 68615 - FILEOPEN: Out Of Memory on loading ODS file
Summary: FILEOPEN: Out Of Memory on loading ODS file
Status: RESOLVED WONTFIX
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
4.1.0.4 release
Hardware: x86-64 (AMD64) Windows (All)
: medium major
Assignee: Not Assigned
URL:
Whiteboard: BSA
Keywords:
Depends on:
Blocks:
 
Reported: 2013-08-27 16:11 UTC by fischer.alexander
Modified: 2014-12-09 14:57 UTC (History)
4 users (show)

See Also:
Crash report or crash signature:


Attachments
Screenshot of Messagebox for the crash (10.14 KB, image/png)
2013-08-27 18:30 UTC, fischer.alexander
Details
ODS-File, which crashes LibreOffice 4.1.x on load (589.86 KB, application/vnd.oasis.opendocument.spreadsheet)
2013-08-28 08:54 UTC, fischer.alexander
Details

Note You need to log in before you can comment on or make changes to this bug.
Description fischer.alexander 2013-08-27 16:11:30 UTC
Problem description: 
With version 4.0.4 there are no problems with opening the sheet. With version 4.1.0 and higher there is always(!) a crash on loading/opening the specific file.

The file contains macros and datas. Other ods-files have no problems while opening.

Steps to reproduce:
1. open the file

Current behavior:
Crashes after short time of loading

Expected behavior:
Load the file successfully
              
Operating System: Windows 7
Version: 4.1.1.2 rc
Last worked in: 4.0.5.2 rc
Comment 1 fischer.alexander 2013-08-27 16:13:46 UTC
Also the portable edition 4.1.0 is crashing, with all older versions (4.0.x - installed or portable) there are no problems.
Comment 2 bfoman (inactive) 2013-08-27 17:06:01 UTC
Please attach problematic document.
Comment 3 fischer.alexander 2013-08-27 18:28:36 UTC
The document contains secret datas, so I can not simply attach it. I will try to anonymize the document (and keep the crash on loading).
Comment 4 fischer.alexander 2013-08-27 18:30:28 UTC
Created attachment 84740 [details]
Screenshot of Messagebox for the crash
Comment 5 bfoman (inactive) 2013-08-27 20:07:56 UTC
Please try to eliminate confidential data according to https://wiki.documentfoundation.org/QA/FAQ#How_can_I_eliminate_confidential_data_from_a_sample_document.3F
If you can't produce reproducible crash document this way, please try to get the backtrace yourself according to:
https://wiki.documentfoundation.org/How_to_get_a_backtrace_with_WinDbg
Debug builds you can build by yourself or download from:
http://dev-builds.libreoffice.org/daily/master/Win-x86@6-debug/
Symbols store is available at:
http://dev-builds.libreoffice.org/daily/master/Win-x86@6-debug/symbols/
Comment 6 fischer.alexander 2013-08-28 08:54:15 UTC
Created attachment 84777 [details]
ODS-File, which crashes LibreOffice 4.1.x on load

File crashes LibreOffice Calc 4.1.x on load
Comment 7 tommy27 2013-08-30 09:28:55 UTC
tested with Win7 64bit
crash REPRODUCIBLE on 4.1.0.4
file can be opened in 4.0.4.2

setting status to NEW and adding Calc expert to CC list

@Alexander
please remember to indicate earlier version where bug is reproducible in the version field (in this case was 4.1.0 release not 4.1.1 since you said it already crashed with 4.1.0)
Comment 8 Markus Mohrhard 2013-09-04 09:18:40 UTC
Can't reproduce a crash in master.
Comment 9 tommy27 2013-09-05 06:07:49 UTC
(In reply to comment #8)
> Can't reproduce a crash in master.

tested on WIn7 64bit.
still crashes LibO 4.1.1.2 and 4.2 master (Time: 2013-08-31_03:30:47)

@Markus
which master version time you used?
as said before my 31st august master has still issues.
Comment 10 Markus Mohrhard 2013-09-05 07:08:37 UTC
Can you then please attach a backtrace? My master build is from yesterday.

The only problem I saw is a high memory consumption so it might just be a std::bad_alloc or something like that.
Comment 11 fischer.alexander 2013-09-05 07:55:11 UTC
(In reply to comment #10)
> Can you then please attach a backtrace? My master build is from yesterday.
> 
> The only problem I saw is a high memory consumption so it might just be a
> std::bad_alloc or something like that.

That is right, the attached document contains a approximatly 20MB XML-file.
But on my system (with 16GB of RAM) the older versions (up to 4.0.x) load this document without any problems (only some higher amount of loading-time).
Comment 12 Markus Mohrhard 2013-09-05 08:44:26 UTC
Well since you use Windows you have a 32bit build and can't use more than 4GB anyway. So a change in how we allocate memory might easily result in a std::bad_alloc now. Whereas my 64bit Linux build will show no issues and happily use more memory.

Debugging why we might need more memory now is quite difficult and not a real regression. Changes in the memory allocation are normal and such big files are always a problem. I can't promise that we can fix it in the near future as we are working hard on the calc core at the moment.

The next version might already improve the situation when more of the work has been merged.
Comment 13 tommy27 2013-09-05 10:39:05 UTC
@Markus
my crash happens with Win7 64bit and 8GB RAM.
4.2-master from 31st august still crashes. I haven't tried yet more recent builds
Comment 14 Markus Mohrhard 2013-09-05 11:21:30 UTC
(In reply to comment #13)
> @Markus
> my crash happens with Win7 64bit and 8GB RAM.
> 4.2-master from 31st august still crashes. I haven't tried yet more recent
> builds

It dows not matter whether you use a 64bit Windows or not. Our Windows builds are all only 32bit so you can not use your full RAM anyway.
Comment 15 tommy27 2013-09-06 00:12:35 UTC
@Markus
thanks for info about 32 and 64 bit.
it still crashes on Win7 64bit using 4.2master (build Sept.3)
Comment 16 fischer.alexander 2013-09-08 20:41:08 UTC
(In reply to comment #12)
> Well since you use Windows you have a 32bit build and can't use more than
> 4GB anyway. So a change in how we allocate memory might easily result in a
> std::bad_alloc now. Whereas my 64bit Linux build will show no issues and
> happily use more memory.
> 
> Debugging why we might need more memory now is quite difficult and not a
> real regression. Changes in the memory allocation are normal and such big
> files are always a problem. I can't promise that we can fix it in the near
> future as we are working hard on the calc core at the moment.
> 
> The next version might already improve the situation when more of the work
> has been merged.

So my questions are:

Do you plan a 64bit-version of LibreOffice for Windows for resolving memory-issues?
Why is the memory-management of LO 4.0.x better (more stable) than for 4.1.x (for Windows)?

For me this bug is a real regression of version 4.1.x related to 4.0.x (I have to use a portable 4.0.x-version parallely to the normal installed 4.1.x).

How can I cleanup/shrink the ODS-file for loadability with LO 4.1.x?
Comment 17 fischer.alexander 2013-11-15 12:12:22 UTC
I have switched to Apache OpenOffice 4.0.1 and can load the document without any problem.
After saving the document the file-size is shrinked.
Comment 18 tommy27 2013-11-16 05:09:38 UTC
I confirm crash under Win7 64bit using 4.1.3.2 and 4.2 alpha build time 2013-11-01_23:37:14
Comment 19 Eike Rathke 2013-11-26 20:56:03 UTC
@Markus: this might be interesting for you ;-)

Loading the document allocates ~7GB of memory, which on Windows given that the application is 32-bit leads to an out-of-memory abort.

The document contains a tremendous amount (5311) of table-cell styles of which most have 10 <style:map> elements that except the style:base-cell-address attribute are identical. Apparently for each cell a separate conditional style is present.

The document seems to originate from Excel as 24 of the <style:map> elements have a style:apply-style-name='Excel_5f_5f_5f_...' where the name is 24586 (!) characters long with _5f repeated all over, making up three different styles ending in ..._1, ..._2 and ..._3. These names are also used once each in a <calcext:condition calcext:apply-style-name='...'>.
Comment 20 Markus Mohrhard 2013-11-27 23:55:04 UTC
(In reply to comment #19)
> @Markus: this might be interesting for you ;-)
> 
> Loading the document allocates ~7GB of memory, which on Windows given that
> the application is 32-bit leads to an out-of-memory abort.
> 
> The document contains a tremendous amount (5311) of table-cell styles of
> which most have 10 <style:map> elements that except the
> style:base-cell-address attribute are identical. Apparently for each cell a
> separate conditional style is present.
> 
> The document seems to originate from Excel as 24 of the <style:map> elements
> have a style:apply-style-name='Excel_5f_5f_5f_...' where the name is 24586
> (!) characters long with _5f repeated all over, making up three different
> styles ending in ..._1, ..._2 and ..._3. These names are also used once each
> in a <calcext:condition calcext:apply-style-name='...'>.

Yes. The conditional formatting is a problem. The old cell based conditional formatting was able to deal with this and in newer builds we should be able to import the old document much better.

The proiblem is that this document has been stored with Libreoffice before I switched to a more intelligent conditional formatting merge strategy. In current builds we should merge all equal conditional formats into one conditional format with a large range( what the original document most likely had as well ). The old calc core made one conditional format per cell out of it while the new conditional format code should merge them if the document does not yet contain new conditional format information.

The best thing for this document is tore it once with an old version and then import it into 4.1(I'm not sure which version already contains my changes) or later. That should produce much better conditional formatting which makes editing the document faster and more user friendly.

Currently I have no better idea how we can improve the handling of this file.
Comment 21 Caolán McNamara 2014-12-09 14:57:20 UTC
We don't have a CANTFIX mode, so using WONTFIX as a replacement.