Bug 101195 - Calc cannot open large xls file with many images ( file: http://rkb1c.ru/soft/35mb.xls )
Summary: Calc cannot open large xls file with many images ( file: http://rkb1c.ru/soft...
Status: RESOLVED WORKSFORME
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
5.1.4.2 release
Hardware: All All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard:
Keywords: filter:xls, perf
Depends on:
Blocks: XLS File-Opening Memory
  Show dependency treegraph
 
Reported: 2016-07-29 14:14 UTC by LKeithJordan
Modified: 2019-04-25 16:20 UTC (History)
8 users (show)

See Also:
Crash report or crash signature:


Attachments
Excel xls file with many images (truncated file for testing) (7.33 MB, application/vnd.ms-excel)
2016-07-29 14:14 UTC, LKeithJordan
Details
Callgrind output from 5.5, opening 35mb.xls (5.44 MB, application/x-xz)
2017-05-31 16:21 UTC, Buovjaga
Details

Note You need to log in before you can comment on or make changes to this bug.
Description LKeithJordan 2016-07-29 14:14:59 UTC
Created attachment 126468 [details]
Excel xls file with many images (truncated file for testing)

I am setting this up as a new bug report because it is a bit different from Bug 100442.

I am running LO Calc Version: 5.1.4.2, and MS Excel 2007 (12.0.6750.5000) on a customized MSI Gaming Laptop with the following specs:
  Processor: Intel Core i-2760QM CPU @ 2.40GHz
  RAM: 16GB
  OS: Win7 64-bit

[NOTE: I have NOT tested this on my Linux edition of LO Calc.]

I originally became involved while answering a post on ask.libreoffice.com (see https://ask.libreoffice.org/en/question/74060/unable-to-open-xls-file-in-lo-calc/?answer=74199#post-id-74199).

The xls file supplied by the OP had many images and 3,033 rows. File size was 34.5MB. Excel 2007 could open the file but LO Calc would crash after trying.

After some tests and exchanges with the OP, I truncated the file to 651 rows for your testing. File size is 7.32MB.

At 2,000+ rows, file open is quite slow but succeeds. As you approach 2,500 rows, file open fails.
Comment 1 MM 2016-07-29 20:43:46 UTC
Unconfirmed with v5.1.5.1 under windows 7 x64.
Unconfirmed with v5.2.0.3 under ubuntu 16.04 x64.

Opens reasonably fast over here.
Comment 2 m_a_riosv 2016-07-29 21:58:52 UTC
No issue, Win10x64
Version: 5.1.5.2 (x64)
Build ID: 7a864d8825610a8c07cfc3bc01dd4fce6a9447e5
CPU Threads: 1; OS Version: Windows 6.19; UI Render: default; 
Locale: es-ES (es_ES); Calc: CL

Please test the memory set in Menu/Tools/LibreOffice/View - Memory cache, it's not less than 48 Mb, default value 64 Mb.
Comment 3 Buovjaga 2016-08-06 19:29:45 UTC
Yep opens quickly.

Set to NEEDINFO.
Change back to UNCONFIRMED, if the problem persists even after checking the setting asked in comment 2. Change to RESOLVED WORKSFORME, if the problem went away.

Arch Linux 64-bit, KDE Plasma 5
Version: 5.3.0.0.alpha0+
Build ID: f3d26af51588af441f62fb69bb7a5432845226ac
CPU Threads: 8; OS Version: Linux 4.6; UI Render: default; 
Locale: fi-FI (fi_FI.UTF-8); Calc: group
Built on August 5th 2016
Comment 4 LKeithJordan 2016-08-08 13:25:32 UTC
(In reply to m.a.riosv from comment #2)
> No issue, Win10x64
> Version: 5.1.5.2 (x64)
> Build ID: 7a864d8825610a8c07cfc3bc01dd4fce6a9447e5
> CPU Threads: 1; OS Version: Windows 6.19; UI Render: default; 
> Locale: es-ES (es_ES); Calc: CL
> 
> Please test the memory set in Menu/Tools/LibreOffice/View - Memory cache,
> it's not less than 48 Mb, default value 64 Mb.

Memory is set at defaults as follows:

Graphics Cache - 
Use for LO 128MB
Memory per object 20MB
Remove from memory after 10min

Cache for inserted objects - 
Number of objects 20
Comment 5 m_a_riosv 2016-08-09 18:28:06 UTC
Please test changing the use of OpenGL:
Menu/Tools/Options/LibreOffice/View
and disabling OpenCL if it is enable:
Menu/Tools/Options/LibreOffice/OpenCL
Comment 6 LKeithJordan 2016-08-09 20:20:29 UTC
(In reply to m.a.riosv from comment #5)
> Please test changing the use of OpenGL:
> Menu/Tools/Options/LibreOffice/View
> and disabling OpenCL if it is enable:
> Menu/Tools/Options/LibreOffice/OpenCL

Disabled "Allow use of OpenCL." No effect.

Further disabled use of software interpreter. No effect.

Would it help if you had the complete file made available by the OP who posted the ask.libreoffice.org question? If so, the OP provided a link to the file at:
http://rkb1c.ru/soft/35mb.xls
Comment 7 tommy27 2016-10-22 14:03:11 UTC
I can open the test file in 30 seconds under Win8.1 x64 using LibO 5.1.5.2 on an SSD drive.
Comment 8 Aron Budea 2016-11-20 04:43:49 UTC
No crash for me, and I checked with 5.1.4.2, 5.2.3.3, 5.3 daily build / Windows 7.
Can you check again with 5.2.3.3?

The large file opened in ages, but it opened in the end (I made some performance improvement to bug 100442 that should reduce loading times in 5.3, not saying it's going to be quick, but it should be much better).
Comment 9 QA Administrators 2017-05-31 10:51:10 UTC Comment hidden (obsolete)
Comment 10 LKeithJordan 2017-05-31 14:07:10 UTC
I have provided all the information I can, particularly since I was reporting a problem actually experienced by someone else on the ask.libreoffice.org forum. The forum post link is included at the beginning of this thread.

I'd like to offer some constructive feedback concerning this thread. I also want to emphasize that this is CONSTRUCTIVE feedback -- no insult intended to anyone here who has tried to help.

One thing I noticed in this thread is that several who responded failed to read or understand my original post. In the original post, I included a truncated file that WOULD open, indicating that the original file was much larger and also indicating the point of failure.

There was no question, then, that the uploaded file (651 rows and 7.32MB) would open -- that was a given. The question concerned problems experienced when trying to open the original file (3,033 rows and 34.5MB). I even provided the exact point of failure (2,500 rows), but all of that seemed to go unnoticed by some responders.

There were two reasons I provided the truncated file. First, I had a problem uploading the significantly larger file. Second, the truncated file provided sufficient material for testers to re-create the 3,033 row file.

Since my last response, I had to delete the spreadsheet files I was holding; I had to release storage space (drive getting too full). As a result, I am no longer able to provide further information. I am therefore changing the status to UNCONFIRMED, and suggest an Administrator change the status to CLOSED.

My thanks to all who responded. If you have further questions, perhaps you could contact the ask.libreoffice.org forum OP at the link I included in my original post.
Comment 11 Buovjaga 2017-05-31 14:50:19 UTC
Ok, Keith, I can explain. If you had provided steps to work on your truncated file, we would not have made the mistake. If you provide a file without steps, testers just immediately assume that "this is a problematic file" and go after it like a pack of rabid dogs.

The active testers go through a massive amount of bug reports, so there is bound to be these sorts of mistakes, when a report is not clear (see this for some stats: https://blog.documentfoundation.org/blog/2017/05/31/libreoffice-quality-assurance-six-months-statistics-part-1/ )

The best approach would have been to just go with http://rkb1c.ru/soft/35mb.xls as the main attraction and not attach the truncated file at all.

For the record, I can open 35mb.xls just fine.

Arch Linux 64-bit, KDE Plasma 5
Version: 5.3.3.2
Build ID: 5.3.3-1
CPU Threads: 8; OS Version: Linux 4.10; UI Render: default; VCL: kde4; Layout Engine: new; 
Locale: fi-FI (fi_FI.UTF-8); Calc: group
Comment 12 Xisco Faulí 2017-05-31 15:37:48 UTC
I downloaded http://rkb1c.ru/soft/35mb.xls

and it open within a minute in

Versión: 5.3.3.2
Id. de compilación: 3d9a8b4b4e538a85e0782bd6c2d430bafe583448
Subproc. CPU: 1; SO: Windows 6.1; Repr. de IU: predet.; Motor de trazado: HarfBuzz; 
Configuración regional: es-ES (es_ES); Calc: group

However, it takes much longer in

Version: 5.5.0.0.alpha0+
Build ID: 36b1e6270bf2fbb333e2a69c4bb5931eba418289
CPU threads: 1; OS: Windows 6.1; UI render: default; 
TinderBox: Win-x86@62-TDF, Branch:MASTER, Time: 2017-05-29_14:06:19
Locale: es-ES (es_ES); Calc: group

Could someone please confirm it?
Comment 13 LKeithJordan 2017-05-31 15:52:29 UTC
(In reply to Buovjaga from comment #11)
> Ok, Keith, I can explain. If you had provided steps to work on your
> truncated file, we would not have made the mistake. If you provide a file
> without steps, testers just immediately assume that "this is a problematic
> file" and go after it like a pack of rabid dogs.
> 
> The active testers go through a massive amount of bug reports, so there is
> bound to be these sorts of mistakes, when a report is not clear (see this
> for some stats:
> https://blog.documentfoundation.org/blog/2017/05/31/libreoffice-quality-
> assurance-six-months-statistics-part-1/ )
> 
> The best approach would have been to just go with
> http://rkb1c.ru/soft/35mb.xls as the main attraction and not attach the
> truncated file at all.
> 
> For the record, I can open 35mb.xls just fine.
> 
> Arch Linux 64-bit, KDE Plasma 5
> Version: 5.3.3.2
> Build ID: 5.3.3-1
> CPU Threads: 8; OS Version: Linux 4.10; UI Render: default; VCL: kde4;
> Layout Engine: new; 
> Locale: fi-FI (fi_FI.UTF-8); Calc: group

Thanks, Buovjaga. I thought my original post was clear, but then I'm not a regular on bugzilla either. I'll keep your advice in mind for the future.


Have a great day,
--Keith
L. Keith Jordan
Comment 14 Buovjaga 2017-05-31 15:56:12 UTC
(In reply to Xisco Faulí from comment #12)
> However, it takes much longer in
> 
> Version: 5.5.0.0.alpha0+
> Build ID: 36b1e6270bf2fbb333e2a69c4bb5931eba418289
> CPU threads: 1; OS: Windows 6.1; UI render: default; 
> TinderBox: Win-x86@62-TDF, Branch:MASTER, Time: 2017-05-29_14:06:19
> Locale: es-ES (es_ES); Calc: group
> 
> Could someone please confirm it?

export OOO_EXIT_POST_STARTUP=1
time libodev libobugs/35mb.xls 

real    1m7,897s
user    1m6,886s
sys     0m0,912s

Arch Linux 64-bit, KDE Plasma 5
Version: 5.5.0.0.alpha0+
Build ID: 963137415bcdf3e7a26ce5d258302f4e39e294db
CPU threads: 8; OS: Linux 4.11; UI render: default; VCL: kde4; 
Locale: fi-FI (fi_FI.UTF-8); Calc: group
Built on May 31st 2017

I will next try a callgrind so Áron can look at it.
Comment 15 Xisco Faulí 2017-05-31 16:11:20 UTC
I think the importing time is quite acceptable taking into account the size of the document. However, visualizing and working with the document seems more fluent in 5.3.3 than in 5.5.0
Comment 16 Buovjaga 2017-05-31 16:21:38 UTC
Created attachment 133752 [details]
Callgrind output from 5.5, opening 35mb.xls
Comment 17 m_a_riosv 2017-06-01 21:40:48 UTC
(In reply to Xisco Faulí from comment #15)
> I think the importing time is quite acceptable taking into account the size
> of the document. However, visualizing and working with the document seems
> more fluent in 5.3.3 than in 5.5.0

By my experience, dev builds are always slower than release builds. I guess they have controls that are not on published versions.

About 10 seconds to open on:
Version: 5.3.3.2 (x64)
Build ID: 3d9a8b4b4e538a85e0782bd6c2d430bafe583448
CPU Threads: 4; OS Version: Windows 6.19; UI Render: GL; Layout Engine: new; 
Locale: es-ES (es_ES); Calc: group
Comment 18 Buovjaga 2017-06-02 05:11:17 UTC
(In reply to m.a.riosv from comment #17)
> (In reply to Xisco Faulí from comment #15)
> > I think the importing time is quite acceptable taking into account the size
> > of the document. However, visualizing and working with the document seems
> > more fluent in 5.3.3 than in 5.5.0
> 
> By my experience, dev builds are always slower than release builds. I guess
> they have controls that are not on published versions.

This is true for builds that have debug or dbgutil enabled, but Xisco's build did not: http://dev-builds.libreoffice.org/daily/master/Win-x86@62-TDF/current/master~2017-05-29_14.06.19_build_info.txt
Comment 19 Xisco Faulí 2017-11-13 22:26:42 UTC
I've just retested in

Version: 6.0.0.0.alpha1+
Build ID: a7f961ddd88df4117de5f7c199881cf75d2905b5
CPU threads: 4; OS: Linux 4.10; UI render: default; VCL: gtk3; 
Locale: ca-ES (ca_ES.UTF-8); Calc: group

and it takes

real	2m16,043s
user	2m14,216s
sys	0m1,176s

to displayed the document ( using OOO_EXIT_POST_STARTUP=1 only takes 20 seconds )
Comment 20 Xavier Van Wijmeersch 2017-11-14 10:19:52 UTC
tested with

Version: 6.0.0.0.alpha1+
Build ID: 0b4d0c87fe201b7cc7f49b2dc14c134c95558a30
CPU threads: 8; OS: Linux 4.9; UI render: default; VCL: kde4; 
TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2017-11-11_02:45:18
Locale: nl-BE (en_US.UTF-8); Calc: group

it takes 7 seconds to open the attachment
Comment 21 Buovjaga 2017-11-14 14:02:08 UTC
Xavier: you need to open http://rkb1c.ru/soft/35mb.xls and not the attachment. I tried it again and it still takes the same amount of time as in comment 14
Comment 22 Xavier Van Wijmeersch 2017-11-14 14:37:31 UTC
Now with the right file it takes 83seconds
Comment 23 Telesto 2017-11-14 16:46:13 UTC
100442
Comment 24 raal 2017-11-23 15:23:11 UTC
I can confirm with 32bit LO, win7-64bit. No crash (no patience from me to wait ), but I killed soffice after 10 minutes. Excel can open this file immediately. Setting as new.

Version: 6.0.0.0.alpha1+ (from bibisect-win32-6.0 repo)
Build ID: 80af51be1aa85733b9c0b696a93edd8c6520811c
CPU threads: 4; OS: Windows 6.1; UI render: default;
Comment 25 QA Administrators 2018-11-24 03:44:04 UTC Comment hidden (obsolete)
Comment 26 Xavier Van Wijmeersch 2018-11-24 14:50:33 UTC
With master build the hole system memory was in 2 seconds used, also a lot of swap memory was used and i did need to shutdown the pc
I used the attachment 35mb.xls for the test

Version: 6.3.0.0.alpha0+
Build ID: 82cc30a832dde298832d9f029e0ead94a9d2d14b
CPU threads: 8; OS: Linux 4.19; UI render: default; VCL: kde4; 
Locale: nl-BE (en_US.UTF-8); UI-Language: en-US
Calc: threaded
Comment 27 Roman Kuznetsov 2019-03-24 13:18:44 UTC
It took over 2,8Gb of used memory while opening of example xls file.

I tried in

Version: 6.3.0.0.alpha0+ (x64)
Build ID: 5a907fe76bc628629fddb5501727642468d38dae
CPU threads: 4; OS: Windows 10.0; UI render: GL; VCL: win; 
TinderBox: Win-x86_64@42, Branch:master, Time: 2019-03-22_00:18:04
Locale: ru-RU (ru_RU); UI-Language: en-US
Calc: threaded
Comment 28 Xisco Faulí 2019-04-23 09:59:37 UTC
it seems http://rkb1c.ru/soft/35mb.xls it much slower now. it takes

real	6m18,398s
user	0m10,091s
sys	0m56,088s

in

Version: 6.3.0.0.alpha0+
Build ID: b34786d2774b261be48de92f65a5d0aa3c32b289
CPU threads: 4; OS: Linux 4.15; UI render: default; VCL: gtk3; 
Locale: ca-ES (ca_ES.UTF-8); UI-Language: en-US
Calc: threaded

with OOO_EXIT_POST_STARTUP=1
Comment 29 Xisco Faulí 2019-04-23 10:00:35 UTC
Similar problem with

Version: 6.1.0.0.alpha1+
Build ID: 3a801799536e6870f2fb111b1cc00b9575a35a39
CPU threads: 4; OS: Linux 4.15; UI render: default; VCL: gtk3; 
Locale: ca-ES (ca_ES.UTF-8); Calc: group

( oldest commit in bibisect-linux64-6.2 )
Comment 30 Xisco Faulí 2019-04-23 11:21:48 UTC
Regression mentioned in comment 28 and comment 29 reported in bug 124901
Comment 31 Xisco Faulí 2019-04-24 10:45:47 UTC
after https://git.libreoffice.org/core/+/af84fc9d906626255aaf136eefc5e55236e0e8a6%5E%21 it takes

real	0m24,458s
user	0m21,872s
sys	0m0,969s

in

Version: 6.3.0.0.alpha0+
Build ID: 90e3b47b52f26420425a7417d2f51b6a386282d9
CPU threads: 4; OS: Linux 4.15; UI render: default; VCL: gtk3; 
Locale: ca-ES (ca_ES.UTF-8); UI-Language: en-US
Calc: threaded

so the fileopen performance issue is fixed, however, it's still lagging while scrolling...
Comment 32 Xisco Faulí 2019-04-25 16:20:11 UTC
(In reply to Xisco Faulí from comment #31)
> after
> https://git.libreoffice.org/core/+/
> af84fc9d906626255aaf136eefc5e55236e0e8a6%5E%21 it takes
> 
> real	0m24,458s
> user	0m21,872s
> sys	0m0,969s
> 
> in
> 
> Version: 6.3.0.0.alpha0+
> Build ID: 90e3b47b52f26420425a7417d2f51b6a386282d9
> CPU threads: 4; OS: Linux 4.15; UI render: default; VCL: gtk3; 
> Locale: ca-ES (ca_ES.UTF-8); UI-Language: en-US
> Calc: threaded
> 
> so the fileopen performance issue is fixed, however, it's still lagging
> while scrolling...

Since this issue has many comments and the slow import is fixed, let's close this one as RESOLVED WORKSFORME ( Improved by different commits ).
Bug 124963 has been reported to cover the slow scrolling issue...