Bug 67099 - >700% performance regression vs. 4.0 when opening ODS files
Summary: >700% performance regression vs. 4.0 when opening ODS files
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
4.1.0.2 rc
Hardware: All All
: high major
Assignee: Kohei Yoshida
URL:
Whiteboard: target:4.2.0 target:4.1.1
Keywords: perf, regression
: 67423 (view as bug list)
Depends on:
Blocks: mab4.1
  Show dependency treegraph
 
Reported: 2013-07-19 19:46 UTC by jammon
Modified: 2015-12-15 11:36 UTC (History)
6 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description jammon 2013-07-19 19:46:50 UTC
Hi,

I have been using Libreoffice 4.0.4 in Ubuntu and Fedora and am impressed with the speed improvement. I have a ods file of 203302 rows and it takes Calc 4.0.4 only 15-20 secs to open. However Calc 4.1.0.1 and 4.1.0.2 (never tried 4.1.0.3) take (Fedora and Debian) about two minutes to open the same spreadsheet. It seems that there is a regression in Calc in handling large ods files.
Comment 1 m_a_riosv 2013-07-19 21:58:27 UTC
Hi jammon, thanks for reporting,
can you attach a sample file (Be aware that all comments posted, including attachments are public) to verify the issue.
Comment 2 jammon 2013-07-20 18:51:22 UTC
Hi,

I tried to attach a sample file but it complained that the file is too big, the file has to be big or else it wouldn't review the problem. Please advise.

Thanks
Comment 3 jammon 2013-07-20 19:32:30 UTC
Hi,

I have uploaded the test file to 
http://depositfiles.com/files/j938xse18

It is 9.19Mb, saved to ods from a .csv file. Calc4.04 loads and opens it in just under a minute while it takes Calc4.1.0rc1 more than 6 minutes.

Remark: I noticed another strange thing. If I save the csv to .xslx then calc loads and opens almost instantly. I don't know why it is that Calc seems to perform better in MS's format than its own. Is this a bug or a feature??
Comment 4 m_a_riosv 2013-07-20 22:50:02 UTC
Reproducible:
Win7x64i3Ultimate
Version 4.0.4.2 (Build ID: 9e9821abd0ffdbc09cd8c52eaa574fa09eb08f2)
  1m15s to load
Version: 4.1.1.0.0+ Build ID: c1f0f890f5065f88164add33707228f8c6d5755
TinderBox: Win-x86@6-debug, Branch:libreoffice-4-1, Time: 2013-07-17_16:17:47
  10m50s to load
Comment 5 Björn Michaelsen 2013-07-28 18:52:50 UTC
700% performance regression => high, major
marking as regression and requesting bibisect
also marking 67423 as dupe
Comment 6 Björn Michaelsen 2013-07-28 18:53:12 UTC
*** Bug 67423 has been marked as a duplicate of this bug. ***
Comment 7 Björn Michaelsen 2013-07-28 18:56:08 UTC
additional comments at http://test.ubuntu-discourse.org/t/libreoffice-4-1-0-released/764/15
Comment 8 Michael Meeks 2013-07-29 08:44:17 UTC
Kohei - possibly related to the partially finished internal structural re-work ? :-)
Comment 9 Kohei Yoshida 2013-07-29 16:51:58 UTC
(In reply to comment #6)
> *** Bug 67423 has been marked as a duplicate of this bug. ***

You shouldn't mark a bug which just says "opening an ods is slow" without a test document a duplicate of this (or any other similar reports).  Each reported case can be unique, can potentially have different root causes.
Comment 10 Kohei Yoshida 2013-07-29 17:50:35 UTC
Could someone spare some time to profile this document?  That would help a lot.
Comment 11 Michael Meeks 2013-07-29 18:24:58 UTC
I'll leave a 4.1 build profile running on this overnight - and see how far it gets :-) the sponsored link download is a pain, so I've up-loaded it here:

http://www.gnome.org/~michael/fdo67099-test.ods

for anyone that wants it (and thanks for the sample doc jammon).
Comment 12 Michael Meeks 2013-07-30 08:46:52 UTC
I've uploaded the trace (which completed overnight) to:

http://www.gnome.org/~michael/callgrind-fdo67099.txt.bz2

for interest:

==17158== Events    : Ir Dr Dw I1mr D1mr D1mw ILmr DLmr DLmw
==17158== Collected : 404574115034 183186563821 107078280387 6030970710 11102314299 634883148 14105730 690495941 12684227
==17158== 
==17158== I   refs:      404,574,115,034
==17158== I1  misses:      6,030,970,710
==17158== LLi misses:         14,105,730
==17158== I1  miss rate:            1.49%
==17158== LLi miss rate:             0.0%
==17158== 
==17158== D   refs:      290,264,844,208  (183,186,563,821 rd + 107,078,280,387 wr)
==17158== D1  misses:     11,737,197,447  ( 11,102,314,299 rd +     634,883,148 wr)
==17158== LLd misses:        703,180,168  (    690,495,941 rd +      12,684,227 wr)
==17158== D1  miss rate:             4.0% (            6.0%   +             0.5%  )
==17158== LLd miss rate:             0.2% (            0.3%   +             0.0%  )
==17158== 
==17158== LL refs:        17,768,168,157  ( 17,133,285,009 rd +     634,883,148 wr)
==17158== LL misses:         717,285,898  (    704,601,671 rd +      12,684,227 wr)
==17158== LL miss rate:              0.1% (            0.1%   +             0.0%  )

HTH :-)
Comment 13 Kohei Yoshida 2013-07-31 02:24:36 UTC
Thanks Michael.  Yup, this is a very familiar pattern i.e. get_block_position() at the top of the call stack.
Comment 14 Kohei Yoshida 2013-08-01 19:56:14 UTC
No bibisect needed.
Comment 15 Commit Notification 2013-08-02 18:27:35 UTC
Kohei Yoshida committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=bf846ac0510da21f7faa601fbd30d9147dc3ee71

fdo#67099: Remove overheads on inserting cells during ods import.



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 16 Commit Notification 2013-08-02 18:27:54 UTC
Kohei Yoshida committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=f3d2b533122f918929f3ef5a77af4972582a5960

fdo#67099: Don't use edit engine for a single unformatted paragraph.



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 17 Kohei Yoshida 2013-08-02 19:57:16 UTC
Backport request:
https://gerrit.libreoffice.org/#/c/5258/
https://gerrit.libreoffice.org/#/c/5259/
Comment 18 Kohei Yoshida 2013-08-02 19:59:54 UTC
Here is my data.

44.7236 sec (4.0)

master: 
277.481 sec (initial)
79.656 sec (use ScDocumentImport)
49.6682 sec (first paragraph string)

4.1:
161.838 sec (initial)
78.678 sec (use ScDocumentImport)
45.9855 sec (first paragraph string)
Comment 19 Commit Notification 2013-08-05 15:21:51 UTC
Kohei Yoshida committed a patch related to this issue.
It has been pushed to "libreoffice-4-1":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=09ce92dc51ebf75cfe021ccbaa4020b87b27360b&h=libreoffice-4-1

fdo#67099: Remove overheads on inserting cells during ods import.


It will be available in LibreOffice 4.1.1.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 20 Commit Notification 2013-08-05 15:31:19 UTC
Kohei Yoshida committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=808350a697a79ebcc1708f18de407294576a793e

fdo#67099: Call Clear() before setting SetText() for the first time.



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 21 Commit Notification 2013-08-05 15:45:53 UTC
Kohei Yoshida committed a patch related to this issue.
It has been pushed to "libreoffice-4-1":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=ec949345c018c97153be17b1d8a48be0ce3a1ff0&h=libreoffice-4-1

fdo#67099: Don't use edit engine for a single unformatted paragraph.


It will be available in LibreOffice 4.1.1.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 22 Kohei Yoshida 2013-08-05 20:08:38 UTC
Fixed.
Comment 23 Robinson Tryon (qubit) 2015-12-15 11:36:33 UTC
Migrating Whiteboard tags to Keywords: (perf)
[NinjaEdit]