Download it now!
Bug 96753 - KDE: big file opening and navigating is slow vs. 4.3
Summary: KDE: big file opening and navigating is slow vs. 4.3
Status: RESOLVED WORKSFORME
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: graphics stack (show other bugs)
Version:
(earliest affected)
5.1.0.1 rc
Hardware: All Linux (All)
: medium normal
Assignee: Not Assigned
URL:
Whiteboard: needsKDE
Keywords: perf, regression
Depends on:
Blocks: KDE
  Show dependency treegraph
 
Reported: 2015-12-29 06:00 UTC by Yury
Modified: 2018-05-24 18:20 UTC (History)
2 users (show)

See Also:
Crash report or crash signature:


Attachments
sample doc, sanitized, exhibiting the slowing down (373.55 KB, application/vnd.oasis.opendocument.text)
2015-12-30 06:58 UTC, Yury
Details
Callgrind with 5.2 (7.50 MB, application/x-7z-compressed)
2016-03-25 18:38 UTC, Buovjaga
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Yury 2015-12-29 06:00:17 UTC
I have a document of ~159 A4 pages here, all text, no formulas, couple of footnotes. Formatting is mainly 'default' and 'body text', headings down to level 3. The ODT container size is 500K.

In 4.3.7.2 (compiled from source) the document opens in 2 seconds max, navigates to any item of outline view (opened in sidebar) without noticeable delay.

In 5.1.0.1 (compiled from source) the document opens in like 8 seconds (I didn't time it with stopwatch; may be even more), the CPU load is 99%. Navigation to an arbitrary item has noticeable delay, like up to 1 second (varies).

Re-saving the file in 5.1.0.1 changes nothing.

The configuration directory of 5.1.0.1 was copied from 4.3.7.2.

The `configure` options for building are the same.

I can't publish this file here, for anybody to grab, but would be willing to send it directly to a person looking into the problem, on undisclosure condition (the doc contains my personal notes on historical subjects and quotes extracted from open literature, so won't be useful for a wide public anyway, but...).
Comment 1 Yury 2015-12-29 06:18:59 UTC
...and it seems it's the interpretation part that goes extra slow. The progressbar for unpacking stage completes quick.
Comment 2 Yury 2015-12-29 06:21:50 UTC
These were the configure options:

./configure \
\
--with-krb5=no \
--with-gssapi=no \
\
--enable-graphite \
--without-fonts \
--without-junit \
--with-system-libgltf=no \
\
--with-external-tar=${HOME}/c/libo/ext5 \
\
--disable-gstreamer-1-0 \
--disable-gstreamer-0-10 \



exit $?
Comment 4 Yury 2015-12-30 06:58:37 UTC
Created attachment 121623 [details]
sample doc, sanitized, exhibiting the slowing down

I was afraid the sanitizing would make the problem "disappear", but, if anything, it gets worse.

Up to 10 secs, of those abnout 8 secs in interpreting stage after unpacking.  Maybe there's a clue there.

Delay on navigation noticeable, too.
Comment 5 Yury 2015-12-30 07:05:23 UTC
BTW, would anybody be interested in python macro 'xx'-ing the Math formulas sources? I've just dusted the file off (used it earlier in the year, for the previous bigfile issue sample).

I shouldn't just slap it on the wiki, should I?
Comment 6 Buovjaga 2015-12-30 14:15:22 UTC
(In reply to Yury from comment #5)
> BTW, would anybody be interested in python macro 'xx'-ing the Math formulas
> sources? I've just dusted the file off (used it earlier in the year, for the
> previous bigfile issue sample).
> 
> I shouldn't just slap it on the wiki, should I?

I think you could slap it in there, if you are able to upload. If there is any problem, I can contact the wiki admin. There is already one template with a macro for sanitizing content: https://wiki.documentfoundation.org/QA/BugReport/Attachments#Confidential_Attachments

Regarding the sample document, I don't have the slow down on opening or delay with navigation. Scrolling vigorously I notice 1 CPU core peaking occasionally, but I guess this is not what you observe. 

Maybe you could try with a TDF build instead of your own?

Win 7 Pro 64-bit Version: 5.2.0.0.alpha0+
Build ID: a4764cfa80270f973da5861d0ddc28298bf16f4d
CPU Threads: 4; OS Version: Windows 6.1; UI Render: default; 
TinderBox: Win-x86@62-merge-TDF, Branch:MASTER, Time: 2015-12-24_22:45:12
Locale: fi-FI (fi_FI)

Version: 5.1.0.1 (x64)
Build ID: bcace328aabc4c8c10b56daa87da0a2ee6579b5a
Threads 4; Ver: Windows 6.1; Render: default; 
Locale: fi-FI (fi_FI)
Comment 7 Yury 2015-12-30 15:50:33 UTC
You were right, thank you. The 5.1.0.1 as downloaded does not exhibit the issue.

As I understand the process, this issue will be now closed as NOTOURPROBLEM or something, right?

Unfortunately (for me), using the TDF build is not a solution (for me); I need the build with certain patches (dealing with MS word import/export) which will be incorporated in the main source, like, never.

Maybe you could help or at least hint where to look? You've seen my configure options, what could be the cause?

Or maybe the built-from-source installation hooks into some libraries on my system, which are, I don't know, of versions of sub-optimal performance or something??..
Comment 8 Yury 2015-12-30 15:51:53 UTC
...I couldn't even use the TDF's 5.0.3.2 on slackware 14.1, as it lacked some kind of library I never heard of.
Comment 9 Buovjaga 2015-12-30 17:30:50 UTC
(In reply to Yury from comment #7)
> You were right, thank you. The 5.1.0.1 as downloaded does not exhibit the
> issue.
> 
> As I understand the process, this issue will be now closed as NOTOURPROBLEM
> or something, right?
> 
> Unfortunately (for me), using the TDF build is not a solution (for me); I
> need the build with certain patches (dealing with MS word import/export)
> which will be incorporated in the main source, like, never.
> 
> Maybe you could help or at least hint where to look? You've seen my
> configure options, what could be the cause?

For build problems you can consult the developer mailing list or IRC.
http://lists.freedesktop.org/mailman/listinfo/libreoffice
#libreoffice-dev @ Freenode

Why do you think those "certain patches" will never be incorporated to LibreOffice? Can you give links in gerrit to the patches?
Comment 10 Yury 2015-12-31 06:46:47 UTC
(In reply to Beluga from comment #9)
> (In reply to Yury from comment #7)
...
> > Unfortunately (for me), using the TDF build is not a solution (for me); I
> > need the build with certain patches (dealing with MS word import/export)
> > which will be incorporated in the main source, like, never.
...
> Why do you think those "certain patches" will never be incorporated to
> LibreOffice? Can you give links in gerrit to the patches?

I doubt the efficiency of talking to developers about something they aren't interested in. Like in ticket here -- it turns out it is possible to build a crippled installation in the same (!) environment as 4.3.7.2, and ticket is closed.

Anyway, the patch (or the essence of it) is in #45284; additional research is in #90739. 

For 4 years now I have to keep a LO build tree just to have the .DOC with formulas imported uncrippled.
Seeing as the build-from-source version is now crippled (granted, on my system), it's again 'work in TDF version, import in built one'.

There were also an issues with bibliography export to Word, seemingly resolved in #88697 (I have some new doubts about the developer's solution, though).
Comment 11 Buovjaga 2015-12-31 12:04:10 UTC
(In reply to Yury from comment #10)
> Like in ticket here -- it turns out it is possible to build a
> crippled installation in the same (!) environment as 4.3.7.2, and ticket is
> closed.

I closed the report, because build problems do not belong in Bugzilla.

Patches have to be submitted to gerrit, not Bugzilla: https://wiki.documentfoundation.org/Development/gerrit
Comment 12 Oliver Specht (CIB) 2016-02-15 08:36:44 UTC
> I closed the report, because build problems do not belong in Bugzilla.
> 
The issue is about a performance regression.
Loading the attached document - not really big - takes a lot longer in current master than in 5.0.
Comment 13 Yury 2016-02-15 08:48:31 UTC
With kindest help from Jan-Marek Glogowski (thank you!), the problem is localised as caused by VCL plugin selected by default setting (same as selected environment variable SAL_USE_VCLPLUGIN=kde).

Plugins selected by SAL_USE_VCLPLUGIN=gtk or SAL_USE_VCLPLUGIN=gen settings do NOT have this problem.
Comment 14 Buovjaga 2016-02-15 09:11:47 UTC
(In reply to Yury from comment #13)
> With kindest help from Jan-Marek Glogowski (thank you!), the problem is
> localised as caused by VCL plugin selected by default setting (same as
> selected environment variable SAL_USE_VCLPLUGIN=kde).
> 
> Plugins selected by SAL_USE_VCLPLUGIN=gtk or SAL_USE_VCLPLUGIN=gen settings
> do NOT have this problem.

Ok that explains why TDF's build did not show the problem.
Comment 15 Buovjaga 2016-02-26 14:39:08 UTC
I could not repro slowness, but I just got a powerful new desktop (Skylake i7-6700k) :)
Cpu hovers at 44%, if I press and hold page down.

I should test in a slower machine.

Arch Linux 64-bit KDE Plasma 5
Version: 5.1.0.3
Build ID: 5.1.0.3 Arch Linux build-1
CPU Threads: 8; OS Version: Linux 4.4; UI Render: default; 
Locale: fi-FI (fi_FI.UTF-8)
Comment 16 Yury 2016-02-26 16:21:49 UTC
(In reply to Beluga from comment #15)
> I could not repro slowness, but I just got a powerful new desktop (Skylake
> i7-6700k) :)
> Cpu hovers at 44%, if I press and hold page down.

I wouldn't call LO spending 44% of such a cpu on a PgDn 'not slow'...

The problems with that VCL plugin came with something packaged in slackware 14.1, anyway, they are not limited to the newest systems and distros.
Comment 17 Buovjaga 2016-02-26 16:43:42 UTC
Yeah, I guess we can throw this to NEW.
Comment 18 Yury 2016-03-24 19:04:34 UTC
Hold that pose, guys!

The 5.1.2RC1 (5.1.2.1.0) I've built from RC source with the same set of configuration options on the same system. The new build:

1) has NO issue with gtk VCL plugin

2) additionally, definitely feels 'snappish' w/r to the response time; very pleasant improvement!

NB I'm building with gtk3 disabled, so maybe all's not all that well with gtk plugin, but I'm completely satisfied with the resolution already. If gtk3 test is needed anyway on my side, specifically, do tell.

My heartfelt thanks to all concerned and involved. :) Just don't let this happen again, please. :))
Comment 19 Buovjaga 2016-03-25 10:45:36 UTC
But wasn't your problem specifically related to GTK3? I still see the same CPU use with PgDn in a fresh build and KDE enabled.

Arch Linux 64-bit, KDE Plasma 5
Version: 5.2.0.0.alpha0+
Build ID: 44a6d8ac3063511a149d4abdd6c2a556b3f477fe
CPU Threads: 8; OS Version: Linux 4.4; UI Render: default; 
Locale: fi-FI (fi_FI.UTF-8)
Built on March 25th 2016
Comment 20 Yury 2016-03-25 12:07:47 UTC
(In reply to Buovjaga from comment #19)
> But wasn't your problem specifically related to GTK3? I still see the same
> CPU use with PgDn in a fresh build and KDE enabled.

I don't think so.

I definitely see gtk3 NOT compiled in (no telltale menus, for one), AND actually I use `gen` plugin anyway (just to have small point, bold face font in UI, not the one taken from gtk settings).

However, the CPU hogging IS there, it's just not so much in the way as before, on 5.1.0/5.1.1. On my system (linux 64-bit, 2 AMD cores on 2.62Ghz, 4G RAM), a single PgDn/PgDn drives `top` CPU for soffice.bin to 0.7...1.0%, and I can easily make it go up to 9...10% by pressing and holding PgUp/PgDn.

Like I said, it's the feel that got plusplus good. :)

If there are any tests you'd like me to take with my 5.1.2 build, please do tell (I have no more 5.1.0/5.1.1 builds here -- space conservation!)
Comment 21 Buovjaga 2016-03-25 18:38:03 UTC
Created attachment 123853 [details]
Callgrind with 5.2

Arch Linux 64-bit, KDE Plasma 5
Version: 5.2.0.0.alpha0+
Build ID: 96c1ae1d8e78ae8b9bd7d4001645cad24d62b720
CPU Threads: 8; OS Version: Linux 4.4; UI Render: default; 
Locale: fi-FI (fi_FI.UTF-8)
Built on March 25th 2016
Comment 22 Michael Meeks 2016-03-25 19:18:21 UTC
Callgrind profile shows this is run with the KDE / Qt backend. I see around 10bn pcycles inside OutputDevice::Layout - of the ~22bn in the trace. That time is split between harfbuzz and fontconfig. Is it possible you have some huge & complicated fontconfig setup ? or a vast number of fonts and/or shapers ?

Oddly nothing jumps out at me from the trace as obviously, dominatingly stupid =)

You say it is slow to load too ? the doc load supposedly takes 4bn pcycles which doesn't seem unreasonable.

No obvious inspiration here - sorry ...
Comment 23 Buovjaga 2016-03-25 19:34:50 UTC
It was never slow to load for me.
Comment 24 Buovjaga 2016-03-27 13:52:09 UTC
I have 962 fonts in my system.
Yury: how many fonts do you have? The total is shown in the lower right corner of KDE font manager.
Comment 25 Yury 2016-03-27 17:29:50 UTC
(In reply to Buovjaga from comment #24)
> I have 962 fonts in my system.
> Yury: how many fonts do you have? The total is shown in the lower right
> corner of KDE font manager.

On my system the number shown is 285.
Comment 26 Yury 2016-09-20 16:32:02 UTC
With

Version: 5.2.1.2
Build ID: 31dd62db80d4e60af04904455ec9c9219178d620
CPU Threads: 2; OS Version: Linux 4.4; UI Render: default; 
Locale: ru-RU (en_GB.UTF-8); Calc: group

I can make the soffice go to 40+ % of cpu load in `top` by pressing and holding Pgdn.
Page rulers switch off and on when page scrolling with perceptible delay.
Delay on 'About' dialog is about 1 sec, maybe more (didn't stopwatch it).
All that's with 'gtk' backend.

With 'gen' vcl backend the cpu load reachable with holding down the PgDn is up to 25%, however, the other delays look about the same.

The fonts number, like discussed before, is 209.
Comment 27 Jan-Marek Glogowski 2016-10-02 18:05:06 UTC
Master got the commit 9f4af777a832d8a0b9a21d793d421fa6228131e0 for slow loading in non-double-buffering backends, like KDE4.

Author:     Jan-Marek Glogowski <glogow@fbihome.de>
AuthorDate: Mon Jul 18 13:21:14 2016 +0200
Commit:     Noel Grandin <noelgrandin@gmail.com>
CommitDate: Wed Jul 27 06:47:03 2016 +0000

    Don't Update() and Flush() status bar draws
    
    From reading the code of vcl::Window::Update, this already calls
    Invalidate and Flush in case of top-level widgets and also handles
    child windows.
    
    And there is no need to invalidate the progress bar text, if we
    just update the progress value.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/master/ . More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

The general performance regression is IMHO due to the new VCL scheduler. There is currently a feature branch feature/new-vcl-scheduler, but some LOK unit tests still fail, so it's WIP.
Comment 28 Buovjaga 2018-05-24 18:20:28 UTC
(In reply to Yury from comment #26)
> I can make the soffice go to 40+ % of cpu load in `top` by pressing and
> holding Pgdn.

I see this with a document created from scratch with just plain text. Let's close this.

Arch Linux 64-bit
Version: 6.1.0.0.alpha1+
Build ID: eeaf6dee2d278eaa037d95a756ad0ffab3314bc2
CPU threads: 8; OS: Linux 4.16; UI render: default; VCL: kde4; 
Locale: fi-FI (fi_FI.UTF-8); Calc: group
Built on May 24th 2018