Bug 100442 - File with many images takes a long time to open and save
Summary: File with many images takes a long time to open and save
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
5.1.3.2 release
Hardware: x86-64 (AMD64) All
: medium major
Assignee: Not Assigned
URL:
Whiteboard: target:5.3.0
Keywords: bibisected, bisected, haveBacktrace, perf, regression
: 108537 (view as bug list)
Depends on:
Blocks: Image-Caching Calc-Images
  Show dependency treegraph
 
Reported: 2016-06-16 20:26 UTC by Denis K
Modified: 2019-04-19 11:37 UTC (History)
8 users (show)

See Also:
Crash report or crash signature:


Attachments
Callgrind with 5.3 (7.04 MB, application/x-xz)
2016-06-18 18:47 UTC, Buovjaga
Details
Bibisect log (3.09 KB, text/plain)
2017-10-31 21:09 UTC, Telesto
Details
Problematic XLSX (16.03 MB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
2019-04-19 11:31 UTC, Buovjaga
Details
Perf flamegraph of opening (1.58 MB, image/svg+xml)
2019-04-19 11:34 UTC, Buovjaga
Details
Perf flamegraph of saving (849.99 KB, image/svg+xml)
2019-04-19 11:37 UTC, Buovjaga
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Denis K 2016-06-16 20:26:18 UTC
Calc very slowly 5-10 min opens/saves the file with 3939 rows and many images. 
During opening this file 100% of disk utilization.

This file without images LO Calc opens/saves normally 5-10 sec.

MS Excel opens/saves this file normally - above 10-20 sec.

Версия: 5.1.3.2 (x64)
ID сборки: 644e4637d1d8544fd9f56425bd6cec110e49301b
Потоков ЦП: 4; Версия ОС: Windows 6.29; Отрисовка ИП: по умолчанию; 
Локаль: ru-RU (ru_RU)

Link to download xls-file with images (more then 10 Mb): https://yadi.sk/d/Sr7Z0SObsZ3BU
Comment 1 Buovjaga 2016-06-17 18:29:14 UTC
For me, it opened in under a minute, but I have a fast 6th generation Intel CPU.

Arch Linux 64-bit, KDE Plasma 5
Version: 5.3.0.0.alpha0+
Build ID: d2106d8c8c1af64924f5423223388dbbe99af352
CPU Threads: 8; OS Version: Linux 4.6; UI Render: default; 
Locale: fi-FI (fi_FI.UTF-8)
Built on June 16th 2016
Comment 2 Denis K 2016-06-18 04:53:28 UTC
Yes, on less powerful processors such as Pentium or Celeron is impossible to work with this file.
Comment 3 Buovjaga 2016-06-18 18:47:33 UTC
Created attachment 125722 [details]
Callgrind with 5.3

Got a callgrind trace.

Arch Linux 64-bit, KDE Plasma 5
Version: 5.3.0.0.alpha0+
Build ID: c13f60e7cd18df6b0ab70289f5b91ee01e4ae126
CPU Threads: 8; OS Version: Linux 4.6; UI Render: default; 
Locale: fi-FI (fi_FI.UTF-8)
Built on June 18th 2016
Comment 4 Michael Meeks 2016-06-30 12:54:48 UTC
A very odd trace; the 'lots of images' thing is pretty clear though in the kcachegrind trace. We spend a huge chunk of our time doing odd graphics things. Much of the time seems to be spent in:

GraphicManager::CheckSizeOfSwappedInGraphics(GraphicObject const*)

And worse -acutally- inside that method ;-) rather than in its callees - 1 trillion pcycles or so (wow)
Comment 5 Michael Meeks 2016-06-30 13:37:05 UTC
Interestingly the GraphicManager:

        for(sal_uInt32 a(0); mnUsedSize >= nMaxCacheSize && a < aCandidates.size(); a++)
        {
            // swap out until we have no more or the goal to use less than nMaxCacheSize
            // is reached
            GraphicObject* pObj = aCandidates[a];
            if( pObj == pGraphicToIgnore )
            {
                continue;
            }
            if (std::find(maObjList.begin(), maObjList.end(), pObj) == maObjList.end())
            {
                // object has been deleted when swapping out another one
                continue;
            }

            // do not swap out when we have less than 16KB data objects
            if(pObj->GetSizeBytes() >= (16 * 1024))
            {
                pObj->FireSwapOutRequest();
            }

Looks like it could (potentially) loop infinitely - if we blew the cache size with lots of small (ie. less than 16k) images ...

I wonder if this is what happens in this case (?).

Of course there is a nice N^2 in the loop there with the std::find and the iteration - but can we -really- have -that- many images ? =)
Comment 6 Michael Meeks 2016-06-30 13:41:15 UTC
The hang on load I got is here:

#5  0x00007fdaa8db73dd in GraphicManager::ImplCheckSizeOfSwappedInGraphics(GraphicObject const*) (this=0x2017da0, pGraphicToIgnore=0xb121d80) at /data/opt/libreoffice/master/svtools/source/graphic/grfmgr2.cxx:217
#6  0x00007fdaa8daeef3 in GraphicObject::ImplAfterDataChange() (this=0xb121d80) at /data/opt/libreoffice/master/svtools/source/graphic/grfmgr.cxx:71
#7  0x00007fdaa8daf94a in GraphicObject::ImplAssignGraphicData() (this=0xb121d80) at /data/opt/libreoffice/master/svtools/source/graphic/grfmgr.cxx:167
#8  0x00007fdaa8db1dc9 in GraphicObject::SetGraphic(Graphic const&, GraphicObject const*) (this=0xb121d80, rGraphic=..., pCopyObj=0x0) at /data/opt/libreoffice/master/svtools/source/graphic/grfmgr.cxx:698
#9  0x00007fdaa99b11e5 in SdrGrafObj::NbcSetGraphic(Graphic const&) (this=0xb121930, rGrf=...) at /data/opt/libreoffice/master/svx/source/svdraw/svdograf.cxx:456
#10 0x00007fdaa99b1281 in SdrGrafObj::SetGraphic(Graphic const&) (this=0xb121930, rGrf=...) at /data/opt/libreoffice/master/svx/source/svdraw/svdograf.cxx:466
#11 0x00007fdaa9b82c25 in SvxGraphicObject::setPropertyValueImpl(rtl::OUString const&, SfxItemPropertySimpleEntry const*, com::sun::star::uno::Any const&) (this=0x7fda53e2e708, rName="Graphic", pProperty=0x2aa7190, rValue=uno::Any("com.sun.star.graphic.XGraphic": {<com::sun::star::uno::XInterface> = {_vptr.XInterface = 0x7fda5250f170}, <No data fields>})) at /data/opt/libreoffice/master/svx/source/unodraw/unoshap2.cxx:1535

So far we have this many images:

(gdb) p maObjList
$1 = std::__debug::vector of length 4511, 

so some 16 million in the N^2 - should be ~instant - modulo the horribly slow debugging iterators (but Denis shouldn't see those in a release build).

(gdb) p pObj->GetSizeBytes()
$2 = 32400
(gdb) p mnUsedSize
$3 = 92467620
(gdb) p nMaxCacheSize
$4 = 67110000

Which is interesting ... on that basis increasing the graphics cache size to something much larger might significantly help as a workaround - but is no real fix, obviously.
Comment 7 Commit Notification 2016-09-05 16:30:44 UTC
Aron Budea committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=fdc867a4bc6ac2810732af898c620a889cde725c

tdf#100442 use unordered_set for GraphicManager's maObjList

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 8 Denis K 2016-11-21 12:35:44 UTC
Bug is present.

Версия: 5.3.0.0.alpha1
ID сборки: f4ca1573fcf445164c068c1046ab5d084e1b005f
Потоков ЦП: 2; Версия ОС: Windows 6.1; Отрисовка ИП: по умолчанию; 
Локаль: ru-RU (ru_RU); Calc: CL
Comment 9 Denis K 2017-02-10 18:42:19 UTC
Nothing changed. The file opens too long.

Версия: 5.3.0.3 (x64)
ID сборки: 7074905676c47b82bbcfbea1aeefc84afe1c50e1
Потоков ЦП: 4; Версия ОС: Windows 6.29; Отрисовка ИП: по умолчанию; Способ разметки: новый; 
Локаль: ru-RU (ru_RU); Calc: single
Comment 10 Xisco Faulí 2017-04-13 09:08:01 UTC
(In reply to Denis K from comment #9)
> Nothing changed. The file opens too long.
> 
> Версия: 5.3.0.3 (x64)
> ID сборки: 7074905676c47b82bbcfbea1aeefc84afe1c50e1
> Потоков ЦП: 4; Версия ОС: Windows 6.29; Отрисовка ИП: по умолчанию; Способ
> разметки: новый; 
> Локаль: ru-RU (ru_RU); Calc: single

How long it takes to open the document in your computer? How long it takes in a previous version of LibreOffice?
Comment 11 Denis K 2017-04-17 20:26:32 UTC
Opening this file takes about 3 minutes in version 5.3.2.2.

Версия: 5.3.2.2 (x64)
ID сборки: 6cd4f1ef626f15116896b1d8e1398b56da0d0ee1
Потоков ЦП: 4; Версия ОС: Windows 6.29; Отрисовка ИП: по умолчанию; Способ разметки: новый; 
Локаль: ru-RU (ru_RU); Calc: single
Comment 12 Denis K 2017-04-17 20:41:30 UTC
And the same time in version 5.2.6.2

Версия: 5.2.6.2 (x64)
ID сборки: a3100ed2409ebf1c212f5048fbe377c281438fdc
Потоков ЦП: 4; Версия ОС: Windows 6.29; Отрисовка ИП: по умолчанию; 
Локаль: ru-RU (ru_RU); Calc: single
Comment 13 Telesto 2017-10-31 20:52:57 UTC
Extreme slow opening with:
Version: 6.0.0.0.alpha1+
Build ID: 13c5dd1d98a480cb01ca8f24242c80e326e4ade8
CPU threads: 4; OS: Windows 6.3; UI render: default; 
TinderBox: Win-x86@42, Branch:master, Time: 2017-10-30_23:52:07
Locale: nl-NL (nl_NL); Calc: CL

Images didn't show up at all (or I didn't wait long enough)

Working fine with:
Version: 4.3.7.2
Build ID: 8a35821d8636a03b8bf4e15b48f59794652c68ba
Comment 14 Telesto 2017-10-31 21:09:39 UTC
Created attachment 137409 [details]
Bibisect log

Bisect to:

author	Zolnai Tamás <tamas.zolnai@collabora.com>	2014-11-07 15:53:47 (GMT)
committer	Zolnai Tamás <tamas.zolnai@collabora.com>	2014-11-07 15:54:58 (GMT)
commit	5771bde234ec65d79501d5eab938e06dfe43e232 (patch)
tree	e2855ef4f3bc1ddbeaa418ee8107c2e91eab742d
parent	673f3dfc3cc4190a01001280330ff0b63cd1cc7f (diff)
Try to fix ambiguity error
Change-Id: I93cf7497e52d065d00f762f61f79a4e8f8609bfc
Comment 15 XTR 2017-12-06 16:58:59 UTC
Bug Still here
Version: 6.0.0.0.beta1 (x64)
Build ID: 97471ab4eb4db4c487195658631696bb3238656c
CPU threads: 4; OS: Windows 6.1; UI render: default; 
Locale: ru-RU (ru_RU); Calc: CL


slow opening because libreoffice makes temporary file for each graphic object in document in user's temp dir,  
MS Excel doing nothing like this - much better performance
Needs major remake document with images loading!

and yes there is some problem with graphic cache management, like Michael Meeks says in ImplCheckSizeOfSwappedInGraphics,  has somebody make test without if(pObj->GetSizeBytes() >= (16 * 1024)) ?
Comment 16 XTR 2017-12-06 17:06:13 UTC Comment hidden (obsolete)
Comment 17 Buovjaga 2017-12-06 17:21:00 UTC
(In reply to XTR from comment #16)
> And BTW, where is graphic cache memory settings in LO 6.0.0.0?  Is it
> automatic now?

It was removed. It is not automatic, but the setting was raised.
Comment 18 Telesto 2017-12-06 18:27:25 UTC
(In reply to XTR from comment #16)
> And BTW, where is graphic cache memory settings in LO 6.0.0.0?  Is it
> automatic now?

FYI: Setting can still be changed in the Expert Configuration 
/org.openoffice.Office.Common/Cache/GraphibManager/TotalCacheSize
/org.openoffice.Office.Common/Cache/GraphibManager/ObjectCacheSize
/org.openoffice.Office.Common/Cache/GraphibManager/ObjectReleaseTime
/org.openoffice.Office.Common/Cache/DrawingEngine/OLE_Objects
and this setting seems to depend on latter:
/org.openoffice.Office.Common/Cache/Writer/OLE_Objects

See bug 110448
Comment 19 Telesto 2018-04-05 17:46:46 UTC Comment hidden (obsolete)
Comment 20 Telesto 2018-04-05 17:47:38 UTC
*** Bug 108537 has been marked as a duplicate of this bug. ***
Comment 21 Telesto 2018-04-05 17:58:41 UTC
Repro with
Version: 6.1.0.0.alpha0+
Build ID: 2325f9ac789cd12e5ecc9d239baf2558e1d678bb
CPU threads: 4; OS: Windows 6.3; UI render: default; 
TinderBox: Win-x86@42, Branch:master, Time: 2018-04-04_23:26:22
Locale: nl-NL (nl_NL); Calc: CL
Comment 22 Telesto 2018-05-23 09:53:06 UTC Comment hidden (obsolete)
Comment 23 Xisco Faulí 2018-06-05 19:15:22 UTC Comment hidden (obsolete)
Comment 24 Buovjaga 2019-04-19 11:31:47 UTC
Created attachment 150881 [details]
Problematic XLSX

Let's attach the file, so it doesn't get lost
Comment 25 Buovjaga 2019-04-19 11:34:44 UTC
Created attachment 150882 [details]
Perf flamegraph of opening

Used exit post startup. Normally the CPU keeps being hammered after opening.

Arch Linux 64-bit
Version: 6.3.0.0.alpha0+
Build ID: 1fee3f1da6291bfbcd75455512726215d41b3e83
CPU threads: 8; OS: Linux 5.0; UI render: default; VCL: gtk3; 
Locale: fi-FI (fi_FI.UTF-8); UI-Language: en-US
Calc: threaded
Built on 18 April 2019
Comment 26 Buovjaga 2019-04-19 11:37:15 UTC
Created attachment 150883 [details]
Perf flamegraph of saving

Saving is actually pretty quick