Bug 98665 - Saving document is quite a bit slower than before
Summary: Saving document is quite a bit slower than before
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Writer (show other bugs)
Version:
(earliest affected)
5.1.1.3 release
Hardware: All All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard: target:5.4.0
Keywords: bibisectRequest, haveBacktrace, perf, regression
Depends on:
Blocks:
 
Reported: 2016-03-14 22:43 UTC by Ben
Modified: 2019-03-13 14:21 UTC (History)
10 users (show)

See Also:
Crash report or crash signature:


Attachments
Ben's machine details (7.21 KB, image/png)
2017-02-02 15:55 UTC, Ben
Details
Callgrind output from 5.4 (8.58 MB, application/x-xz)
2017-02-03 16:16 UTC, Buovjaga
Details
Content.xml file at the end where the bibliography starts (41.67 KB, image/png)
2017-05-31 00:39 UTC, Ben
Details
Memory settings for Ben's testing on Windows 10 machine (7.34 KB, image/png)
2017-07-09 20:43 UTC, Ben
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ben 2016-03-14 22:43:47 UTC
Saving a document seems to be quite a bit slower in 5.5.1 (my impression is at least a factor 2 slower) than in 4.3.7.2.
Comment 1 Usama 2016-03-15 11:45:32 UTC Comment hidden (obsolete)
Comment 2 Ben 2016-03-15 14:56:35 UTC
Usama,

I think the 4.3.7.2 (32 bit?) version got uninstalled upon the installing 64 bit 5.5.1 version on Windows 10, because I cannot find the 4.3.7.2 version anymore. So I cannot measure it. When I tried to install 4.3.7.2, I saw a message that the newer version of LibreOffice (5.5.1) would need to be removed first. Hence, I cannot measure the performance of both version simultaneously. It now takes about 50 seconds to save my 450 page document which has lots of formulas, tables and text.

As an aside, unfortunately, a new Bibliography database was created when I installed 5.5.1 and 4.3.7.2 was auto-removed. I lost all my Bibliography references in the DB. This may need to be taken into account in the Installer.
Comment 3 Usama 2016-03-15 17:27:21 UTC
Hello Ben,

Thank you for providing this info. Unfortunately I do not have a machine with a 4.x.x version. Let's wait for another helper.

Regarding your issue with the Bibliography database I suggest you search for an open bug regarding it, if you there is no one then open a new bug report. Data loss issues should take high priority.
Comment 4 Armin Le Grand 2016-03-16 14:54:11 UTC
@Ben, you can install another version to a directory of your choice on Win using the shell or 'execute', the 'msiexec /a <packagename.msi>'. You will be asked to where to install. HTH!
Comment 5 Ben 2016-03-16 14:57:14 UTC
Armin,

Unfortunately, I'm not an expert coder / administrator.
I look forward to the next official version to download and install.
Any idea when this might be created?
Comment 6 Buovjaga 2016-03-24 17:18:03 UTC
An easy way to install various versions is this tool: https://wiki.documentfoundation.org/SI-GUI
Comment 7 raal 2016-05-12 06:42:22 UTC Comment hidden (obsolete)
Comment 8 Alex Thurgood 2016-07-19 14:25:23 UTC
(In reply to Usama from comment #3)


> Regarding your issue with the Bibliography database I suggest you search for
> an open bug regarding it, if you there is no one then open a new bug report.
> Data loss issues should take high priority.


Unfortunately a known and longstanding issue with installation packets. Each new version of LO replaces the buit-in bibliography database (because it is hardcoded into the app). As a user, one needs to remember to save the bibliography database before installing a new version, and then copy it back to the appropriate place.
Comment 9 QA Administrators 2017-01-31 00:29:54 UTC Comment hidden (obsolete)
Comment 10 Ben 2017-02-02 15:54:05 UTC
I have a 500 page document with lots of formulas and references. It takes about 70 seconds to save it on a Windows 10 machine (see attachment) after a single character change. This may be representative for the size of the document. Any further optimization would be very much appreciated but the ticket can be closed as far as I'm concerned.
Comment 11 Ben 2017-02-02 15:55:28 UTC
Created attachment 130853 [details]
Ben's machine details

I experience about the same performance with I5 and I7 machine.
Comment 12 Buovjaga 2017-02-02 15:57:48 UTC
So you cannot attach the document publicly? This is one way to anonymize https://wiki.documentfoundation.org/QA/Bugzilla/Sanitizing_Files_Before_Submission#Sanitize_file_text
Comment 13 Ben 2017-02-02 17:53:32 UTC
Unfortunately, I cannot attach the document. It is a book I'm working on. I can see if I can build a document by copying the same pieces multiple times into it. I may have done this before for another ticket. I can try later.
Comment 14 Ben 2017-02-02 21:34:31 UTC
I tried to upload a document such that the save time can be tested. However, I got an error message: 413 Request Entity Too Large.

You can find the document I tried to upload under ticket ID 98731 as an attachment.
Comment 15 Buovjaga 2017-02-03 07:35:29 UTC
(In reply to Ben from comment #14)
> I tried to upload a document such that the save time can be tested. However,
> I got an error message: 413 Request Entity Too Large.
> 
> You can find the document I tried to upload under ticket ID 98731 as an
> attachment.

Ok, I confirm the regression when trying with attachment 123998 [details].

Saving in 5.4 takes 70 seconds.
Saving in 4.3 takes 16 seconds.

I think I will do a callgrind trace on Linux later.

Win 7 Pro 64-bit Version: 5.4.0.0.alpha0+
Build ID: c6dd735afb2e1b3837c4f8c5659f52fafab4c56f
CPU Threads: 4; OS Version: Windows 6.1; UI Render: default; 
TinderBox: Win-x86@42, Branch:master, Time: 2017-01-30_01:52:54
Locale: fi-FI (fi_FI); Calc: group

4.3.0.1
Comment 16 Buovjaga 2017-02-03 16:16:27 UTC
Created attachment 130884 [details]
Callgrind output from 5.4

I took a callgrind from the slow saving.

Arch Linux 64-bit, KDE Plasma 5
Version: 5.4.0.0.alpha0+
Build ID: b12823aa81003e80372bd89db79bd6ba8e032a95
CPU Threads: 8; OS Version: Linux 4.9; UI Render: default; VCL: kde4; 
Locale: fi-FI (fi_FI.UTF-8); Calc: group
Built on February 1st 2016
Comment 17 Michael Stahl (allotropia) 2017-02-03 21:25:14 UTC
the obvious problem in the callgrind of comment #16 is that
the table styles export is cubic.

SwXTextTables::getByIndex calls SwDoc::GetTableFrameFormatCount,
which does a loop over SwDoc::mpTableFrameFormatTable, then it calls
SwDoc::GetTableFrameFormat, which does another loop over the array.

SwXTextTableStyle::isInUse calls SwXTextTables::getByIndex
in a loop for n/2 times on average, where n is # of tables.

SwXTextCellStyle::isInUse calls SwXTextTableStyle::isInUse
once, and is called once per cell.

so we do about n * n/2 * m just to determine if styles are used,
that's pretty bad.

the SwX*Styles code was apparently added in commit
17f440e7e7f1621edebc58f8be5e85b68ee7dcf4
"GSoC Table Styles, TableStyle isInUse, isUserDefined, mutex fixes"
Comment 18 Tomaz Vajngerl 2017-02-07 17:31:03 UTC
Hi, 

I was optimizing this the other day - the patch in gerrit [1] should help a bit, but it would be nice if we didn't have to filter the collection constantly.

[1] https://gerrit.libreoffice.org/34008
Comment 19 Buovjaga 2017-02-08 13:49:19 UTC
(In reply to Tomaz Vajngerl from comment #18)
> Hi, 
> 
> I was optimizing this the other day - the patch in gerrit [1] should help a
> bit, but it would be nice if we didn't have to filter the collection
> constantly.
> 
> [1] https://gerrit.libreoffice.org/34008

Ok, on my Linux machine the slow saving took about 55 seconds.
With your patch, it took about 8 seconds!

Arch Linux 64-bit, KDE Plasma 5
Version: 5.4.0.0.alpha0+
Build ID: 3598740b1cf8f679567e1536768084924e547616
CPU Threads: 8; OS Version: Linux 4.9; UI Render: default; VCL: kde4; 
Locale: fi-FI (fi_FI.UTF-8); Calc: group
Built on February 8th 2016
Comment 20 Commit Notification 2017-02-17 22:41:14 UTC
Tomaž Vajngerl committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=1334702ec3c92484c70954ce8474882ae5da6764

tdf#98665 optimize table format style access

It will be available in 5.4.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 21 Xisco Faulí 2017-04-13 09:05:54 UTC
Hello,
Is this bug fixed?
If so, could you please close it as RESOLVED FIXED?
Comment 22 Ben 2017-05-17 22:54:52 UTC
Although it may not be not pertinent to this issue, opening a 542 page document is faster than before.

Saving a document, even after a single character change take longer. It  executes two cycles. The first cycle, which shows the progress bar, is very fast. The second cycle, which does not show as progress bar anymore, takes minutes.
Comment 23 Ben 2017-05-17 22:56:03 UTC
My last comment pertains to Version: 5.4.0.0.alpha1 (x64) on Windows 10.
Comment 24 Buovjaga 2017-05-18 05:56:18 UTC
(In reply to Ben from comment #22)
> Although it may not be not pertinent to this issue, opening a 542 page
> document is faster than before.
> 
> Saving a document, even after a single character change take longer. It 
> executes two cycles. The first cycle, which shows the progress bar, is very
> fast. The second cycle, which does not show as progress bar anymore, takes
> minutes.

My test on Windows had the first cycle take 16 seconds and then maybe 5-10 seconds for the UI to become responsive again. So definitely better than the time in my comment 15

Win 7 Pro 64-bit Version: 5.4.0.0.alpha0+ (x64)
Build ID: 1f73ac90da31b62ccf6f040c68e5c0150171240f
CPU threads: 4; OS: Windows 6.1; UI render: default; 
TinderBox: Win-x86_64@42, Branch:master, Time: 2017-04-20_07:32:30
Locale: fi-FI (fi_FI); Calc: CL
Comment 25 Ben 2017-05-31 00:37:10 UTC
Version: 5.4.0.0.alpha1 (x64)

1. Used 9.4 Mbyte test file attached in ticket ID 98731. Time to save 30 sec

2. Used another test file of 18 Mbyte (basically use content of ticket ID 98731 and copied the content many times over). Time to save:
1st time 40 sec
2nd time: 1 min, 20 sec
3rd time: 1 min, 20 sec
4rd time (closed file, re-opened, and saved): 1:25sec

3. Used the file of the book I'm working on which is currently 16 Mbyte. Time to save 2 min, 25 sec.

The file content seems to have an effect on save time.

The test file in ID 98731 does not contain a bibliography while both my book and the 18Mbyte file do so. The former uses an external bibliography database.
The content.xml file contains a lot of repeating characters, see attachment, which seems abnormal.
Comment 26 Ben 2017-05-31 00:39:05 UTC
Created attachment 133737 [details]
Content.xml file at the end where the bibliography starts
Comment 27 Ben 2017-05-31 00:40:26 UTC
Cannot load the test file I mentioned under item 2 due to upload file limit of 10 Mbyte.
Comment 28 Ben 2017-05-31 00:49:03 UTC
Sharable link to file mentioned under time 2

https://drive.google.com/file/d/0B24odiUNdUjtMmhnV1RHYWdhRlk/view?usp=sharing
Comment 29 Ben 2017-07-05 19:31:19 UTC
I removed the bibliography but it still took about 4 minutes to save the document. Machine i7 with hard drive.

I also used a DELL i5-6300U CPU @2.4 GHZ laptop with 16 GB of RAM and a 500 GB SSD drive. This did not much affect the time to save the document: still 3 minutes.
Comment 30 Ben 2017-07-07 20:53:28 UTC
I styled my 540 page document by adding Style Pages: Font Page (for chapter start), left page, right page, Table of Contents page, Acknowledgement page, Title page. I'm not sure if this is the cause, but believe it is. Now it takes more than 5 minutes to save the document.
Comment 31 Buovjaga 2017-07-09 18:48:18 UTC
(In reply to Ben from comment #28)
> Sharable link to file mentioned under time 2
> 
> https://drive.google.com/file/d/0B24odiUNdUjtMmhnV1RHYWdhRlk/view?usp=sharing

I tried saving this 3 times in a row and the save time stays constant: 37 seconds.

Arch Linux 64-bit, KDE Plasma 5
Version: 6.0.0.0.alpha0+
Build ID: b91cb7d137b5d8fd203bbdc1c4e3d0e851fd5aa6
CPU threads: 8; OS: Linux 4.11; UI render: default; VCL: kde4; 
Locale: fi-FI (fi_FI.UTF-8); Calc: group
Built on July 9th 2017
Comment 32 Ben 2017-07-09 20:42:24 UTC
Buovjaga,

I tried Version: 6.0.0.0.alpha0+ (x64)
Build ID: 2f2eba56d1f8ec5cdcadb4852e8856858477c008
CPU threads: 8; OS: Windows 6.19; UI render: default; 
TinderBox: Win-x86_64@42, Branch:master, Time: 2017-07-09_11:22:42
Locale: en-US (en_US); Calc: group

My time to save the provided test document was 3 minutes and 40 seconds. Memory settings are attached. Could they have an influence?

The difference between Linux and Windows might also have an effect.
Comment 33 Ben 2017-07-09 20:43:54 UTC
Created attachment 134566 [details]
Memory settings for Ben's testing on Windows 10 machine
Comment 34 Buovjaga 2018-04-16 14:19:12 UTC
The file in the Google Drive link clearly has some different problem.

I can open a new report for it after doing a new callgrind. I confirm it is slower to save on Windows (2 min) than Linux (30 sec).
Comment 35 Buovjaga 2018-04-17 13:02:53 UTC
(In reply to Buovjaga from comment #34)
> The file in the Google Drive link clearly has some different problem.
> 
> I can open a new report for it after doing a new callgrind. I confirm it is
> slower to save on Windows (2 min) than Linux (30 sec).

Created bug 117066