Created attachment 96007 [details]
calc document with many comments (~8k)

calc document with many comments (~8k) showing performance problems in load/save, copy/paste.

(In reply to comment #1)
> Created attachment 96007 [details]
> calc document with many comments (~8k)
> 
> calc document with many comments (~8k) showing performance problems in
> load/save, copy/paste.

I can definitely confirm this. Loading of files with 5000+ comments takes forever and even makes calc crash reproducably.

Can you get a valgrind trace of the crash on load ? =)

with libreoffice 4.2.3.3 i get this error when i duplicate the comments in the example file when i try to save the file:

"Error saving the document test-1:
Write Error.
The file could not be written."


it doesn't crash for me, so i can't give a valgrind trace. are there some libreoffice specific docs how to make a valgrind trace on windows?
are these useful without debug symbols?

Using the very recent master build, for me

(In reply to comment #0)

> - copy and paste cells with comments is very slow

It works quite snappy.  Copy and paste works quick, moving of cells etc also works fine.

> - saving / loading such a document is very slow

Yes, these are slow.

> - memory usage on save is enourmous

Around 800 MB VM (on Linux) which is quite reasonable.

(In reply to comment #5)

> > - copy and paste cells with comments is very slow
> 
> It works quite snappy.  Copy and paste works quick, moving of cells etc also
> works fine.

Did you test with the attached document? If you have only one cell with one comment this one is indeed processed quickly. But if you have many *other* cells with comments copy/paste of this one cell is slow.

(In reply to comment #6)
> (In reply to comment #5)
> 
> > > - copy and paste cells with comments is very slow
> > 
> > It works quite snappy.  Copy and paste works quick, moving of cells etc also
> > works fine.
> 
> Did you test with the attached document?

Yes. My comments are all based on the attached document.

BTW, how does the load and save compare with the previous versions, say, 4.1.6?  Does anyone have any data on that?

(In reply to comment #8)
> BTW, how does the load and save compare with the previous versions, say,
> 4.1.6?  Does anyone have any data on that?

Tested it with LO 4.1.6.2 on WinXP 32Bit. Behaviour is the same - huge memory consumption, very very long time to process, saving file failed with error:

Steps i have done:

1. use the attached document
2. select whole columns A and B
3. copy selection (lasts about 1 minute)
4. select whole columns C and D
5. paste (lasts about 2 minutes)
6. save document

Step 5 lasts about 30 minutes and fails with error (translated):

Error saving document test.
Writing problem.
File could not be written.

(see screenshot)

Created attachment 98458 [details]
A screenshot of the error message when saving document.

Created attachment 98773 [details]
picture / profile of comment loading

Profile of load at:

http://users.freedesktop.org/~michael/callgrind-comment-load-34ae7b16d7e.txt.gz

Can be viewed in KCachegrind; help appreciated =)

163k calls to SvXMLTokenMap::SvXMLTokenMap is a bit odd - 6% of the load time.

Much of the grief seems to be the drawinglayer's fetish for measuring text repeatedly =) and SfxItemSet thrash.

I would assume that calc knows the size the comment boxes should be anyway (not that they are even visible so ... ;-)

Matuš Kukan committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=d2e0465d406b33139f3a97e1738020d6a7a6f6c3

fdo#76324: Use one static SvXMLTokenMap object, it's faster.



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.

Profiles of copy and paste of the columns can be found at:
http://people.freedesktop.org/~mkukan/callgrind.calc-comments-copy.out.gz
http://people.freedesktop.org/~mkukan/callgrind.calc-comments-paste.out.gz
I was not patient enough to wait until the operation ends under valgrind, but should be useful anyway.
That pasting is really strange: it creates new document out of clipboard data, then pastes the comments into temporary document from there and then finally pastes the columns into real document you see - or quite possibly I am totally mistaken of course :-)
Hope this helps.

Matuš Kukan committed a patch related to this issue.
It has been pushed to "libreoffice-4-3":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=29162832feafca9259dc1589d10f3cfd43ea8126&h=libreoffice-4-3

fdo#76324: Use one static SvXMLTokenMap object, it's faster.


It will be available in LibreOffice 4.3.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.

which version do i have to download to test the change?

is

http://dev-builds.libreoffice.org/daily/master/Win-x86@39/current/

correct?

if so, copy & paste 4k of comments is still incredible slow, it takes about 1 minute.

saving the document with 8k doesn't work, soffice aborts while saving with "Error saving the document test: Write Error. The file could not be written".

(In reply to comment #16)
> which version do i have to download to test the change?
> 
> is
> 
> http://dev-builds.libreoffice.org/daily/master/Win-x86@39/current/
> 
> correct?

Yes, I think so.

> if so, copy & paste 4k of comments is still incredible slow, it takes about
> 1 minute.
> 
> saving the document with 8k doesn't work, soffice aborts while saving with
> "Error saving the document test: Write Error. The file could not be written".

http://cgit.freedesktop.org/libreoffice/core/commit/?id=d2e0465d406b33139f3a97e1738020d6a7a6f6c3 was only about loading the document.

Matuš Kukan committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e7a3aacff7d28577dee371ed5b27317522db7b3b

fdo#76324: Make pasting a lot of cell notes faster by disabling broadcasting.



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.

Matuš Kukan committed a patch related to this issue.
It has been pushed to "libreoffice-4-3":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=6188a7197aae39a21ef80d1b177f5be613dc9808&h=libreoffice-4-3

fdo#76324: Make pasting a lot of cell notes faster by disabling broadcasting.


It will be available in LibreOffice 4.3.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.

Matuš Kukan committed a patch related to this issue.
It has been pushed to "libreoffice-4-2":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=90132a32f8bc2e27c097d79c0cc1ddd7cae35da6&h=libreoffice-4-2

fdo#76324: Make pasting a lot of cell notes faster by disabling broadcasting.


It will be available in LibreOffice 4.2.6.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.

I believe pasting is much faster after e7a3aacff7d28577dee371ed5b27317522db7b3b,
so this bug is about saving the file now?

If anybody is interested, here is a profile:
people.freedesktop.org/~mkukan/callgrind.calc-comments-save.out.gz
There are probably more issues but one is with SfxItemSets.. that's not something for me.

MABs should be priority highest.

(In reply to spam from comment #0)
> when using many comments in cells of a spreadsheet, calc becomes very slow:
> 
> - copy and paste cells with comments is very slow

I do not reproduce this anymore in 4.4.0.0 alpha2

> - saving / loading such a document is very slow
> - memory usage on save is enourmous

this is still reproducible and I get error on save.

moving this to mab4.3 since 4.2.x reached the end of life

- in 5.0.0.2 (x64) its still very slow when copy / pasting
- saving a document with a lot of comments still isn't possible: lo uses a lot of memory and then aborts with the same error message

I was able to copy and paste the columns several times without any issue.  The application does hang for a second while it is pasting but I don't think this should be considered a bug given the amount of operations being done.  Potentially this issue has been resolved in the newer LO versions?

OS: Linux Mint 17.2
LO: 5.0.2.2

> I was able to copy and paste the columns several times without any issue.  The application does hang for a second while it is pasting but I don't think this should be considered a bug given the amount of operations being done.  Potentially this issue has been resolved in the newer LO versions?

imo it is better, but not fixed. still opening the example document takes ~20 seconds with LO: 5.0.2.2.

also CTRL+C takes a very long time, the same for CTRL+V.

very likely the complexity of these operations are still O(c^n) when it should be O(n).

my test file bug-comment.ods 340 ko  with a lot of comments

Linux 3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) i686 GNU/Linux  XFCE 4.12
vendor_id	: AuthenticAMD
model name	: Mobile AMD Sempron(tm) Processor 3600+
memory 3 GO
LBO Version: 4.3.3.2 32bits
Build ID: 430m0(Build:2)
Locale 	 fr_FR		
			 		
Open file and save = First recording time		~15 minutes and 26s
Modify file and save second recording time		~ 7s

Delete all comments - time to delete 15 minutes and 15 s
Save the new file witout comments ~5s

Other test
----------
Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux  XFCE 4.12
vendor_id	: GenuineIntel
model name	: Intel(R) Core(TM) i3-4170 CPU @ 3.70GHz  
memory 4 GO
+ disk SSD	Kingston SSDNow V300 Series 120 Go
LBO Version: 5.0.4.2  (1:5.0.4~rc2-2~bpo8+1 jessie-backport)
Build ID: 00m0(Build:2)
Locale : fr-FR (fr_FR.UTF-8)
Open file and save = First recording time		~3 minutes and 25s
Modify file and save second recording time		~ 2s

Delete all comments - time to delete 3 minutes and 20 s
Save the new file witout comments ~2s

For informations
-----------------
Test on
my test file bug-comment.ods 340 ko  with a lot of comments
XP SP3-SSE memory 1Go
LBO Version	 4.4.5.2
Build ID	 a22f674fd25a3b6f45bdebf25400ed2adff0ff99
Locale 	 fr_FR		 		 		
Open file and save = First recording time		~23s	
Modify file and save second recording time		~23s

Vista memory 3Go	
vendor_id	: AuthenticAMD
model name	: Mobile AMD Sempron(tm) Processor 3600+
LBO Version: 5.0.3.2	
Build ID: e5f16313668ac592c1bfb310f4390624e3dbfb75	
Locale 	 fr_FR		 		 		
Open file and save = First recording time	~20s
Modify file and save second recording time	~10s

Created attachment 123072 [details]
file bug_comment.ods + bug_NOcomment.ods

From  Comment 27 JCE 2016-02-14 13:14:34 UTC 
new test on Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux  XFCE 4.12
With 
with  LBO Version: 5.0.5.2

bug-comment.tar.gz
- file bug_comment.ods  with a lot of comments 
Open file and save = First recording time		~3 minutes and 25s
- file bug_NOcomment.ods 
Modify file and save second recording time		~ 2s

Problem is still there in LibreOffice 5.1.2.2.

Created attachment 125689 [details]
new test file

tested on a file with ~25k comments on these versions:
Version: 5.3.0.0.alpha0+
Build ID: a8bd44573b75d1399257d6f5d052611439607189
CPU Threads: 2; OS Version: Linux 4.1; UI Render: default; 
TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2016-06-13_23:46:49
Locale: it-IT (it_IT.UTF-8)
OS:openSUSE Leap 42.1 (x86_64)

and the results are:
opening: 47 s 
closing: 14 s
saving: 2.41 m

and on:
Versione: 5.2.0.0.beta2
Build ID: ae12e6f168ba39f137fc110174a37c482ce68fa4
Thread CPU: 2; Versione SO: Linux 4.1; Resa interfaccia: predefinito; 
Versione locale: it-IT (it_IT.UTF-8)
OS:openSUSE Leap 42.1 (x86_64)

and the results are:
opening: 47 s
closing: 15 s
saving: 2.47 m

Created attachment 126728 [details]
A screenshot of performance in LO v5.1.4.2 (x64)

was taken *after* step 5 in comment #32 *before* saving

retested with

Version: 5.1.4.2 (x64) (8GB of MEM, 6,5GB free)
Build-ID: f99d75f39f1c57ebdd7ffc5f42867c12031db97a
CPU-Threads: 2; BS-Version: Windows 6.1; UI-Render: Standard; 
Gebietsschema: de-DE (de_DE)

with the following results (slightly different from original report, minor improvements noticeable)

> 1. use the attached document
loading time: 15s (LO was already running); memory consumption seems normal

> 2. select whole columns A and B
time: instantly; memory consumption seems normal

> 3. copy selection (lasts about 1 minute)
time: 26s; memory consumption seems normal

> 4. select whole columns C and D
time: instantly; memory consumption seems normal

> 5. paste (lasts about 2 minutes)
time: 17s; memory consumption seems normal

(screenshot in attachment #126728 [details] was taken here)

> 6. save document
Crash *without* any error message after 6s while memory consumption raised rapidly to about 500MB (of 6,5 free GB)

Any changes were lost.

I can confirm this bug and have some additional information.

Saving documents with many comments takes minutes or even hours, depending on complexity and hardware.

To be more precise: You press the save button and then LO seems to hang for a long time (other operations or even starting writer will not work, high CPU load on one core) and then it suddenly starts to save, so you see the progress bar on the bottom.

As stated already by someone else, it only takes long the first time the document is saved. If you save it later again, it is quick. Only after closing LO and opening the document again, the first save is slow again.

Interestingly, it seems to be relevant, which sheet is visible when you hit the save button. So as a workaround, I now switch to a sheet without comments, hit save and then it is quick. I can later save it on whatever sheet I am and it is still quick, as described above.

I thought it might have something to do with the thumbnail generation, so I tried to turn that off in the document properties, but it does not work, this option is automatically turned on after you close LO and open the document again. It might be another bug.

I use LibreOffice 5.1.4.2 that came with Ubuntu 16.04.
I am under the impression I did not have these problems before upgrading from Ubuntu 14.04 to 16.04.

Created attachment 130332 [details]
Spreadsheet with many comments

I'm experiencing similar symptoms in LO version 5.1.4.2 build ID 1:5.1.4-0ubuntu1, under Ubuntu 16.04 x64, with the attached file (which I think is somewhat smaller and has somewhat fewer comments than example files previously posted on this thread).  Like Peter, I believe the trouble started with a fairly recent (c. July 2016) upgrade.

Steps to reproduce problem:

1. Open the attached file in LO Calc.
2. Highlight (for example) cell E181.
3. Press ctrl-c to copy.
4. Highlight (for example) cell E182.
5. Press ctrl-v to paste.
6. Press ctrl-s to save.
7. CPU usage of soffice.bin jumps to near 100% and stays there, and LO hangs indefinitely (when I say "indefinitely" - on my last test, I gave up waiting and killed the soffice.bin process after about 25 minutes).

I can also confirm Peter's observations, that the situation is greatly improved if one creates a blank worksheet, and switches to that blank worksheet being the visible sheet, prior to saving, and that having done this once in a particular LO process, subsequent saves in the same LO process work well regardless of which sheet is visible, and that the problem then comes back again once LO is closed and re-opened.

Another oddity which also arose around July 2016 and may or may not be related: "save" no longer checks whether or not the document has been modified since it was last saved - it overwrites anyway.  This is the case irrespective of whether or not there are lots of cell comments (in fact, it applies to all LO components, not just Calc.)

Out of curiosity, I just installed Gnumeric 1.12.28, and tried the same series of steps, applied to the same file, in that package.  Saving completed nice and quickly.  Might there be mileage in a comparison of the comment-handling bits of the two codebases?

@Dan H
please DO NOT change version field in an improper way as you did.
it MUST indicate EARLIEST version the bug appeared (as indicated), not the latest.

so reverting it back to 4.2.2.1

My apologies.  I guess I should have read the documentation ;-).

Another datum that might be useful: I just retried my sequence of steps 1-6 from comment #34 in LO 5.1.4.2 (x64) build f99d75f39f1c57ebdd7ffc5f42867c12031db97a, under Windows 7 x64, and got a successful save within a few seconds.

However, with a larger version of the spreadsheet (i.e. with more non-blank cells and more comments -  about 11000 of each), the above Windows version of LO crashed on save with a "LibreOffice has stopped working" message.  Worse still, Peter's "make a blank worksheet visible" trick doesn't prevent the crash under Windows.  I can't post this larger example file because, er, LO crashes when I try to save it.

*** Bug 105499 has been marked as a duplicate of this bug. ***

As the person who posted the duplicate Bug 105499 I can confirm that it also happens with the test-document for this bug. 

Steps that, for me, are enough to reproduce the bug:
- open document (opens in reasonable time, but takes longer than the 180kB should need)
- hover over some comments, move mouse around
- just select the whole columns A+B by clicking on the columns Headers (grey A and B)

The last step freezes LO, CPU explodes. I have to kill LO.

I also played around with hardware acceleration, OpenGL, but it happens regardless of these settings. 

I also removed my ~/.config/libreoffice to test if the configuration is the problem - but no difference.

So, where are we at in terms of moving towards either a fix or a workaround for this? For example:

Has anyone found _any_ currently-available combination of OS environment and LO build in which the bug _can't_ be reproduced?  Or even better, has anyone done a bisect sequence that has isolated which commit/revert (either to the LO git tree or to a library's git tree) introduces/removes the bug?

Does anyone know which file in the LO source code tree contains the code that is being executed when the hang happens (last August, I tried searching the source tree myself, but I couldn't find the comment-handling code at all).

Thanks.

> Does anyone know which file in the LO source code tree contains the code that is being executed when the hang happens (last August, I tried searching the source tree myself, but I couldn't find the comment-handling code at all).

see the screenshot in https://bugs.documentfoundation.org/show_bug.cgi?id=76324#c11

Profiling.

Sorry it's taken me so long to say "thank you" for that last piece of information, but, er, thank you.

A few observations:

- If the heavy CPU usage really does relate to measuring text in order to produce a comment box of the appropriate size, that provides a possible explanation of why Gnumeric and Excel are able to perform faster, since they both appear to leave the appropriate sizing of the comment boxes for the user to do manually.

- Shortly after that trace was posted, a patch was posted on here that was supposed to mitigate the particular issue revealed in the trace.  Do any (or all) currently widely-available builds of LO contain that patch?

- Since both Peter and I have vague memories that the performance was better before we upgraded from Ubuntu 14.04 to 16.04, my next move is going to be to downgrade back to 14.04 (the LO build ID in which is 1:4.2.8-0ubuntu5).  I'll let y'all know whether it helps.

Results:

I repeated the procedure described in comment #34 under Ubuntu 14.04 (LO 4.2.8.2
Build ID: 420m0(Build:2)).  The save process at step 7 was completed without trouble in 20 seconds.

Could it be that Matúš' patch successfully solved the problem, and then someone  (partially) reverted it and reintroduced (the "save" part of) the bug before tommy27 got a chance to undertake the test described in the penultimate paragraph of comment #23?

I don't have time to retest now.

basically you are saying that the bug is not present in 4.2.8, right?

what about 5.2.5 and 5.3.0? is the issue still present or not?
can we label this as RESOLVED FIXED?

I think it's correct to say that the bug is not present in 4.2.8, _in the enivronment of Ubuntu 14.04_.

However, I don't think we can mark it as RESOLVED FIXED, because, from my own tests:

- The bug _is_ present in 5.1.4.2, in the environment of Ubuntu 16.04.
- The bug _is_ present in 5.1.4.2, in the environment of Windows 7.

and from your (tommy27's) tests:

- The bug _is_ present in 4.4.0.0 (you didn't mention what OS)

I now have another (puzzling) datum to add:

The bug is not present in 5.0.6.2, in the environment of Scientific Linux 7.3.

(Incidentally, I used exactly the same contents of the ~/.libreoffice directory in all three of the Linux environments I tested.  However, I also ran a separate 5.1.4.2/Ubuntu 16.04 test with a ~/.libreoffice tree freshly created by starting LO with no ~/.libreoffice present, and got the same result, i.e. the bug _was_ present.)

I don't remember the exact release dates of 4.2.8 and 5.2.4 but consider that 4.2.8 is a very late release of the 4.2.x branch and maybe a committ that was done in 5.2.5 could have been backported in 4.2.8 so that's why you don't see it in 5.2.4.

you should test 5.2.5 or 5.3.0 so have a clearer picture of the bug status

Well, it's good news on that front.  I installed (on Scientific Linux 7.3) LO 5.3.0.3-3, from the binary RPMs on libreoffice.org, and the bug was absent (by the same test set out in comment #34).

Anyone fancy trying the equivalent on Ubuntu, Windows or any other OS?

let's label this one as WORKSFORME
thanks for testing it in 5.3.x

i tested "calc document with many comments (~8k)" in 5.3.0.3 x64 / windows 7 x64:

copy & paste takes ~1 Minute and calc hangs in this time. when saving the document calc crashes:

http://crashreport.libreoffice.org/stats/crash_details/f4d601b5-bb77-48c7-ba26-75e131c1b84a

no, this is definitly not fixed.

crash seems reproduceable for me:

http://crashreport.libreoffice.org/stats/crash_details/c26b4027-c841-462e-8e4e-929c54b6b963

even opening "calc document with many comments (~8k)" and then directly saving it leads to a crash:

crashreport.libreoffice.org/stats/crash_details/9bdefcc2-dab5-4c29-8c51-0f3d287ae732

crashreport.libreoffice.org/stats/crash_details/de47c3e7-0f57-4e44-a24d-38a05b3c7ac0

sorry, enough bug-report-spam for now ;-)

let's keep this bug closed as RESOLVED FIXED and use bug 105888 for the crash...

*** Bug 97698 has been marked as a duplicate of this bug. ***

calc 5.3.0.3 is still very slow when working with the "calc document with many comments (~8k)" document.

it is faster then when the bug report was created, yes. IMHO its still way to slow as seeing "The Application is not responding" is a bug.

-> The initial reported problem still exists but became better:
1. opening the document takes ~30s
2. copying all comments takes ~15s
3. pasting all comments takes ~15s

when doing the same in a document with numbers only in the cells all of the operations are done "instantly".

the resulting document with comments is ~10 times larger in comparison to the numbers only version.


Not reopened as its out of my responsibility here. i hope such hangs are seen as bug, so please reopen when it applies.

I just opened a document with many (100 < x <1000) comments. 

If I copy a complete worksheet (move or copy) or rearrange the worksheet simply to another location at the bottom tab Calc crashes. 

Using 5.3.3.2 on Linux 64. Hardware Acceleration is disabled and it doesn't happen if I open a new worksheet or use another, less heavily commented document - therefore I think it is related to the comment problem.

Another way to experience the slowness is to delete a column. I'm using 5.3.4.2, Linux 64.

I have tried to copy 5 cells with comments to 8000 lines and libreoffice calc crashed. 

Product Details  
Version: 5.4.4.1 (x64)
Build ID: da790616461e15a10c95a80eb8ef8ee7b726c114
CPU threads: 4; OS: Windows 6.19; UI render: default; 
Locale: en-US (en_US); Calc: CL

Just a quick note that I'm continuing to test for this bug every time my machine pulls any package upgrades from the Scientific Linux repositories.  If/when the bug ever occurs on SL, I'll post a list of package upgrades that have happened since the last-known-good state, which I hope will help us narrow down what's causing the trouble.

*** Bug 114377 has been marked as a duplicate of this bug. ***

Version: 6.0.1.1
Build ID: SlackBuild for 6.0.1 by Eric Hameleers
CPU threads: 8; OS: Linux 4.14; UI render: default; VCL: kde4; 
Locale: nl-BE (en_US.UTF-8); Calc: group

deleting a column take's 10seconds
copy/past a column take's 5 seconds

Sorry to be the bearer of bad news, but this bug, or some variant of it, is still with us, and often getting worse on upgrade.  Specifically,

- on upgrading from Scientific Linux 7.4 (which contains LibreOffice 5.0.6.2) to Scientific Linux 7.5 (which contains LibreOffice 5.3.6.1), the time taken to save the "Many_comments_sample" spreadsheet posted as attachment #130332 [details] upthread increases from 25 seconds to 52 seconds (on the same hardware).

- I've also got a somewhat larger spreadsheet with about 16000 comments (which I can't post here because it contains some confidential data); in LibreOffice 5.0.6.2 under Scientific Linux 7.4, saving this spreadsheet takes about 30 seconds; whereas in LibreOffice 6.0.5.2 under Fedora 28 (and on faster hardware), saving this spreadsheet takes 3 minutes 50 seconds.

(In both cases, these results were obtained as per the test procedure I set out in comment #34 upthread.)

An oddity that might be of some diagnostic relevance: in the Fedora 28 version of LibreOffice, the "undo" operation for the single-cell copy and paste mentioned in the procedure at comment #34 takes much longer than the copy and paste operation itself.

opening the doc take's 4 seconds, saving 7 seconds. copy/past is fast
no high cpu load or high memory load
did follow description from comment 34 with attachment 130332 [details]
OS is slackware64current

Version: 5.2.7.2
Build ID: 2b7f1e640c46ceb28adf43ee075a6e8b8439ed10
CPU Threads: 8; OS Version: Linux 4.14; UI Render: default; VCL: kde4; 
Locale: nl-BE (en_US.UTF-8); Calc: group

Version: 5.4.8.0.0+
Build ID: cc68977f1be22ac0f4a15eb37e05ccba13a7a554
CPU threads: 8; OS: Linux 4.14; UI render: default; VCL: kde4; 
TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:libreoffice-5-4, Time: 2018-05-12_11:32:19
Locale: nl-BE (en_US.UTF-8); Calc: group

Let's keep this as fixed where it belongs. New issues should get new reports. I opened a report for a hang with GTK3: bug 121698

For me the bug is not resolved at all. Only the Linux version seems to be fixed. In Windows only slight improvements were noticable. The main problem seems to be already present. See below.

@Buovjaga: this is *not* a "new issue", so reopened.

Tested with Debian buster x64:

Version: 6.1.3.2
Build-ID: 1:6.1.3-2
CPU-Threads: 2; BS: Linux 4.18; UI-Render: Standard; VCL: x11; 
Gebietsschema: de-DE (de_DE.UTF-8); Calc: group threaded

Result:
- loading/saving copying/pasting of/in "calc document with many comments (~8k)" is substantially faster, so i would not longer consider it to be a problem
- loading/saving copying/pasting of/in "calc document with many comments (~8k)" is still slower than it would be with "normal" cell contents instead of comments
- no crashes
- no huge memory useage
- file saved successfully
- file saved in fair time

Tested with Windows Server 2008 R2 x64:

Version: 6.0.6.2 (x64)
Build-ID: 0c292870b25a325b5ed35f6b45599d2ea4458e77
CPU-Threads: 16; BS: Windows 6.1; UI-Render: Standard; 
Gebietsschema: de-DE (de_DE); Calc: group

Result:
- loading: 35s
- copying columns A+B: 85s
- pasting copied columns to C+D: 50s, mem useage grew to 280 MB
- saving: 90s, mem useage grew to 1.200 MB
- closing! calc: 150s (!)

During all the steps above, only one (of 16) cpu cores was used with a load of 100%.

Fine, let's keep as NEW. It's just that this has nearly 70 comments, has received fixing commits both in this report and other reports. The report is starting to get difficult to approach.

I do confirm the slowness on Windows 10 with latest daily build.

Here's an interesting datum: in LibreOffice 6.1.4.2-1.fc29 under Fedora 29, the save-after-small-edit test, on one of my production spreadsheets with many thousands of comments, takes about 80 seconds if both the GTK2 and GTK3 integration packages are installed, but takes only about 25 seconds if only the GTK2 integration package is installed.

dear guys, 

this is! a bug, 

it is a big! bug, 

and it's in the program for over 5 years now, still unresolved, 

ok, ranting doesn't help, but points like this are  not a good showpiece for a program being 'actively developed' ... :-( 

i think i stepped in a variant of (or something left while plastering on) this bug, i observed performance issues and tried to bring it to the point, it took me hours and days - see there: https://bugs.documentfoundation.org/show_bug.cgi?id=125545 and there: https://bugs.documentfoundation.org/show_bug.cgi?id=124692 till i came to the point it might be 'comment related'. 

how many people shall spend how many time and collect how many frustration till somebody takes this point serious and does something against it? 

most people won't do any analysis, when they use comments they will start with a few, no problem, use some more, no problem, and on continuation they will observe firstly little, then bigger and in the end major performance problems. 

but nobody tells them the reason, and they have no hint on the source, thus they will helplessly search around (as i did), be frustrated working with calc, and either switch to another product or change to other work. 

that's not a good thing! 

my two cents: 

- the issue is partially covered by enhancements in the program and faster hardware and OS's, but it's still in and 'working' (try changing the pattern in the first problem file to two cells with different content and different comments, and fill some area like a chessboard, thus breaking 'shared formulas' and similarity compression, you'll see massive performance impacts also in very fast systems, (90 seconds of 'not responding' and 'fan requesting cpu load' for a simple mouseclick in a sheet with 32k filled cells and 16k comments is not! 'the spreadsheet you ever wanted').

- if the problem comes from the calculation of the size of textboxes ??? - mentioned somewhere above - *switch it off!!!* while the comments are hidden. imho the normal use of comments is to have additional info in it which isn't needed all the time and shouldn't eat the screen, thus invisible and shown on mouseover. for this use its absolutely sufficient to calculate one! textbox for the cell actually showing it's comment. 

- in some situations i observed some flickering of comment boxes and 'resistance against the cursor and editing' of text in comments, may be there is something wrong, may be with new graphics features, may be ... 

- with the stuff provided in this thread it schould be a short and direct way to the source of the problem in the code, simple files, trivial data, simple 'functioning', there is little chance for the bug hiding against profiling, 

any help appreciated!

would / could somebody with sufficient rights set the importance to major please?

Severity is high/major per https://wiki.documentfoundation.org/images/0/06/Prioritizing_Bugs_Flowchart.jpg,

(first question 'no', second question 'yes'), 

imho raising of severity and resolving isn't only neccessary / good for affected users, but also a great relief for QA team (to get rid of the work of managing duplicates) ;-)

*** Bug 125545 has been marked as a duplicate of this bug. ***

pls. read comment https://bugs.documentfoundation.org/show_bug.cgi?id=125545#c10

sorry for insisting, and sorry for bad news, but there is still a bug, and it's bad. 

- there is also something critical in the handling of the bug here, it 'blocks it from visibility' - 

second problem first, new announcements about variants or 'things left' of this? bug after changes in versions are thrown out as 'duplicate', while this bug is kept somewhere between 'low prio', no-repo, delayed, but there is almost one 'property' of the actual situation - not yet mentioned in this thread - that hides the problem from being visible to testers, thus collecting norepo-comments: 

the -problem- seems off in plenty situations after a fresh load of affected files, it appears / re-appears after the first edit in the sheet and subsequent autosave. 

the 'issue left', or the problem for me: high cpu load on working in sheets / files with many comments, 

all sorts of hangs, crashes, 'not repsonding', time and cpu consuming load and save processes, delays in reaction to mouseclicks, and so on, 

easy test to check: - in windows - 
- produce a file with many cells with data, 
- hover around with the mouse, 
- no cpu load, check with the 'power'? tab (german: Leistung) in the task manager, 
- make a little change, wait for autosave (you can set it to a short time for this test, tools - options - load/save - general), 
- hover around with the mouse, 
- nopro, 
- add comments to plenty cells (just copy a not trivial pattern of cells with comments to a bigger area), 
- hover around, 
- no load, 
- wait for autosave, 
- hover around 
- significant cpu load. 

(set back autosave to a normal value after the test, too many saves might kill your ssd disk)

(it's not neccessary to click into the cells, just moving the mouse / pointer (touchpad in my case), (which changes the focus for which cell the comment is to be evaluated) is enough to produce load)

(it doesn't help that the load is 'only' 12 to 15 percent on powerful systems, it's the difference from being 0 to 1 percent before autosave what says 'problem') 

(also 'easements' like saving in 'fods' format (it has significantly mitigated the problem in some situations) or 'restart in safe mode' (no difference on my tests) do not bring away the problem) 

is this 'pinpointing' helpful to identify the source of the problem? 

shall i open a new bug for this special scenario and is there hope it will not be closed as duplicate? 

reg. 



b.

*** Bug 113599 has been marked as a duplicate of this bug. ***

Created attachment 154503 [details]
Flamegraph

On pc Debian x86-64 with master sources updated today + enable-symbols, I retrieved a Flamegraph during copying n times a cell with just a comment.

Noel: considering the Flamegraph, thought you might have some idea for some optims? (perhaps by finding a way to reduce notify calls?)

@Julien the cost is all in the std::sort, and we should be able to avoid that, since we already have a map that we can use to look the shape up

(In reply to Noel Grandin from comment #77)
> @Julien the cost is all in the std::sort, and we should be able to avoid
> that, since we already have a map that we can use to look the shape up

I agree most of time is spent in std::sort. But this one is called from accessible part which isn't used by gen rendering or Windows I think.
So we'll just speed up Linux non gen rendering part here.
That's why I had talked about notify calls.
Sometimes, we notify n times because of a loop, instead of just notifying once (eg for redrawing) after the end of loop.

Anyway, just to be sure to understand:
should we replace all "maZOrderedShapes" uses by "maShapesMap" or should we use "maShapesMap" in ScShapeDataLess to speedup sorting? Or did you have something else on mind?

@Julien, I have a patch the improves things a little, but what we really want here is to avoid creating the ScAccessibleDocument at all for the fake background ScDocument we use when copying stuff to the clipboard.

So what I would suggest it that you put a breakpoint on the ScAccessibleDocument constructor, do a copy operation, and see what path invokes it.
Than see if you can pass a boolean down to prevent ScAccessibleDocument being created at all. 

No idea if that will actually work out, or if it will cause other breakage, it's just one of those "try it and see" things <shrug> :-)

Noel Grandin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/e51a2917ab19156f5a5d2b9474a5a46a52e9e527

tdf#76324 speedup copy operation with lots of comments in calc

It will be available in 6.4.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.

Created attachment 154561 [details]
bt1

I updated master sources (b56a4eaf5eb856c36d3539dc7d2e6fa94b6551f8).

LO calls the AccessibleDocument ctr only when launching Calc.
Here's the bt from the ctr.

Created attachment 154562 [details]
bt when creating first comment

Here's the bt when creating the first comment

Created attachment 154563 [details]
Flamegraph

New Flamegraph following https://cgit.freedesktop.org/libreoffice/core/commit/?id=e51a2917ab19156f5a5d2b9474a5a46a52e9e527

Created attachment 154565 [details]
perf flamegraph (gen rendering)

I didn't find a way to prevent AccessibleDocument to be created so I launched LO with gen rendering which doesn't call accessible part it seems. (at least, I don't see AccessibleDocument in this Flamegraph)

So for anyone curious, I did a deeper look and the root cause is the lifecycle mess of ScPostIt/ScCaption where it needs to go to all the effort of creating an invisible caption object and inserting it into the drawing layer just to copy.

The proper fix is to rewrite that beast to only create caption objects WHEN THEY ARE ACTUALLY VISIBLE ON THE SCREEN. 
Which is harder than it sounds, because various bits of code want to have a real drawing layer available, and various bits of state are tied to the lifecycle of the associated ScDocument, which causes trouble when copying from one document to another and the first document goes away before the paste action occurs, and similar nasties.

in

Version: 6.3.1.0.0+
Build ID: 8b81a453b22611f25674f5e44ae411d78c2fcada
CPU threads: 4; OS: Linux 4.15; UI render: default; VCL: gtk3; 
Locale: ca-ES (ca_ES.UTF-8); UI-Language: en-US
Calc: threaded

I killed LibreOffice after 10 minutes doing the following

1. Open the attached document
2. Select Column A and B
3. Copy

in

Version: 6.4.0.0.alpha0+
Build ID: d5b7627a0e738c0866b819910153b96b611813f8
CPU threads: 4; OS: Linux 4.15; UI render: default; VCL: gtk3; 
Locale: ca-ES (ca_ES.UTF-8); UI-Language: en-US
Calc: threaded

LibreOffice responded after 7 minutes...

Noel Grandin committed a patch related to this issue.
It has been pushed to "libreoffice-6-3":

https://git.libreoffice.org/core/commit/63b420264558ab2e4123666e4080dc4ee1b5b55c

tdf#76324 speedup copy operation with lots of comments in calc

It will be available in 6.3.3.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.

*** Bug 125687 has been marked as a duplicate of this bug. ***

*** Bug 127758 has been marked as a duplicate of this bug. ***

It seems that something that was changed for this bug which resulted in a regression for some files with a large number of comments that worked fine in version 6.2.7 but with version 6.3.* becomes basically unusable after an initial save.  See bug 125619.

pls. also look at: 

https://bugs.documentfoundation.org/show_bug.cgi?id=125619#c8 and #c9

regarding the data structure for comments used in the file.

I've done what I can, no longer looking at this bug

reaction on mouseclicks in files with plenty comments is painfully slow again in 6.4 versions, 

it had been much better in 6.2, something 'regressed' from 

ver. 6.2 (tested 6.2.7.1 and 6.2.8.2), 

to 

ver. 6.4 (tested 6.4.0.1, 6.4.0.2, 6.4.0.3), 

pasting a checkerboard pattern of 4 cells (A1:B2) with different values and comments to a 100x100 area (A1:CV100) takes about 10.9 sec. in 6.2, slightly faster in 6.4, 

clicking a cell in that area to 'make it active' (setting the focus to that cell) is quick and 'responsive' in 6.2, while in 6.4 it takes about two seconds to react (sometimes you have to wait for the first save, save as, or autosave to see the problem) ... one can! use a spreadsheet with that slow reaction, but it's a pain in the ass, especially when you know that better performance is possible, 

as there has been better performance i conclude it's possible, and as there are versions with good and poor performance to compare i think it could be possible to find the point where it evolves from ... 

(it's a minefield of performance issues around comments, comments getting lost, moved to other cells, slow in load / save, paste, high cpu load on mousemoves, slow reaction to clicks and so on, most other reported bugs have been closed as duplicates of this one, thus i'm putting this info here despite it's not an exact match of the original post)

all problems have been much better - better problems, not better performance - in all tests i did with linux versions, thus up to now this bug is blocking my migration from win to lin  :-( 

i'd judge this part (pasting cells with comments) of the bug 'fixed' once the copy of a cell with comment to a full column takes less than three times the time as for a cell without comment. actual situation is: without comment: less than a seconds, with comments: crash ('not responding' for hours)

in reply to @Dan H from c#41: 
'Has anyone found _any_ currently-available combination of OS environment and LO build in which the bug _can't_ be reproduced?'
win7x64 with LO 6.2.7.1 and 6.2.8.2 x64 'unpar' (unparallelized computation, no 'threaded' and no 'openCL') are the best performing versions regarding comment slowdowns i have seen, they are not fast (as most operations are much faster on files without or with very few comments), but they are usable, while most other versions aren't once you have too many comments in a file ... 

(maybe with parallelized computation performance is somewhat better, but it has unsolved bugs thus isn't yet usable for productive environments)

both problems, slowdowns when pasting commented cells and delayed reaction on mouseclicks grow with the amount of commented cells in the file, 

reg. 

b.

*** Bug 131672 has been marked as a duplicate of this bug. ***

@Telesto and @Timur, 

summary: 

calc has 'comment issues', 

similar is affecting 'tracking notes' and comments in writer, 

slow reaction (click delays), slow load / save, unnecc. cpu load, high mem usage, high cpu load on mousemoves, immense mem usage after autosave / save as ... even system freezes and crashes, 

acc. to @Noel it's resulting from unproductive work done for the 'drawing layer', see c#85, 

maybe this patch / description is relevant for this issue: https://cgit.freedesktop.org/libreoffice/core/commit/?id=d464d505fbf6e53a38afdd3661d320fac8c760d6

and also 'non-linear interpretation of mousemoves' may step in? 

and all that's monstering around since 2014 ... with 21+ bugs and duplicates crying for help, 

i - assume - there might be one cardinal mistake being the source of all theese performance issues, e.g. wrong order when processing the comments or unnecessary recursions ... 

as @Julien wrote in c78: 'Sometimes, we notify n times because of a loop, instead of just notifying once (eg for redrawing) after the end of loop.'

best performing versions (balanced between performance and click delays): 6.2.7.1 and 6.2.8.2, do me a favour and install and try theese versions (parallel install), if you can confirm my findings try to find someone who is able to save and port the better 'click reaction' of the 6.2 versions to the actual versions, 

besides: for theese issues windows versions (6.2) perform much better than linux, 

until solved i'd suggest: 

someone cares for theese things, they are important, 

or to implement a warning message: 'using too many comments / notes / post-it's / captions / annotations or tracking notes may slow down your system'

to avoid ever recurring questions and irritated users. 

reg. 



b.

Eike: thought you might be interested in this one since it concerns Calc and there are several duplicates.
If needed, I can generate other Flamegraph graphs with different renderings.

i think i have isolated and visualized one of the 'most evil' aspects / effects of this bug in: 

https://bugs.documentfoundation.org/show_bug.cgi?id=125619#c23

may be it's worth to have a look there ...

*** Bug 138151 has been marked as a duplicate of this bug. ***

in

Version: 7.5.0.0.alpha0+ / LibreOffice Community
Build ID: e4d23c27288b99c3ed3cfa332ff308b31c01f97d
CPU threads: 4; OS: Linux 5.14; UI render: default; VCL: gtk3
Locale: ru-RU (ru_RU.UTF-8); UI: en-US
Calc: threaded Jumbo

I opened the example https://bugs.documentfoundation.org/attachment.cgi?id=96007 for 10 sec only

Cope/paste the cells with comments is just instantly

Someone please retest it in latest master build too please

I tested with attachment 96007 [details] and opening takes about 14 secs stopwatch time on my Win 10 virtual machine.

attachment 123072 [details] takes 3 secs to open.

attachment 125689 [details] takes 16 secs to open and 50 secs to save.

Let's finally close this as fixed and keep it that way. This report was already too long in 2018 and now it's high time to move on. There are other reports mentioned in the comments and See Also that likewise should be re-tested.

Version: 7.5.0.0.alpha0+ (x64) / LibreOffice Community
Build ID: 1638b4f78af70b7b97d0a081ed51390dd87bf1f9
CPU threads: 2; OS: Windows 10.0 Build 19044; UI render: Skia/Raster; VCL: win
Locale: fi-FI (fi_FI); UI: en-US
Calc: threaded Jumbo