Bug 76324 - CALC becomes very slow with 5000+ comments
Summary: CALC becomes very slow with 5000+ comments
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
4.2.2.1 release
Hardware: All All
: high major
Assignee: Not Assigned
URL:
Whiteboard: target:4.4.0 target:4.3.0.0.beta2 tar...
Keywords: dataLoss, perf
: 97698 105499 113599 114377 125545 (view as bug list)
Depends on:
Blocks: Calc-Comments
  Show dependency treegraph
 
Reported: 2014-03-18 15:10 UTC by abma
Modified: 2019-06-06 07:34 UTC (History)
15 users (show)

See Also:
Crash report or crash signature:


Attachments
calc document with many comments (~8k) (183.44 KB, application/vnd.oasis.opendocument.spreadsheet)
2014-03-18 15:22 UTC, abma
Details
A screenshot of the error message when saving document. (11.40 KB, image/png)
2014-05-05 08:49 UTC, a07cd040897db54e103c
Details
picture / profile of comment loading (413.83 KB, image/png)
2014-05-09 19:09 UTC, Michael Meeks
Details
file bug_comment.ods + bug_NOcomment.ods (572.56 KB, application/gzip)
2016-02-29 10:20 UTC, JCE
Details
new test file (94.81 KB, application/vnd.oasis.opendocument.spreadsheet)
2016-06-16 08:44 UTC, massimo.zaffaina
Details
A screenshot of performance in LO v5.1.4.2 (x64) (10.73 KB, image/png)
2016-08-10 15:44 UTC, a07cd040897db54e103c
Details
Spreadsheet with many comments (100.40 KB, application/vnd.oasis.opendocument.spreadsheet)
2017-01-11 21:58 UTC, Dan H
Details

Note You need to log in before you can comment on or make changes to this bug.
Description abma 2014-03-18 15:10:38 UTC
when using many comments in cells of a spreadsheet, calc becomes very slow:

- copy and paste cells with comments is very slow
- saving / loading such a document is very slow
- memory usage on save is enourmous


to reproduce, creat a document with ~5000 cells with comments:

insert 1, add comment to that cell, copy this cell, mark two cells, paste, copy four cells, paste, ...
Comment 1 abma 2014-03-18 15:22:38 UTC
Created attachment 96007 [details]
calc document with many comments (~8k)

calc document with many comments (~8k) showing performance problems in load/save, copy/paste.
Comment 2 a07cd040897db54e103c 2014-03-18 15:55:23 UTC
(In reply to comment #1)
> Created attachment 96007 [details]
> calc document with many comments (~8k)
> 
> calc document with many comments (~8k) showing performance problems in
> load/save, copy/paste.

I can definitely confirm this. Loading of files with 5000+ comments takes forever and even makes calc crash reproducably.
Comment 3 Michael Meeks 2014-04-11 19:07:00 UTC
Can you get a valgrind trace of the crash on load ? =)
Comment 4 abma 2014-04-14 12:39:30 UTC
with libreoffice 4.2.3.3 i get this error when i duplicate the comments in the example file when i try to save the file:

"Error saving the document test-1:
Write Error.
The file could not be written."


it doesn't crash for me, so i can't give a valgrind trace. are there some libreoffice specific docs how to make a valgrind trace on windows?
are these useful without debug symbols?
Comment 5 Kohei Yoshida 2014-04-30 02:27:49 UTC
Using the very recent master build, for me

(In reply to comment #0)

> - copy and paste cells with comments is very slow

It works quite snappy.  Copy and paste works quick, moving of cells etc also works fine.

> - saving / loading such a document is very slow

Yes, these are slow.

> - memory usage on save is enourmous

Around 800 MB VM (on Linux) which is quite reasonable.
Comment 6 a07cd040897db54e103c 2014-04-30 08:06:41 UTC
(In reply to comment #5)

> > - copy and paste cells with comments is very slow
> 
> It works quite snappy.  Copy and paste works quick, moving of cells etc also
> works fine.

Did you test with the attached document? If you have only one cell with one comment this one is indeed processed quickly. But if you have many *other* cells with comments copy/paste of this one cell is slow.
Comment 7 Kohei Yoshida 2014-04-30 13:59:54 UTC
(In reply to comment #6)
> (In reply to comment #5)
> 
> > > - copy and paste cells with comments is very slow
> > 
> > It works quite snappy.  Copy and paste works quick, moving of cells etc also
> > works fine.
> 
> Did you test with the attached document?

Yes. My comments are all based on the attached document.
Comment 8 Kohei Yoshida 2014-04-30 14:01:44 UTC
BTW, how does the load and save compare with the previous versions, say, 4.1.6?  Does anyone have any data on that?
Comment 9 a07cd040897db54e103c 2014-05-05 08:48:22 UTC
(In reply to comment #8)
> BTW, how does the load and save compare with the previous versions, say,
> 4.1.6?  Does anyone have any data on that?

Tested it with LO 4.1.6.2 on WinXP 32Bit. Behaviour is the same - huge memory consumption, very very long time to process, saving file failed with error:

Steps i have done:

1. use the attached document
2. select whole columns A and B
3. copy selection (lasts about 1 minute)
4. select whole columns C and D
5. paste (lasts about 2 minutes)
6. save document

Step 5 lasts about 30 minutes and fails with error (translated):

Error saving document test.
Writing problem.
File could not be written.

(see screenshot)
Comment 10 a07cd040897db54e103c 2014-05-05 08:49:28 UTC
Created attachment 98458 [details]
A screenshot of the error message when saving document.
Comment 11 Michael Meeks 2014-05-09 19:09:07 UTC
Created attachment 98773 [details]
picture / profile of comment loading
Comment 12 Michael Meeks 2014-05-09 19:19:05 UTC
Profile of load at:

http://users.freedesktop.org/~michael/callgrind-comment-load-34ae7b16d7e.txt.gz

Can be viewed in KCachegrind; help appreciated =)

163k calls to SvXMLTokenMap::SvXMLTokenMap is a bit odd - 6% of the load time.

Much of the grief seems to be the drawinglayer's fetish for measuring text repeatedly =) and SfxItemSet thrash.

I would assume that calc knows the size the comment boxes should be anyway (not that they are even visible so ... ;-)
Comment 13 Commit Notification 2014-05-30 12:06:48 UTC
Matuš Kukan committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=d2e0465d406b33139f3a97e1738020d6a7a6f6c3

fdo#76324: Use one static SvXMLTokenMap object, it's faster.



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 14 Matúš Kukan 2014-05-30 12:23:47 UTC
Profiles of copy and paste of the columns can be found at:
http://people.freedesktop.org/~mkukan/callgrind.calc-comments-copy.out.gz
http://people.freedesktop.org/~mkukan/callgrind.calc-comments-paste.out.gz
I was not patient enough to wait until the operation ends under valgrind, but should be useful anyway.
That pasting is really strange: it creates new document out of clipboard data, then pastes the comments into temporary document from there and then finally pastes the columns into real document you see - or quite possibly I am totally mistaken of course :-)
Hope this helps.
Comment 15 Commit Notification 2014-06-02 06:52:49 UTC
Matuš Kukan committed a patch related to this issue.
It has been pushed to "libreoffice-4-3":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=29162832feafca9259dc1589d10f3cfd43ea8126&h=libreoffice-4-3

fdo#76324: Use one static SvXMLTokenMap object, it's faster.


It will be available in LibreOffice 4.3.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 16 abma 2014-06-02 11:26:45 UTC
which version do i have to download to test the change?

is

http://dev-builds.libreoffice.org/daily/master/Win-x86@39/current/

correct?

if so, copy & paste 4k of comments is still incredible slow, it takes about 1 minute.

saving the document with 8k doesn't work, soffice aborts while saving with "Error saving the document test: Write Error. The file could not be written".
Comment 17 Matúš Kukan 2014-06-09 14:52:20 UTC
(In reply to comment #16)
> which version do i have to download to test the change?
> 
> is
> 
> http://dev-builds.libreoffice.org/daily/master/Win-x86@39/current/
> 
> correct?

Yes, I think so.

> if so, copy & paste 4k of comments is still incredible slow, it takes about
> 1 minute.
> 
> saving the document with 8k doesn't work, soffice aborts while saving with
> "Error saving the document test: Write Error. The file could not be written".

http://cgit.freedesktop.org/libreoffice/core/commit/?id=d2e0465d406b33139f3a97e1738020d6a7a6f6c3 was only about loading the document.
Comment 18 Commit Notification 2014-06-10 11:36:01 UTC
Matuš Kukan committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e7a3aacff7d28577dee371ed5b27317522db7b3b

fdo#76324: Make pasting a lot of cell notes faster by disabling broadcasting.



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 19 Commit Notification 2014-06-10 11:37:34 UTC
Matuš Kukan committed a patch related to this issue.
It has been pushed to "libreoffice-4-3":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=6188a7197aae39a21ef80d1b177f5be613dc9808&h=libreoffice-4-3

fdo#76324: Make pasting a lot of cell notes faster by disabling broadcasting.


It will be available in LibreOffice 4.3.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 20 Commit Notification 2014-06-11 12:42:54 UTC
Matuš Kukan committed a patch related to this issue.
It has been pushed to "libreoffice-4-2":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=90132a32f8bc2e27c097d79c0cc1ddd7cae35da6&h=libreoffice-4-2

fdo#76324: Make pasting a lot of cell notes faster by disabling broadcasting.


It will be available in LibreOffice 4.2.6.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 21 Matúš Kukan 2014-06-12 07:50:22 UTC
I believe pasting is much faster after e7a3aacff7d28577dee371ed5b27317522db7b3b,
so this bug is about saving the file now?

If anybody is interested, here is a profile:
people.freedesktop.org/~mkukan/callgrind.calc-comments-save.out.gz
There are probably more issues but one is with SfxItemSets.. that's not something for me.
Comment 22 Björn Michaelsen 2014-07-12 20:13:04 UTC
MABs should be priority highest.
Comment 23 tommy27 2014-11-19 04:31:49 UTC
(In reply to spam from comment #0)
> when using many comments in cells of a spreadsheet, calc becomes very slow:
> 
> - copy and paste cells with comments is very slow

I do not reproduce this anymore in 4.4.0.0 alpha2

> - saving / loading such a document is very slow
> - memory usage on save is enourmous

this is still reproducible and I get error on save.

moving this to mab4.3 since 4.2.x reached the end of life
Comment 24 abma 2015-07-16 15:03:15 UTC
- in 5.0.0.2 (x64) its still very slow when copy / pasting
- saving a document with a lot of comments still isn't possible: lo uses a lot of memory and then aborts with the same error message
Comment 25 David Fischer 2015-10-02 17:34:05 UTC
I was able to copy and paste the columns several times without any issue.  The application does hang for a second while it is pasting but I don't think this should be considered a bug given the amount of operations being done.  Potentially this issue has been resolved in the newer LO versions?

OS: Linux Mint 17.2
LO: 5.0.2.2
Comment 26 abma 2015-10-15 17:28:42 UTC
> I was able to copy and paste the columns several times without any issue.  The application does hang for a second while it is pasting but I don't think this should be considered a bug given the amount of operations being done.  Potentially this issue has been resolved in the newer LO versions?

imo it is better, but not fixed. still opening the example document takes ~20 seconds with LO: 5.0.2.2.

also CTRL+C takes a very long time, the same for CTRL+V.

very likely the complexity of these operations are still O(c^n) when it should be O(n).
Comment 27 JCE 2016-02-14 13:14:34 UTC
my test file bug-comment.ods 340 ko  with a lot of comments

Linux 3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) i686 GNU/Linux  XFCE 4.12
vendor_id	: AuthenticAMD
model name	: Mobile AMD Sempron(tm) Processor 3600+
memory 3 GO
LBO Version: 4.3.3.2 32bits
Build ID: 430m0(Build:2)
Locale 	 fr_FR		
			 		
Open file and save = First recording time		~15 minutes and 26s
Modify file and save second recording time		~ 7s

Delete all comments - time to delete 15 minutes and 15 s
Save the new file witout comments ~5s

Other test
----------
Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux  XFCE 4.12
vendor_id	: GenuineIntel
model name	: Intel(R) Core(TM) i3-4170 CPU @ 3.70GHz  
memory 4 GO
+ disk SSD	Kingston SSDNow V300 Series 120 Go
LBO Version: 5.0.4.2  (1:5.0.4~rc2-2~bpo8+1 jessie-backport)
Build ID: 00m0(Build:2)
Locale : fr-FR (fr_FR.UTF-8)
Open file and save = First recording time		~3 minutes and 25s
Modify file and save second recording time		~ 2s

Delete all comments - time to delete 3 minutes and 20 s
Save the new file witout comments ~2s

For informations
-----------------
Test on
my test file bug-comment.ods 340 ko  with a lot of comments
XP SP3-SSE memory 1Go
LBO Version	 4.4.5.2
Build ID	 a22f674fd25a3b6f45bdebf25400ed2adff0ff99
Locale 	 fr_FR		 		 		
Open file and save = First recording time		~23s	
Modify file and save second recording time		~23s

Vista memory 3Go	
vendor_id	: AuthenticAMD
model name	: Mobile AMD Sempron(tm) Processor 3600+
LBO Version: 5.0.3.2	
Build ID: e5f16313668ac592c1bfb310f4390624e3dbfb75	
Locale 	 fr_FR		 		 		
Open file and save = First recording time	~20s
Modify file and save second recording time	~10s
Comment 28 JCE 2016-02-29 10:20:11 UTC
Created attachment 123072 [details]
file bug_comment.ods + bug_NOcomment.ods

From  Comment 27 JCE 2016-02-14 13:14:34 UTC 
new test on Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux  XFCE 4.12
With 
with  LBO Version: 5.0.5.2

bug-comment.tar.gz
- file bug_comment.ods  with a lot of comments 
Open file and save = First recording time		~3 minutes and 25s
- file bug_NOcomment.ods 
Modify file and save second recording time		~ 2s
Comment 29 fmgtack+libreoffice 2016-05-22 17:50:25 UTC
Problem is still there in LibreOffice 5.1.2.2.
Comment 30 massimo.zaffaina 2016-06-16 08:44:15 UTC
Created attachment 125689 [details]
new test file

tested on a file with ~25k comments on these versions:
Version: 5.3.0.0.alpha0+
Build ID: a8bd44573b75d1399257d6f5d052611439607189
CPU Threads: 2; OS Version: Linux 4.1; UI Render: default; 
TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2016-06-13_23:46:49
Locale: it-IT (it_IT.UTF-8)
OS:openSUSE Leap 42.1 (x86_64)

and the results are:
opening: 47 s 
closing: 14 s
saving: 2.41 m

and on:
Versione: 5.2.0.0.beta2
Build ID: ae12e6f168ba39f137fc110174a37c482ce68fa4
Thread CPU: 2; Versione SO: Linux 4.1; Resa interfaccia: predefinito; 
Versione locale: it-IT (it_IT.UTF-8)
OS:openSUSE Leap 42.1 (x86_64)

and the results are:
opening: 47 s
closing: 15 s
saving: 2.47 m
Comment 31 a07cd040897db54e103c 2016-08-10 15:44:47 UTC
Created attachment 126728 [details]
A screenshot of performance in LO v5.1.4.2 (x64)

was taken *after* step 5 in comment #32 *before* saving
Comment 32 a07cd040897db54e103c 2016-08-10 15:46:21 UTC
retested with

Version: 5.1.4.2 (x64) (8GB of MEM, 6,5GB free)
Build-ID: f99d75f39f1c57ebdd7ffc5f42867c12031db97a
CPU-Threads: 2; BS-Version: Windows 6.1; UI-Render: Standard; 
Gebietsschema: de-DE (de_DE)

with the following results (slightly different from original report, minor improvements noticeable)

> 1. use the attached document
loading time: 15s (LO was already running); memory consumption seems normal

> 2. select whole columns A and B
time: instantly; memory consumption seems normal

> 3. copy selection (lasts about 1 minute)
time: 26s; memory consumption seems normal

> 4. select whole columns C and D
time: instantly; memory consumption seems normal

> 5. paste (lasts about 2 minutes)
time: 17s; memory consumption seems normal

(screenshot in attachment #126728 [details] was taken here)

> 6. save document
Crash *without* any error message after 6s while memory consumption raised rapidly to about 500MB (of 6,5 free GB)

Any changes were lost.
Comment 33 Peter 2016-10-24 17:13:25 UTC
I can confirm this bug and have some additional information.

Saving documents with many comments takes minutes or even hours, depending on complexity and hardware.

To be more precise: You press the save button and then LO seems to hang for a long time (other operations or even starting writer will not work, high CPU load on one core) and then it suddenly starts to save, so you see the progress bar on the bottom.

As stated already by someone else, it only takes long the first time the document is saved. If you save it later again, it is quick. Only after closing LO and opening the document again, the first save is slow again.

Interestingly, it seems to be relevant, which sheet is visible when you hit the save button. So as a workaround, I now switch to a sheet without comments, hit save and then it is quick. I can later save it on whatever sheet I am and it is still quick, as described above.

I thought it might have something to do with the thumbnail generation, so I tried to turn that off in the document properties, but it does not work, this option is automatically turned on after you close LO and open the document again. It might be another bug.

I use LibreOffice 5.1.4.2 that came with Ubuntu 16.04.
I am under the impression I did not have these problems before upgrading from Ubuntu 14.04 to 16.04.
Comment 34 Dan H 2017-01-11 21:58:46 UTC
Created attachment 130332 [details]
Spreadsheet with many comments

I'm experiencing similar symptoms in LO version 5.1.4.2 build ID 1:5.1.4-0ubuntu1, under Ubuntu 16.04 x64, with the attached file (which I think is somewhat smaller and has somewhat fewer comments than example files previously posted on this thread).  Like Peter, I believe the trouble started with a fairly recent (c. July 2016) upgrade.

Steps to reproduce problem:

1. Open the attached file in LO Calc.
2. Highlight (for example) cell E181.
3. Press ctrl-c to copy.
4. Highlight (for example) cell E182.
5. Press ctrl-v to paste.
6. Press ctrl-s to save.
7. CPU usage of soffice.bin jumps to near 100% and stays there, and LO hangs indefinitely (when I say "indefinitely" - on my last test, I gave up waiting and killed the soffice.bin process after about 25 minutes).

I can also confirm Peter's observations, that the situation is greatly improved if one creates a blank worksheet, and switches to that blank worksheet being the visible sheet, prior to saving, and that having done this once in a particular LO process, subsequent saves in the same LO process work well regardless of which sheet is visible, and that the problem then comes back again once LO is closed and re-opened.

Another oddity which also arose around July 2016 and may or may not be related: "save" no longer checks whether or not the document has been modified since it was last saved - it overwrites anyway.  This is the case irrespective of whether or not there are lots of cell comments (in fact, it applies to all LO components, not just Calc.)
Comment 35 Dan H 2017-01-12 00:18:50 UTC
Out of curiosity, I just installed Gnumeric 1.12.28, and tried the same series of steps, applied to the same file, in that package.  Saving completed nice and quickly.  Might there be mileage in a comparison of the comment-handling bits of the two codebases?
Comment 36 tommy27 2017-01-12 06:23:18 UTC
@Dan H
please DO NOT change version field in an improper way as you did.
it MUST indicate EARLIEST version the bug appeared (as indicated), not the latest.

so reverting it back to 4.2.2.1
Comment 37 Dan H 2017-01-12 17:38:00 UTC
My apologies.  I guess I should have read the documentation ;-).
Comment 38 Dan H 2017-01-20 12:04:26 UTC
Another datum that might be useful: I just retried my sequence of steps 1-6 from comment #34 in LO 5.1.4.2 (x64) build f99d75f39f1c57ebdd7ffc5f42867c12031db97a, under Windows 7 x64, and got a successful save within a few seconds.

However, with a larger version of the spreadsheet (i.e. with more non-blank cells and more comments -  about 11000 of each), the above Windows version of LO crashed on save with a "LibreOffice has stopped working" message.  Worse still, Peter's "make a blank worksheet visible" trick doesn't prevent the crash under Windows.  I can't post this larger example file because, er, LO crashes when I try to save it.
Comment 39 Buovjaga 2017-01-25 17:22:01 UTC
*** Bug 105499 has been marked as a duplicate of this bug. ***
Comment 40 Patrik 2017-01-25 18:07:22 UTC
As the person who posted the duplicate Bug 105499 I can confirm that it also happens with the test-document for this bug. 

Steps that, for me, are enough to reproduce the bug:
- open document (opens in reasonable time, but takes longer than the 180kB should need)
- hover over some comments, move mouse around
- just select the whole columns A+B by clicking on the columns Headers (grey A and B)

The last step freezes LO, CPU explodes. I have to kill LO.

I also played around with hardware acceleration, OpenGL, but it happens regardless of these settings. 

I also removed my ~/.config/libreoffice to test if the configuration is the problem - but no difference.
Comment 41 Dan H 2017-01-30 17:39:54 UTC
So, where are we at in terms of moving towards either a fix or a workaround for this? For example:

Has anyone found _any_ currently-available combination of OS environment and LO build in which the bug _can't_ be reproduced?  Or even better, has anyone done a bisect sequence that has isolated which commit/revert (either to the LO git tree or to a library's git tree) introduces/removes the bug?

Does anyone know which file in the LO source code tree contains the code that is being executed when the hang happens (last August, I tried searching the source tree myself, but I couldn't find the comment-handling code at all).

Thanks.
Comment 42 abma 2017-01-30 18:34:13 UTC
> Does anyone know which file in the LO source code tree contains the code that is being executed when the hang happens (last August, I tried searching the source tree myself, but I couldn't find the comment-handling code at all).

see the screenshot in https://bugs.documentfoundation.org/show_bug.cgi?id=76324#c11

Profiling.
Comment 43 Dan H 2017-03-02 18:53:12 UTC
Sorry it's taken me so long to say "thank you" for that last piece of information, but, er, thank you.

A few observations:

- If the heavy CPU usage really does relate to measuring text in order to produce a comment box of the appropriate size, that provides a possible explanation of why Gnumeric and Excel are able to perform faster, since they both appear to leave the appropriate sizing of the comment boxes for the user to do manually.

- Shortly after that trace was posted, a patch was posted on here that was supposed to mitigate the particular issue revealed in the trace.  Do any (or all) currently widely-available builds of LO contain that patch?

- Since both Peter and I have vague memories that the performance was better before we upgraded from Ubuntu 14.04 to 16.04, my next move is going to be to downgrade back to 14.04 (the LO build ID in which is 1:4.2.8-0ubuntu5).  I'll let y'all know whether it helps.
Comment 44 Dan H 2017-03-03 19:42:15 UTC
Results:

I repeated the procedure described in comment #34 under Ubuntu 14.04 (LO 4.2.8.2
Build ID: 420m0(Build:2)).  The save process at step 7 was completed without trouble in 20 seconds.

Could it be that Matúš' patch successfully solved the problem, and then someone  (partially) reverted it and reintroduced (the "save" part of) the bug before tommy27 got a chance to undertake the test described in the penultimate paragraph of comment #23?
Comment 45 tommy27 2017-03-04 08:29:07 UTC
I don't have time to retest now.

basically you are saying that the bug is not present in 4.2.8, right?

what about 5.2.5 and 5.3.0? is the issue still present or not?
can we label this as RESOLVED FIXED?
Comment 46 Dan H 2017-03-04 09:57:40 UTC
I think it's correct to say that the bug is not present in 4.2.8, _in the enivronment of Ubuntu 14.04_.

However, I don't think we can mark it as RESOLVED FIXED, because, from my own tests:

- The bug _is_ present in 5.1.4.2, in the environment of Ubuntu 16.04.
- The bug _is_ present in 5.1.4.2, in the environment of Windows 7.

and from your (tommy27's) tests:

- The bug _is_ present in 4.4.0.0 (you didn't mention what OS)

I now have another (puzzling) datum to add:

The bug is not present in 5.0.6.2, in the environment of Scientific Linux 7.3.

(Incidentally, I used exactly the same contents of the ~/.libreoffice directory in all three of the Linux environments I tested.  However, I also ran a separate 5.1.4.2/Ubuntu 16.04 test with a ~/.libreoffice tree freshly created by starting LO with no ~/.libreoffice present, and got the same result, i.e. the bug _was_ present.)
Comment 47 tommy27 2017-03-04 10:30:53 UTC
I don't remember the exact release dates of 4.2.8 and 5.2.4 but consider that 4.2.8 is a very late release of the 4.2.x branch and maybe a committ that was done in 5.2.5 could have been backported in 4.2.8 so that's why you don't see it in 5.2.4.

you should test 5.2.5 or 5.3.0 so have a clearer picture of the bug status
Comment 48 Dan H 2017-03-06 23:29:09 UTC
Well, it's good news on that front.  I installed (on Scientific Linux 7.3) LO 5.3.0.3-3, from the binary RPMs on libreoffice.org, and the bug was absent (by the same test set out in comment #34).

Anyone fancy trying the equivalent on Ubuntu, Windows or any other OS?
Comment 49 tommy27 2017-03-07 06:41:45 UTC
let's label this one as WORKSFORME
thanks for testing it in 5.3.x
Comment 50 abma 2017-03-07 11:13:00 UTC
i tested "calc document with many comments (~8k)" in 5.3.0.3 x64 / windows 7 x64:

copy & paste takes ~1 Minute and calc hangs in this time. when saving the document calc crashes:

http://crashreport.libreoffice.org/stats/crash_details/f4d601b5-bb77-48c7-ba26-75e131c1b84a

no, this is definitly not fixed.
Comment 52 abma 2017-03-07 11:30:10 UTC
even opening "calc document with many comments (~8k)" and then directly saving it leads to a crash:

crashreport.libreoffice.org/stats/crash_details/9bdefcc2-dab5-4c29-8c51-0f3d287ae732

crashreport.libreoffice.org/stats/crash_details/de47c3e7-0f57-4e44-a24d-38a05b3c7ac0

sorry, enough bug-report-spam for now ;-)
Comment 53 Xisco Faulí 2017-03-07 12:33:11 UTC
let's keep this bug closed as RESOLVED FIXED and use bug 105888 for the crash...
Comment 54 Xisco Faulí 2017-03-07 12:53:30 UTC
*** Bug 97698 has been marked as a duplicate of this bug. ***
Comment 55 abma 2017-03-07 16:17:03 UTC
calc 5.3.0.3 is still very slow when working with the "calc document with many comments (~8k)" document.

it is faster then when the bug report was created, yes. IMHO its still way to slow as seeing "The Application is not responding" is a bug.

-> The initial reported problem still exists but became better:
1. opening the document takes ~30s
2. copying all comments takes ~15s
3. pasting all comments takes ~15s

when doing the same in a document with numbers only in the cells all of the operations are done "instantly".

the resulting document with comments is ~10 times larger in comparison to the numbers only version.


Not reopened as its out of my responsibility here. i hope such hangs are seen as bug, so please reopen when it applies.
Comment 56 Patrik 2017-06-13 07:38:34 UTC
I just opened a document with many (100 < x <1000) comments. 

If I copy a complete worksheet (move or copy) or rearrange the worksheet simply to another location at the bottom tab Calc crashes. 

Using 5.3.3.2 on Linux 64. Hardware Acceleration is disabled and it doesn't happen if I open a new worksheet or use another, less heavily commented document - therefore I think it is related to the comment problem.
Comment 57 Jan 2017-07-28 10:06:02 UTC
Another way to experience the slowness is to delete a column. I'm using 5.3.4.2, Linux 64.
Comment 58 perie_gut 2017-12-10 03:38:18 UTC

I have tried to copy 5 cells with comments to 8000 lines and libreoffice calc crashed. 

Product Details  
Version: 5.4.4.1 (x64)
Build ID: da790616461e15a10c95a80eb8ef8ee7b726c114
CPU threads: 4; OS: Windows 6.19; UI render: default; 
Locale: en-US (en_US); Calc: CL
Comment 59 Dan H 2017-12-10 18:44:02 UTC
Just a quick note that I'm continuing to test for this bug every time my machine pulls any package upgrades from the Scientific Linux repositories.  If/when the bug ever occurs on SL, I'll post a list of package upgrades that have happened since the last-known-good state, which I hope will help us narrow down what's causing the trouble.
Comment 60 Buovjaga 2017-12-17 16:58:45 UTC
*** Bug 114377 has been marked as a duplicate of this bug. ***
Comment 61 Xavier Van Wijmeersch 2018-02-16 10:43:28 UTC
Version: 6.0.1.1
Build ID: SlackBuild for 6.0.1 by Eric Hameleers
CPU threads: 8; OS: Linux 4.14; UI render: default; VCL: kde4; 
Locale: nl-BE (en_US.UTF-8); Calc: group

deleting a column take's 10seconds
copy/past a column take's 5 seconds
Comment 62 Dan H 2018-07-04 12:33:21 UTC
Sorry to be the bearer of bad news, but this bug, or some variant of it, is still with us, and often getting worse on upgrade.  Specifically,

- on upgrading from Scientific Linux 7.4 (which contains LibreOffice 5.0.6.2) to Scientific Linux 7.5 (which contains LibreOffice 5.3.6.1), the time taken to save the "Many_comments_sample" spreadsheet posted as attachment #130332 [details] upthread increases from 25 seconds to 52 seconds (on the same hardware).

- I've also got a somewhat larger spreadsheet with about 16000 comments (which I can't post here because it contains some confidential data); in LibreOffice 5.0.6.2 under Scientific Linux 7.4, saving this spreadsheet takes about 30 seconds; whereas in LibreOffice 6.0.5.2 under Fedora 28 (and on faster hardware), saving this spreadsheet takes 3 minutes 50 seconds.

(In both cases, these results were obtained as per the test procedure I set out in comment #34 upthread.)
Comment 63 Dan H 2018-07-04 12:37:12 UTC
An oddity that might be of some diagnostic relevance: in the Fedora 28 version of LibreOffice, the "undo" operation for the single-cell copy and paste mentioned in the procedure at comment #34 takes much longer than the copy and paste operation itself.
Comment 64 Xavier Van Wijmeersch 2018-07-04 15:48:44 UTC
opening the doc take's 4 seconds, saving 7 seconds. copy/past is fast
no high cpu load or high memory load
did follow description from comment 34 with attachment 130332 [details]
OS is slackware64current

Version: 5.2.7.2
Build ID: 2b7f1e640c46ceb28adf43ee075a6e8b8439ed10
CPU Threads: 8; OS Version: Linux 4.14; UI Render: default; VCL: kde4; 
Locale: nl-BE (en_US.UTF-8); Calc: group

Version: 5.4.8.0.0+
Build ID: cc68977f1be22ac0f4a15eb37e05ccba13a7a554
CPU threads: 8; OS: Linux 4.14; UI render: default; VCL: kde4; 
TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:libreoffice-5-4, Time: 2018-05-12_11:32:19
Locale: nl-BE (en_US.UTF-8); Calc: group
Comment 65 Buovjaga 2018-11-24 17:39:24 UTC
Let's keep this as fixed where it belongs. New issues should get new reports. I opened a report for a hang with GTK3: bug 121698
Comment 66 a07cd040897db54e103c 2018-12-21 09:07:23 UTC
For me the bug is not resolved at all. Only the Linux version seems to be fixed. In Windows only slight improvements were noticable. The main problem seems to be already present. See below.

@Buovjaga: this is *not* a "new issue", so reopened.

Tested with Debian buster x64:

Version: 6.1.3.2
Build-ID: 1:6.1.3-2
CPU-Threads: 2; BS: Linux 4.18; UI-Render: Standard; VCL: x11; 
Gebietsschema: de-DE (de_DE.UTF-8); Calc: group threaded

Result:
- loading/saving copying/pasting of/in "calc document with many comments (~8k)" is substantially faster, so i would not longer consider it to be a problem
- loading/saving copying/pasting of/in "calc document with many comments (~8k)" is still slower than it would be with "normal" cell contents instead of comments
- no crashes
- no huge memory useage
- file saved successfully
- file saved in fair time

Tested with Windows Server 2008 R2 x64:

Version: 6.0.6.2 (x64)
Build-ID: 0c292870b25a325b5ed35f6b45599d2ea4458e77
CPU-Threads: 16; BS: Windows 6.1; UI-Render: Standard; 
Gebietsschema: de-DE (de_DE); Calc: group

Result:
- loading: 35s
- copying columns A+B: 85s
- pasting copied columns to C+D: 50s, mem useage grew to 280 MB
- saving: 90s, mem useage grew to 1.200 MB
- closing! calc: 150s (!)

During all the steps above, only one (of 16) cpu cores was used with a load of 100%.
Comment 67 Buovjaga 2018-12-21 16:37:15 UTC
Fine, let's keep as NEW. It's just that this has nearly 70 comments, has received fixing commits both in this report and other reports. The report is starting to get difficult to approach.

I do confirm the slowness on Windows 10 with latest daily build.
Comment 68 Dan H 2019-01-13 19:15:33 UTC
Here's an interesting datum: in LibreOffice 6.1.4.2-1.fc29 under Fedora 29, the save-after-small-edit test, on one of my production spreadsheets with many thousands of comments, takes about 80 seconds if both the GTK2 and GTK3 integration packages are installed, but takes only about 25 seconds if only the GTK2 integration package is installed.
Comment 69 b. 2019-05-29 04:48:57 UTC
dear guys, 

this is! a bug, 

it is a big! bug, 

and it's in the program for over 5 years now, still unresolved, 

ok, ranting doesn't help, but points like this are  not a good showpiece for a program being 'actively developed' ... :-( 

i think i stepped in a variant of (or something left while plastering on) this bug, i observed performance issues and tried to bring it to the point, it took me hours and days - see there: https://bugs.documentfoundation.org/show_bug.cgi?id=125545 and there: https://bugs.documentfoundation.org/show_bug.cgi?id=124692 till i came to the point it might be 'comment related'. 

how many people shall spend how many time and collect how many frustration till somebody takes this point serious and does something against it? 

most people won't do any analysis, when they use comments they will start with a few, no problem, use some more, no problem, and on continuation they will observe firstly little, then bigger and in the end major performance problems. 

but nobody tells them the reason, and they have no hint on the source, thus they will helplessly search around (as i did), be frustrated working with calc, and either switch to another product or change to other work. 

that's not a good thing! 

my two cents: 

- the issue is partially covered by enhancements in the program and faster hardware and OS's, but it's still in and 'working' (try changing the pattern in the first problem file to two cells with different content and different comments, and fill some area like a chessboard, thus breaking 'shared formulas' and similarity compression, you'll see massive performance impacts also in very fast systems, (90 seconds of 'not responding' and 'fan requesting cpu load' for a simple mouseclick in a sheet with 32k filled cells and 16k comments is not! 'the spreadsheet you ever wanted').

- if the problem comes from the calculation of the size of textboxes ??? - mentioned somewhere above - *switch it off!!!* while the comments are hidden. imho the normal use of comments is to have additional info in it which isn't needed all the time and shouldn't eat the screen, thus invisible and shown on mouseover. for this use its absolutely sufficient to calculate one! textbox for the cell actually showing it's comment. 

- in some situations i observed some flickering of comment boxes and 'resistance against the cursor and editing' of text in comments, may be there is something wrong, may be with new graphics features, may be ... 

- with the stuff provided in this thread it schould be a short and direct way to the source of the problem in the code, simple files, trivial data, simple 'functioning', there is little chance for the bug hiding against profiling, 

any help appreciated!

would / could somebody with sufficient rights set the importance to major please?
Comment 70 b. 2019-05-30 06:22:05 UTC
Severity is high/major per https://wiki.documentfoundation.org/images/0/06/Prioritizing_Bugs_Flowchart.jpg,

(first question 'no', second question 'yes'), 

imho raising of severity and resolving isn't only neccessary / good for affected users, but also a great relief for QA team (to get rid of the work of managing duplicates) ;-)
Comment 71 Xisco Faulí 2019-05-30 10:34:48 UTC
*** Bug 125545 has been marked as a duplicate of this bug. ***
Comment 72 b. 2019-05-30 21:53:34 UTC
pls. read comment https://bugs.documentfoundation.org/show_bug.cgi?id=125545#c10
Comment 73 b. 2019-06-01 06:09:19 UTC
sorry for insisting, and sorry for bad news, but there is still a bug, and it's bad. 

- there is also something critical in the handling of the bug here, it 'blocks it from visibility' - 

second problem first, new announcements about variants or 'things left' of this? bug after changes in versions are thrown out as 'duplicate', while this bug is kept somewhere between 'low prio', no-repo, delayed, but there is almost one 'property' of the actual situation - not yet mentioned in this thread - that hides the problem from being visible to testers, thus collecting norepo-comments: 

the -problem- seems off in plenty situations after a fresh load of affected files, it appears / re-appears after the first edit in the sheet and subsequent autosave. 

the 'issue left', or the problem for me: high cpu load on working in sheets / files with many comments, 

all sorts of hangs, crashes, 'not repsonding', time and cpu consuming load and save processes, delays in reaction to mouseclicks, and so on, 

easy test to check: - in windows - 
- produce a file with many cells with data, 
- hover around with the mouse, 
- no cpu load, check with the 'power'? tab (german: Leistung) in the task manager, 
- make a little change, wait for autosave (you can set it to a short time for this test, tools - options - load/save - general), 
- hover around with the mouse, 
- nopro, 
- add comments to plenty cells (just copy a not trivial pattern of cells with comments to a bigger area), 
- hover around, 
- no load, 
- wait for autosave, 
- hover around 
- significant cpu load. 

(set back autosave to a normal value after the test, too many saves might kill your ssd disk)

(it's not neccessary to click into the cells, just moving the mouse / pointer (touchpad in my case), (which changes the focus for which cell the comment is to be evaluated) is enough to produce load)

(it doesn't help that the load is 'only' 12 to 15 percent on powerful systems, it's the difference from being 0 to 1 percent before autosave what says 'problem') 

(also 'easements' like saving in 'fods' format (it has significantly mitigated the problem in some situations) or 'restart in safe mode' (no difference on my tests) do not bring away the problem) 

is this 'pinpointing' helpful to identify the source of the problem? 

shall i open a new bug for this special scenario and is there hope it will not be closed as duplicate? 

reg. 



b.
Comment 74 Timur 2019-06-06 07:33:45 UTC
*** Bug 113599 has been marked as a duplicate of this bug. ***