Not NEW due to   
 <https://wiki.documentfoundation.org/QA/BugTriage#Set_Status_.26_Prioritize>
I will do some more tests

@Florian Reisinger
V. 3.6.5.2 has been mentioned by whom?
What OS have been tested?
What specific characteristics of reporter's documents are under suspect? 120 pages is not a very large documents.
How have the documents been created?
We should have original reporter from ML here.

Document contains 2500 ODF Errors, most of them 'unexpected attribute "ls-langcode"'

@Giorgio Migliaccio: Please comment, why it is not ODF conform....
If there is no 100% ODF compitable ODF testfile I am going to close it as NOTOURBUG

@Rainer: I did the testing both on Win 7 (BTW It is in the description)

In my case: Win7 x64

the original reporter is in CC
It is just about the loading speed
I don't know how this is created --> If creation is not done by us --> NOTOURBUG, is it?

So, this should be all information I have...

PS: Could you please link to the ODF validator used...

Currently my suspect is that it might have to do with the lots of lots of Comments in the document. AOOo 3.4.1 seems a little faster for FILEOPEN, but has similar problems, I think that never worked better.
Especially to delete Comments is very difficult

*** Bug 60419 has been marked as a duplicate of this bug. ***

(In reply to comment #3)
> @Giorgio Migliaccio: Please comment, why it is not ODF conform....
> If there is no 100% ODF compitable ODF testfile I am going to close it as
> NOTOURBUG
> 
> @Rainer: I did the testing both on Win 7 (BTW It is in the description)
> 
> In my case: Win7 x64
> 
> the original reporter is in CC
> It is just about the loading speed
> I don't know how this is created --> If creation is not done by us -->
> NOTOURBUG, is it?
> 
> So, this should be all information I have...
> 
> PS: Could you please link to the ODF validator used...

We've created the documents with LibreOffice 3.5.4 but, for integrating it into our own solution, we needed to extend the file with some of our own attributes which will contain a guid/key. 
So those attributes on your elements are the only thing that's added by us and is not according specs, but shouldn't be the reason why these documents are behaving as they are in LibreOffice/OpenOffice.org.
The tests were run on Windows 7 32 and 64-bit and 4 GB of RAM with LibreOffice versions 3.4.5, 3.5.4, 3.5.5, 3.6.x and 4.0 beta. All gave the same results, were 3.4.5 gave slightly better results with our full-blown resolved document, containing all the sub-documents.

Validator: <http://odf-validator2.rhcloud.com/odf-validator2/>

Documents with thousands of comments are not a usual application for an Office Suite, so I reduce priority.


@GiorgioMigliaccio:
Please do not cite and do not use your mail client for Comments.
Is there any possibility for you to check whether the comments cause the problems?

Hm. The document contains more than 200000 Frames for 42000 words - this is a VRY unusual document.

@GiorgioMigliaccio:
I wonder why I get offered to delete comments what seem to be related to a PW protected Area, what might be a REAL bug.
Can you please check that and submit a different Bug (and add me to CC) if my suspect concerning Comment deletion is correct?

@Rainer : this document indeed contains a lot of frames. We mark ranges of text which are conditional or need to be looped with a start 'tag' and end 'tag', which are put physically in the document using frames, and also variables are put there using frames which get a specific attribute and guid so we can link it to our own objectmodel which specifies some additional info for these ranges. We've tried and investigated all LibreOffice (or OpenOffice.org) had to offer us at the time we started and this came out being the most flexible solution.
Trying to delete a comment for user x just crashes LibreOffice 4.0.0.2.
I'll try to remove them from the xml, manually, and see if this improves the document speed and memory usage in LO.

(In reply to comment #9)
That would be great, we will have to separate the task into bite-sized different bugs. I believe most promising would be to create several documents based on this sample here:
One only with lots of frames
One only with lots of comments
One only with lots of ...

For each of those troublemakers a separate Bug (with "Block this one) should be submitted. So the possible roots can be examined in separate steps, what would increase chance for a quick fix.

Ok, found it.
The particular document(s) were containing hundres of the following lines:
<form:form form:name="Form" form:apply-filter="true" form:command-type="table" form:control-implementation="ooo:com.sun.star.form.component.Form" office:target-frame="" xlink:href="" xlink:type="simple">
               <form:properties>
                  <form:property form:property-name="PropertyChangeNotificationEnabled" office:value-type="boolean" office:boolean-value="true"/>
               </form:properties>
</form:form>


Removing these from the content.xml sped up the document again to open in 5 secs and use only some 50 MB of memory!

I also tried removing the frames and the comments from the document, and this didn't change much, even though there were hundreds of them in the document, so, this seems implemented very well! :-)

But the main culprit seems to be the form:form tag for some reason, I don't even know how or when it got in to the documents.
The strange thing is that the same entry is occurring hundreds of times in the same document.

Created attachment 74383 [details]
Without Comment tags

Created attachment 74384 [details]
Without Frame tags

Created attachment 74385 [details]
Solution - Without Form tags

@GiorgioMigliaccio (In reply to comment #11)
Thank you for the additional sample documents, I will try next week to narrow down influence to the visible worse performance compared to LibO 3.4.5.

The most promising way to get a more visible improvement for your root problem with the "form form form" tags might be that you do some own tests how they get into the documents. And if there is a suspect that it's a LibO bug you should submit an additional new bug with a detailed description how a developer can reproduce the bug with very simple test equipment. Please feel free to add me to CC.

I believe this bug is a duplicate of bug 61558.  I've found that toggling the "Hidden Paragraphs" option under the view menu will cause it to start responding properly while selecting "Update All" under the tools menu will cause an extreme lag.

Thanks David.
Will try that.

We still didn't manage to find out why or when these form:form tags got into the document, which caused the incredibly slowdown and memory consumption.

But you mentioned 'Hidden paragraphs', and I know some of our document authors were playing with that feature, it might as well be the root cause of the problem.
So, perhaps, once you toggle that feature on/off, you get the form:form tags.

I will test this and give feedback.

I don't think it actually has anything to do with hidden paragraphs since I have several large documents that act the same way and AFAIK they don't have any hidden paragraphs.

I can open attachment 74336 [details] in about half minute using Pentium 1.3Ghz & 2Gb ram (LO 4.0.4.2 - Win7 32bit). 
Confirm crash when closing that file (perhaps freezing because when open LO again not opening document recovery).

Still same slowness all around. Closing is just slow, doesn't crash.

Win 7 Pro 64-bit Version: 5.1.0.0.alpha1+
Build ID: 01a189abcd9a4ca472a74b3b2c000c9338fc2c91
TinderBox: Win-x86@39, Branch:master, Time: 2015-06-14_07:46:28
Locale: fi-FI (fi_FI)

*** Bug 95244 has been marked as a duplicate of this bug. ***

This problem occurs with even something like only a thousand comments.

From observation, it's clear that at least one of the algorithms involved with handling comments is O(N^2) or worse.

Editing the manuscript of a novel for publication is not that unusual a use case, and the use of comments for the editing and revision process by the author and an external editor is the norm.  Even deleting comments that have been addressed, novels normally go through several rounds of revision, so the number of comments stays reasonably high for quite some time.

I had to break the file for my novel (the version with comments) into two halves for LO to be usable to work on it.

On a separate, unsplit version with o comments, I revised and added comments as needed.  Performance stayed fine until I reached about the 50% mark, when a slight effect on performance was detectable.  The degradation in performance rose slowly, then faster.  It was noticeable by the 65% - 70% mark; now at the 77% mark, with something like a thousand comments now added, performance is becoming a serious problem.  Just typing text into the body of the docmuent now lags, the characters appearing only at the rate of one or two per second.

Splitting the file in two is going to be necessary, I suspect, which is unfortunate, as it is not only added work but LO appears to sometimes lose the formatting of italics in some paragraphs, meaning that the etxt cannot be trusted and needs to be painstakingly compared when I paste the documents together to make the final single file for generating the epub and other versions.

Would a simple (doubling) growable array of references to comments be a candidate data structure to handle comments?  The O(N^2) algorithm is the performance killer.

I'm seriously considering the idea of buying a copy of MS Office to run under Wine or Crossover because of this problem, especially as it's been prioritised so low. :-(

Well, the severity should be higher, it's true.

It's not at the unusable point quite yet.  I've pasted some of my reply comments inside the editor's comments, and deleted mine, to get reduce N. I may resort to inserting comments into the body of the document. :-(

Incidentally, the combination of the slow performance, plus LO's tendency to scroll to a different point within the comment when you click the cursor in it, means that on dozens of occasions today, I've modified the wrong part of the comment (because the insertion point, or the etxt I've sweep-selected, isn't what LO actually used because it repositioned the comment text before applying my edits).  So I then had to undo and reposition the comment text and then re-paste or retype my edit to the comment.  I.e. this behaviour induces errors, too.

That combines especially badly with the (separately reported bug) where LO scrolls back to the previous comment if the insertion point was in an earlier comment (possibly pages back), before applying your edit.

I'm now using LO 5.1.0.3.  The issue is not improved.  Today I also noticed a new problem: in the document with many comments, as I address each, I delete the comment (so that performance will gradually improve).

For comments that require further discussion, I copy the text, with comment included, into the new version of the document and then reply to the comment.

Currently, it takes 10-20 seconds for the many-comment document to respond when I click on the comment.  I have found that if I click, wait, then hover or click on the drop-down comment-actions button, at some stage the "Activate this button..." balloon help will pop up, and then if I click, on the arrow, the menu will appear within a second or so.

BUT, for comments for which the associated text is still selected, when I click on the "Delete" action in the comment menu, often now, instead of deleting the comment, the selected text that the comment refers to is deleted!  This happens about 50% of the time.

The first three times it happened, I convinced myself I must have hit the Delete key instead of selecting the Delete Comment menu item: but that is not the case.  The Delete Comment *sometimes* deletes the associated text, if it is selected.

You can Undo, and the select Delete Comment again, and the comment will be reliably deleted on this second attempt.  However, another oddity I noticed is that when the comment has been successfully deleted, if I Undo that operation, while the comment reappears, it points to just the very end of the span of text that it was previously associated with.

Aron Budea committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=f91674bd4b5022a63dc5e6a89fe9a1b832d96798

tdf#60418: improve perf of opening/closing odts with form tags

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.

(In reply to Rainer Bielefeld Retired from comment #7)
> Validator: <http://odf-validator2.rhcloud.com/odf-validator2/>
> 
> Documents with thousands of comments are not a usual application for an
> Office Suite, so I reduce priority.
> 
> 
> @GiorgioMigliaccio:
> Please do not cite and do not use your mail client for Comments.
> Is there any possibility for you to check whether the comments cause the
> problems?

I disagree: currently, Microsoft Word documents are the generally preferred method of reviewing and critiquing manuscripts, between authors and editors. It's wonderful that LO can export .docx format, and handles the comments.  But it is quite common to have 2-10 comments on most pages of a manuscript, and if the manuscript has 400 pages, having 2000 comments would not be unusual at all.

And there is clearly an N^2 algorithm in there, from my observations: as I delete comments, the speed improves.

And with around 2000 comments, to avoid 10+ second waits for every interaction, by splitting the file into a 1st half and a 2nd half, performance went from unusable to good.

I look forward to trying out the new version: I hope it addresses not just the problem found related to forms, but also to the number of comments!

(In reply to Luke Kendall from comment #29)
> I look forward to trying out the new version: I hope it addresses not just
> the problem found related to forms, but also to the number of comments!

I'm just a random person looking at random issues trying to understand the codebase better, who found some quick wins, so don't get your hopes up. I checked only with the 3rd attachment (the 1st didn't even open), and only opening and closing times. I'll probably try to look into this some more in the future, but can't promise anything.

(In reply to Aron Budea from comment #30)
> (In reply to Luke Kendall from comment #29)
> > I look forward to trying out the new version: I hope it addresses not just
> > the problem found related to forms, but also to the number of comments!
> 
> I'm just a random person looking at random issues trying to understand the
> codebase better, who found some quick wins, so don't get your hopes up. I
> checked only with the 3rd attachment (the 1st didn't even open), and only
> opening and closing times. I'll probably try to look into this some more in
> the future, but can't promise anything.

Okay, thanks, Aron.  I'll keep my fingers crossed: good luck!

I've upgraded to LO 5.2.5.1 now, and also received the Word .docx for the 2nd half of my novel, with his comments inserted.

The 2nd half was longer than the 1st - about 87k words.

When I tried to use it, in LO, LO was basically unresponsive.  I managed to type a three words, and (after a minute or two) undo that change.  And after that, I couldn't get any of the menus to respond, in a minute of clicking and waiting.

At that point I gave up, because waiting over a minute for every UI action was clearly impractical.

So I divided that file into two (making a file for the 3rd quarter, and one for the 4th quarter).  Once I'd done that, performance was (barely) acceptable: e.g. I could click on the drop-down arrow on a comment and after a few clicks, and waiting about 6 seconds, the comment menu would pop up.

It's quite clear that MS Office does not use an O(N^2) algorithm for its handling of comments, and LO does.

It would be great if someone could locate the section of code which is using multiple passes down a linear array or list, and replace it with something that's not O(N^2).

At least I have a workaround, since I can just work on my main document (which has at most a couple of hundred comments) and correct that, while referring to my editor's commented version (halved or halved again until it becomes responsive in LO), and delete each comment as I deal with it, so that performance quadratically improves as the no. of comments drop.

LO as it stands however is not currently a suitable tool for someone working as an editor who needs to add comments to a novel.  For that, as LO stands, you need the Microsoft product, unfortunately.

Dear all, I opened bug report 113599 that is similar (but not so similar to be a duplicate IMHO). thanks

There should be some improvement here, for Master & LibO 6.1.1 (based on bug 119130)

(In reply to Telesto from comment #39)
> There should be some improvement here, for Master & LibO 6.1.1 (based on bug
> 119130)

Tested with attachment 74336 [details].

It is true that with 6.1.0, modifying and saving causes a hang. With master I was able to save.
The heavy CPU use when moving inside the document remains.
If you stay on the first page after opening, the CPU runs at 100% constantly. After you scroll away, it calms down. Yet every time you scroll, it peaks.

Arch Linux 64-bit
Version: 6.2.0.0.alpha0+
Build ID: 401cba4c20fbc930f034168872642428d7459218
CPU threads: 8; OS: Linux 4.18; UI render: default; VCL: gtk3_kde5; 
Locale: fi-FI (fi_FI.UTF-8); Calc: threaded
Built on August 20th 2018

Arch Linux 64-bit
Version: 6.1.0.3
Build ID: 6.1.0-2
CPU threads: 8; OS: Linux 4.18; UI render: default; VCL: gtk3_kde5; 
Locale: fi-FI (fi_FI.UTF-8); Calc: group threaded

Created attachment 145115 [details]
Callgrind output from master (fileopen)

Opening attachment 74336 [details]

Arch Linux 64-bit
Version: 6.2.0.0.alpha0+
Build ID: 0ffa7a733d834647dfd59b864c52a015028822b6
CPU threads: 8; OS: Linux 4.18; UI render: default; VCL: gtk3_kde5; 
Locale: fi-FI (fi_FI.UTF-8); Calc: threaded
Built on September 21st 2018

*** Bug 120484 has been marked as a duplicate of this bug. ***

I can confirm that with 6.1.2.1, I can now open and even edit the manuscript with comments.  In contrast, with 6.0.5 I gave up after 20 mins of waiting for it to load the file and render the first page (at 100% CPU).
With 6.1.2.1 it will open the file and load it in 2 mins, and I can scroll and even edit.

*** Bug 120102 has been marked as a duplicate of this bug. ***

*** This bug has been marked as a duplicate of bug 61558 ***

This bug might be related to 120102 and 61558 that relate only to scrolling performance.

But please note that the bug I reported was independent of scrolling.  For any document with thousands of comments, ANY edit of it, or interaction with LO, was incredibly slow, previously.

Please don't just close without explanation. If some of the duplicates is not reproducible anymore, all cannot be closed unless all should be tested.

Created attachment 150877 [details]
Perf flamegraph from lm.odt

Opening attachment 74336 [details]

Normally it seems to hang forever, UI stays grey.

Arch Linux 64-bit
Version: 6.3.0.0.alpha0+
Build ID: 1fee3f1da6291bfbcd75455512726215d41b3e83
CPU threads: 8; OS: Linux 5.0; UI render: default; VCL: gtk3; 
Locale: fi-FI (fi_FI.UTF-8); UI-Language: en-US
Calc: threaded
Built on 18 April 2019

hardware and LO improved witin the years, 7.1 can handele this document, 
but slow and laggy, 
(i start understanding the problem of long threads ... but ... long threads evolve for long unresolved bugs ...) 
observation: massive cpu load when hovering with the mouse pointer above the doc, 
similar to calc issue, 
you are! aware that similar problems exist for calc? someone attached a calc sheet, 
likely 'related',

*** Bug 137491 has been marked as a duplicate of this bug. ***

LO opens the file for 12 sec and closes for only one second in

Version: 7.3.0.0.alpha0+ (x64) / LibreOffice Community
Build ID: 47a8a65022e3fd7624c95d0341b4809aad11fddb
CPU threads: 8; OS: Windows 10.0 Build 19043; UI render: Skia/Raster; VCL: win
Locale: ru-RU (ru_RU); UI: en-US
Calc: threaded

It took around 1Gb of memory.

If a main problem was in a opening/closing problem then we can close it, but

the file still has a problem with scrolling, there are delays

Buovjaga, Aron what do you think?

(In reply to Roman Kuznetsov from comment #51)
> LO opens the file for 12 sec and closes for only one second in
> 
> Version: 7.3.0.0.alpha0+ (x64) / LibreOffice Community
> Build ID: 47a8a65022e3fd7624c95d0341b4809aad11fddb
> CPU threads: 8; OS: Windows 10.0 Build 19043; UI render: Skia/Raster; VCL:
> win
> Locale: ru-RU (ru_RU); UI: en-US
> Calc: threaded
> 
> It took around 1Gb of memory.
> 
> If a main problem was in a opening/closing problem then we can close it, but
> 
> the file still has a problem with scrolling, there are delays
> 
> Buovjaga, Aron what do you think?

The file at bug 137491 (duplicate) still pretty slow. (On opening; And track-changes reject all).

attachment 74336 [details] does open in only 10 secs for me now. Only the lagginess with scrolling remains, but even that is not horrible. I think we can lower severity.

Version: 7.4.0.0.beta1+ / LibreOffice Community
Build ID: cdf48e57da6b8a6a5eb4131340fa2c14be135714
CPU threads: 8; OS: Linux 5.18; UI render: default; VCL: kf5 (cairo+xcb)
Locale: fi-FI (fi_FI.UTF-8); UI: en-US
Calc: threaded