Download it now!
Bug 131675 - Memory usage goes through the roof saving document with 120k of comments
Summary: Memory usage goes through the roof saving document with 120k of comments
Status: RESOLVED DUPLICATE of bug 119650
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
6.0.6.2 release
Hardware: All All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard:
Keywords: perf
Depends on:
Blocks: Calc-Comments CPU-AT-100% Memory
  Show dependency treegraph
 
Reported: 2020-03-29 15:51 UTC by Telesto
Modified: 2020-04-26 21:56 UTC (History)
3 users (show)

See Also:
Crash report or crash signature:


Attachments
perf flamegraph (when opening the file) (599.58 KB, application/x-bzip)
2020-04-26 09:13 UTC, Julien Nabet
Details
Flamegraph (when adding a new sheet) (26.52 KB, application/x-bzip)
2020-04-26 09:15 UTC, Julien Nabet
Details
Flamegraph (when file saving) (368.40 KB, application/x-bzip)
2020-04-26 09:23 UTC, Julien Nabet
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Telesto 2020-03-29 15:51:47 UTC
Description:
Memory usage goes through the roof saving document with 120k of comments

Steps to Reproduce:
1. Open attachment 156113 [details]  bug 128402
2. Insert a new sheet
3. Press Save

Actual Results:
At least 6 GB of memory usage 

Expected Results:
Something more reasonable for a 1 MB file


Reproducible: Always


User Profile Reset: No



Additional Info:
Version: 7.0.0.0.alpha0+ (x64)
Build ID: 7ae9c9572ccac55c0926b8a9779bb63c4236291c
CPU threads: 4; OS: Windows 6.3 Build 9600; UI render: Skia/Raster; VCL: win; 
Locale: nl-NL (nl_NL); UI-Language: en-US
Calc: CL
Comment 1 Roman Kuznetsov 2020-04-25 23:58:05 UTC
opening process takes 100% CPU.
saving process takes 100% CPU, over 10 minutes of time and over 7,5Gb of memory, LO frozen and I killed it

Version: 7.0.0.0.alpha0+
Build ID: 6c59c9d2b8818674640a50656ffba90f9cd3900e
CPU threads: 4; OS: Mac OS X 10.15.4; UI render: default; VCL: osx; 
Locale: ru-RU (ru_RU.UTF-8); UI-Language: en-US
Calc: threaded

the same problem in 6.0 on macOS

Julien, can you make your perfgraphs here for three cases with that file from description:

1. File opening
2. A new sheet adding (I saw 100% CPU in that case too)
3. File saving
Comment 2 Telesto 2020-04-26 08:19:46 UTC
(In reply to Roman Kuznetsov from comment #1)
> opening process takes 100% CPU.
> saving process takes 100% CPU, over 10 minutes of time and over 7,5Gb of
> memory, LO frozen and I killed it
> 
> Version: 7.0.0.0.alpha0+
> Build ID: 6c59c9d2b8818674640a50656ffba90f9cd3900e
> CPU threads: 4; OS: Mac OS X 10.15.4; UI render: default; VCL: osx; 
> Locale: ru-RU (ru_RU.UTF-8); UI-Language: en-US
> Calc: threaded
> 
> the same problem in 6.0 on macOS
> 
> Julien, can you make your perfgraphs here for three cases with that file
> from description:
> 
> 1. File opening
> 2. A new sheet adding (I saw 100% CPU in that case too)
> 3. File saving

Not sure which part is covered by bug 76324 and bug 125619
Comment 3 Telesto 2020-04-26 08:38:13 UTC
See also bug 114377 and bug 131672

The issue: multiple issues stacked onto each other. 
* File opening/saving speed with comments should be a parsing problem
* Memory usage on save with lots of comments  probably a symptom
* Slowdown after save bug 125619
* Insert a new sheet; slow. Probably around for a while. Not specific to comments as far i know
* And we have bug 131709, which you can run into testing something else
Comment 4 Julien Nabet 2020-04-26 09:13:47 UTC
Created attachment 159947 [details]
perf flamegraph (when opening the file)

Here's a Flamegraph retrieved on pc Debian x86-64 with master sources updated today.
It concerns only the file opening part.
Comment 5 Julien Nabet 2020-04-26 09:15:43 UTC
Created attachment 159948 [details]
Flamegraph (when adding a new sheet)

Remark: no delay here to add a sheet.
Comment 6 Julien Nabet 2020-04-26 09:23:21 UTC
Created attachment 159949 [details]
Flamegraph (when file saving)
Comment 7 Roman Kuznetsov 2020-04-26 09:31:14 UTC
Noel, could you look at it please?
Comment 8 b. 2020-04-26 19:39:15 UTC
Hello @Telesto, Roman and Julien, 

thanks for taking care for this problem, 

it is quite old and often discussed, see bugs: 131675 131672 129228 128402 127758 125619 125545 124692 123418 119650 119636 119075 114377 113599 106433 106385 105888 105499 97698 88194 76324 60418 56268 54563 54018 34519 and some more ... 

it is one of the bigger general problems besides 
- copy and paste, 
- autocalc / shared formulas / threading / openCL, 
- iterative calculations, and 
- floating point precision, 
  that are probably difficult to solve. 

'comments' is linked to 'copy/paste' somewhere in the drawing layer, where things have been specially arranged to allow clipboard pasting even if the source file has been closed in the meantime? 

Noel has already taken care of this problem and in https://bugs.documentfoundation.org/show_bug.cgi?id=119650#c11 he said: 'this is what I could do...', and pinpoints the problem. 

best progress I know: 6.2.7.1 and 6.2.8.2 - win - are comparatively good, 

catchwords I picked up: 

'non linear transformation between grid and display', 

'non linear transformation between grid and pointing device - accelerated moves', 

drawing layer (memory?) is registered by some obscure stuff, 

'layout for captions / annotations / notes / comments / tracking notes is done three times', 

A suspicion I have, some things in calc are processed 'from bottom to top, then right to left', but the comments are stored in the files (and in memory?) from left to right, then top to bottom ... an iteration 'calculate the space for comment 10 - you need the space for comment 9 - calculate the space for comment 9 - you need 8 ... need 1 ... then coming to 9, again calculating 8 ...1, and so on ... would lead to the fact that the calculation time would increase with every further comment (don't know how many times) ... but such a thing increases unneccessarily ...

[SUM(n = 1 to k+1: n*(n+1)/2)) instead of k+1 calculations for comment k+1??? observations look more like n*(n+1)/2]
Comment 9 Telesto 2020-04-26 21:56:22 UTC
@b. Thanks for reference.. this is duplicate of bug 119650; 

@Roman, your choice if you wanna open a new bug for file-opening..

@b. Part 2. Thanks for enthusiasm & reports 
A) Some advise: don't 'hijack' each and every Calc bug report, to mention the bigger general problems or note that 6.2.7.1 and 6.2.8.2 are working pretty good. Please keep it a bit on topic
B) Don't write long a life story, but keep it as short as possible.

Yes, there are quite a number of fundamental issues in Calc [be aware, Writer has surely the same amount of them]. Please do report them with Steps to reproduce.

However, one bug in each report. Only the steps needed; not a story about OpenCL, multithreading.. And preferably not reposting a known bug (however, can happen sometimes :-). 

I looked at bug 131339 & bug 124577 comment 10. Both are regressions which can be bibisected.

*** This bug has been marked as a duplicate of bug 119650 ***