Bug Hunting Session
Bug 99938 - Calc uses 100% CPU and leaks memory when calculating moving average of full column
Summary: Calc uses 100% CPU and leaks memory when calculating moving average of full c...
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
4.2.8.2 release
Hardware: All All
: medium normal
Assignee: Dennis Francis
URL:
Whiteboard: target:6.3.0
Keywords: perf
Depends on:
Blocks: Memory
  Show dependency treegraph
 
Reported: 2016-05-18 20:41 UTC by Robert Clipsham
Modified: 2019-06-27 08:31 UTC (History)
6 users (show)

See Also:
Crash report or crash signature:


Attachments
CSV file used to reproduce the problem (2.21 KB, text/csv)
2016-05-18 20:41 UTC, Robert Clipsham
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Robert Clipsham 2016-05-18 20:41:10 UTC
Created attachment 125159 [details]
CSV file used to reproduce the problem

When calculating a moving average for the attached CSV file, libreoffice uses 100% CPU, and uses increasing amounts of memory until I kill it.

Steps to reproduce:

1) Open the attached CSV file
2) Select both columns
3) Data -> Statistics -> Moving Average
   - Input range should be `$scores2.$A$1:$B$1048576`
   - Results To should be `$C$1`
   - Grouped by columns
   - Interval 20

Clicking OK will then cause 100% CPU usage on one core, and gradually more memory usage. It managed to take 10 minutes on my machine and use 2GB of RAM before I killed it.

If I select only the 142 rows of data then it calculates it almost instantly.

Expected behaviour:

The average should ignore the empty rows, and give a result as swiftly as just selecting the top 142.

Libreoffice Version: LibreOffice 5.1.3.2 10m0(Build:2)
Operating System: Arch Linux
Kernel: Linux 4.5.4-1-ARCH x86_64
Comment 1 m.a.riosv 2016-05-18 22:32:50 UTC
Hi @Robert, thanks for reporting.

Reproducible.
Version: 5.1.3.2 (x64)
Build ID: 644e4637d1d8544fd9f56425bd6cec110e49301b
CPU Threads: 4; OS Version: Windows 6.19; UI Render: default; 

Version: 5.0.6.2 (x64) Build ID: b3fbfa99158a1030fb79f0ba72b6851afc3c7895-GL

Version: 4.2.8.2
Comment 2 QA Administrators 2017-05-22 13:39:11 UTC Comment hidden (obsolete)
Comment 3 Robert Clipsham 2017-05-22 19:32:59 UTC
Still reproducible:

Version: 5.3.3.2
Build ID: 5.3.3-1
CPU Threads: 4; OS Version: Linux 4.10; UI Render: default; VCL: gtk3; Layout Engine: new;
Comment 4 m.a.riosv 2017-05-22 22:59:12 UTC
Still
Version: 5.3.3.2 (x64)
Build ID: 3d9a8b4b4e538a85e0782bd6c2d430bafe583448
CPU Threads: 4; OS Version: Windows 6.19; UI Render: GL; Layout Engine: new; 
Locale: es-ES (es_ES); Calc: group
Comment 5 Julien Nabet 2017-08-12 23:09:01 UTC
On pc Debian x86-64 with master sources updated yesterday, I could reproduce this.

Alex: could you give a try with XCode memory analyzer? (BTW, about this tool, which tuto have you used?)
Comment 6 QA Administrators 2018-10-09 02:52:57 UTC Comment hidden (obsolete)
Comment 7 Dennis Francis 2019-03-22 16:44:41 UTC
Still reproducible in master (HEAD=09b9499de17991bbd6b808c0c7923fda6e616e79). I'll try to fix it.
Comment 8 Commit Notification 2019-05-20 13:28:46 UTC
Dennis Francis committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/2fd8f56ed1bf9b547093c4ac6c3231fff001b748%5E%21

tdf#99938 : Moving Average : By default trim input range...

It will be available in 6.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 9 Commit Notification 2019-05-20 13:44:06 UTC
Dennis Francis committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/1bfbe2a44021ca4ae6716caa39fc8a375914be5c%5E%21

tdf#99938 : Allow batch of formula-cells to be written...

It will be available in 6.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 10 Dennis Francis 2019-05-20 17:47:43 UTC
Now there is a checkbox for "Trim input range to actual data content" (checked by default) in the moving average dialog. The underlying cause for slowness was also mitigated to some extend, so even without this new checkbox checked, the computation for the $A$1:$B$1048576 range as said in the reproduction steps finishes in a few seconds.
Comment 11 Roman Kuznetsov 2019-05-20 18:36:44 UTC
(In reply to Dennis Francis from comment #10)
> Now there is a checkbox for "Trim input range to actual data content"
> (checked by default) in the moving average dialog. The underlying cause for
> slowness was also mitigated to some extend, so even without this new
> checkbox checked, the computation for the $A$1:$B$1048576 range as said in
> the reproduction steps finishes in a few seconds.

Dennis, thank you for fix. Can you please write about this changing in to Release Notes 6.3 in our wiki?
Comment 12 Dennis Francis 2019-05-29 06:05:04 UTC
(In reply to Roman Kuznetsov from comment #11)
> (In reply to Dennis Francis from comment #10)
> > Now there is a checkbox for "Trim input range to actual data content"
> > (checked by default) in the moving average dialog. The underlying cause for
> > slowness was also mitigated to some extend, so even without this new
> > checkbox checked, the computation for the $A$1:$B$1048576 range as said in
> > the reproduction steps finishes in a few seconds.
> 
> Dennis, thank you for fix. Can you please write about this changing in to
> Release Notes 6.3 in our wiki?

Done.