Bug Hunting Session
Bug 99607 - High CPU load with filter active
Summary: High CPU load with filter active
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
4.2.0.4 release
Hardware: All All
: high normal
Assignee: Markus Mohrhard
URL:
Whiteboard: target:5.3.0 target:5.2.3
Keywords: bibisectRequest, perf, regression
Depends on:
Blocks:
 
Reported: 2016-05-01 13:23 UTC by Johnny_M
Modified: 2016-10-10 04:02 UTC (History)
5 users (show)

See Also:
Crash report or crash signature:


Attachments
Example file to display the issue (1.01 MB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
2016-05-01 13:23 UTC, Johnny_M
Details
Screenshot with filter active for a while (102.56 KB, image/png)
2016-05-01 13:24 UTC, Johnny_M
Details
Screenshot after filter deactivation (107.52 KB, image/png)
2016-05-01 13:25 UTC, Johnny_M
Details
Screenshot with filter active, Debian 32bit (110.26 KB, image/png)
2016-05-01 20:52 UTC, Johnny_M
Details
Screenshots of tests with WinXP and Linux Mint (LO 4.2.6, 4.3.5, 5.0.5) (56.38 KB, image/png)
2016-05-04 16:10 UTC, Johnny_M
Details
Screenshot with filter active, Windows 10 Home, 64bit (143.55 KB, image/png)
2016-05-06 11:29 UTC, Johnny_M
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Johnny_M 2016-05-01 13:23:00 UTC
Created attachment 124761 [details]
Example file to display the issue

While a filter is active in Calc, the CPU resource usage is significantly high. It seems to depend on the number of data items on the sheet, as well as their diversity, how high the CPU resources rise. On a lower-spec machine, using a filter on a large spreadsheet will be accompanied by significant delays in filter selection, activation and general LO responsiveness.

Test procedure:
1. Open the attached "SA8000_Q1_2015_public_list.xlsx" file (found on the internet to illustrate the issue, which was originally observed with an even greater impact on a different XLSX file, but which can't be provided due to confidentiality reasons).
2. Activate any one of the filters it contains.
(Note: Select only one data point for filtering. Because selecting two (e.g., "CHINA" and "China" under Country) seems to make the issue go away after an initial CPU resource rise.)
3. Deactivate the filter.

Expected results:
2. A resource rise due to filter change, followed by the resources going back down to idle afterwards.
3. Same as point 2.

Actual results:
2. A continuous high CPU resource consumption. (Switching between the CPU cores.)
3. As expected - the resource consumption goes back to idle.


Linux Mint 17.1, 64 bit, on a Netbook with the dual core Intel Atom N450 mobile processor and 2GB RAM
LO as provided by ppa:libreoffice/libreoffice-5-0
Version: 5.0.5.2
Build ID: 1:5.0.5~rc2-0ubuntu1~trusty1
Comment 1 Johnny_M 2016-05-01 13:24:25 UTC
Created attachment 124762 [details]
Screenshot with filter active for a while
Comment 2 Johnny_M 2016-05-01 13:25:34 UTC
Created attachment 124763 [details]
Screenshot after filter deactivation
Comment 3 raal 2016-05-01 18:19:55 UTC
No repro Version: 5.2.0.0.alpha0+
Build ID: 170a473597534cf59887b1d817538322e7039862
CPU Threads: 4; OS Version: Linux 4.2; UI Render: default; 
TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2016-04-19_00:41:06
Comment 4 Johnny_M 2016-05-01 20:52:51 UTC
Created attachment 124770 [details]
Screenshot with filter active, Debian 32bit

(In reply to raal from comment #3)
> No repro Version: 5.2.0.0.alpha0+

Thanks for testing!

I can reproduce it with current master on Debian 32-bit though (same machine, liveUSB):
Version: 5.2.0.0.alpha1+
Build ID: 039b75d6cdc26dcce03e37c67115405e6f2a8ebe
CPU Threads: 2; OS Version: Linux 4.3; UI Render: default; 
TinderBox: Linux-rpm_deb-x86@71-TDF, Branch:master, Time: 2016-05-01_01:31:14
Locale: en-US (en_US.UTF-8)
Comment 5 Cor Nouws 2016-05-03 12:06:54 UTC
I cannot reproduce this in Version: 5.2.0.0.alpha1+
Build ID: 07a641b110beee4f7c76617fcd6ed558025321a2
CPU Threads: 2; OS Version: Linux 4.4; UI Render: default; 
TinderBox: Linux-rpm_deb-x86@71-TDF, Branch:master, Time: 2016-05-02_02:15:45
Locale: nl-NL (nl_NL.UTF-8)
and in 5.2.1.1

But I do notice a peak when starting the filter as suggested.
Comment 6 Johnny_M 2016-05-04 11:59:17 UTC
Thanks for testing!

The impact of this issue might depend on the power of the used machine. E.g., I noticed on other, smaller files that the CPU usage was increased (e.g. to 50%), but did not go all the way to 100% like on a larger file like the one attached.

Additional information: I tested on a five-year-old netbook with the Intel Atom N450 CPU, 1 core, 2 threads, 1.66GHz (see http://ark.intel.com/products/42503/Intel-Atom-Processor-N450-512K-Cache-1_66-GHz ), 2GB RAM, 1024x600 screen.

The process appearing responsible in the System Monitor is "soffice.bin". Its memory consumption does not go up with the issue.
Comment 7 Johnny_M 2016-05-04 16:10:30 UTC
Created attachment 124843 [details]
Screenshots of tests with WinXP and Linux Mint (LO 4.2.6, 4.3.5, 5.0.5)

Additional tests on the same machine:

- Windows XP Home SP3, 32-bit:
Version: 4.3.5.2
Build-ID: 3a87456aaa6a95c63eea1c1b3201acedf0751bd5

Version: 5.0.5.2
Build-ID: 55b006a02d247b5f7215fc6ea0fde844b30035b3


- Linux Mint 17.1 (original ISO as liveUSB), 64-bit:
Version: 4.2.6.3
Build ID: 420m0(Build:3)


Both 4.x versions above show a smaller issue impact than the 5.x versions tested so far. But there is still a significant continuous resource increase. So, I'm going to update the affected version and OSes accordingly.
Comment 8 Johnny_M 2016-05-06 11:29:43 UTC
Created attachment 124879 [details]
Screenshot with filter active, Windows 10 Home, 64bit

The issue is also reproducible on the following:

I. A new desktop PC with Intel Core i5-6400 CPU, 3.2GHz, cuadcore
Intel H110 chipset
4GB RAM (3.89GB available)
CPU-built-in graphics, 1920x1080 screen
Windows 10 Home 64-bit
LO Version: 5.0.5.2
Build-ID: 55b006a02d247b5f7215fc6ea0fde844b30035b3


II. A notebook from the year 2002
mobile AMD Athlon XP2200+

518MHz, 512MB RAM (448MB available)
Radeon IGP 320M graphics card
1024x768 screen

Tested with Windows XP Professional SP3, 32-bit
No Java installed
LO (LibreOfficePortablePrevious_5.0.5_MultilingualAll.paf.exe):
Version: 5.0.5.2
Build-ID: 55b006a02d247b5f7215fc6ea0fde844b30035b3


--------------------------------------------------

ADDITIONAL TESTS

Note: I'm going to use following PC configurations on these tests:
- Config 1: The Intel Atom N450 netbook from the comment 6 with Linux Mint 17.1 (original ISO as liveUSB), 64-bit

- Config 2: The above AMD Athlon XP2200+ notebook from this comment (point II), with Windows XP Professional SP3, 32-bit

All tested LO versions downloaded from https://downloadarchive.documentfoundation.org/libreoffice/old/


Tests:

A) Issue not reproducible / different issue

a) LO 3.3.0
- PC config 2 with LibreOfficePortable_3.3.0.paf.exe
OOO330m19 (Build:6)

tag libreoffice-3.3.0.4

=> There is a different issue: CPU load goes up on filter activation, but then comes back down of itself after a few minutes

b) LO 3.5.3.2
- PC config 2 with LibreOfficePortable_3.5.3.1_MultilingualNormal.paf.exe:
LibreOffice 3.5.3.2

Build-ID: 235ab8a-3802056-4a8fed3-2d66ea8-e241b8

=> There is a different issue: File import takes ca. 10-15min, after which the file is not displayed correctly - only column A is shown, while the rest of the used columns is a white field.
Then as in the point c below.

c) LO 3.6.4.3
- PC config 2 with LibreOfficePortable_3.6.4_MultilingualAll.paf.exe
Version 3.6.4.3 (Build ID: 2ef5aff)

=> There is a different issue: Continuously elevated CPU load (around 65%) after file import. The load goes down ("soffice.bin" at 0% CPU) when one tries closing the modified (e.g. by filter activation) file and the "Save changes?" dialog appears (which is canceled). No issue with filter activation after that.

d) LO 4.0.4.2
- PC config 1 with LibreOffice_4.0.4.2_Linux_x86-64_deb.tar.gz
Version 4.0.4.2 (Build ID: 9e9821abd0ffdbc09cd8c52eaa574fa09eb08f2)
- PC config 2 with LibreOfficePortable_4.0.4_MultilingualAll.paf.exe
Version 4.0.4.2 (Build ID: 9e9821abd0ffdbc09cd8c52eaa574fa09eb08f2)

=> As in the point c above.

e) LO 4.1.6.2
- PC config 1 with LibreOffice_4.1.6.2_Linux_x86-64_deb.tar.gz
Version: 4.1.6.2
Build ID: 40ff705089295be5be0aae9b15123f687c05b0a

=> As in the point c above.


B) Issue reproducible
a) LO 4.2.0.1
- PC config 1 with LibreOffice_4.2.0.1_Linux_x86-64_deb.tar.gz
Version: 4.2.0.1
Build ID: 7bf567613a536ded11709b952950c9e8f7181a4a

b) LO 4.2.4.2
- PC config 1 with LibreOffice_4.2.4.2_Linux_x86-64_deb.tar.gz
Version: 4.2.4.2
Build ID: 63150712c6d317d27ce2db16eb94c2f3d7b699f8


------------------------------------------------

SUMMARY

- The issue first appears in LO 4.2.0.1. The LO 4.1.6.2 is the last good one, with a different issue (see next point).

- Appearance of this issue coincides with disappearance of a different issue: continuous elevated CPU usage by "soffice.bin" (ca. 20% with PC config 1, ca. 60% with PC config 2) after file import, until a file closure attempt. So that there may be a relation.
Comment 9 Johnny_M 2016-05-06 11:41:11 UTC
Setting the version to the 4.2.0.4 release, since the 4.2.0.1 RC is not available.
Comment 10 Buovjaga 2016-05-07 12:17:31 UTC
I repro in that CPU remains at 100% and only relaxes after I deactivate the filtering.

Johnny: would you like to join the QA team in triaging all the unconfirmed nasties? https://wiki.documentfoundation.org/QA/Triage_For_Beginners

64-bit, KDE Plasma 5
Build ID: 5.1.2.2 Arch Linux build-1
CPU Threads: 8; OS Version: Linux 4.5; UI Render: default; 
Locale: fi-FI (fi_FI.UTF-8)
Comment 11 Johnny_M 2016-05-07 18:42:45 UTC
(In reply to Buovjaga from comment #10)
> I repro in that CPU remains at 100% and only relaxes after I deactivate the
> filtering.

Thanks for testing!

> Johnny: would you like to join the QA team in triaging all the unconfirmed
> nasties? https://wiki.documentfoundation.org/QA/Triage_For_Beginners

Thanks for suggesting! Its a bit hard at the moment, because that poor little Intel Atom N450 CPU netbook above is currently my best machine. I'll be glad to get more involved once I get a new machine some time in the future. In the meantime I'm mostly limiting myself to just submitting and supporting the issues I experience. Thanks to all those who do more!
Comment 12 Buovjaga 2016-05-07 18:45:09 UTC
(In reply to Johnny_M from comment #11)
> Thanks for suggesting! Its a bit hard at the moment, because that poor
> little Intel Atom N450 CPU netbook above is currently my best machine. I'll
> be glad to get more involved once I get a new machine some time in the
> future. In the meantime I'm mostly limiting myself to just submitting and
> supporting the issues I experience. Thanks to all those who do more!

Cool :) Drop by the -qa IRC channel, when you are ready: https://wiki.documentfoundation.org/Website/IRC
Comment 13 Cor Nouws 2016-05-07 20:29:50 UTC
possibly 64 bits only?
Comment 14 Johnny_M 2016-05-07 22:58:33 UTC
(In reply to Cor Nouws from comment #13)
> possibly 64 bits only?

It doesn't seem to be. I've been able to reproduce it on following, per comments above:
- 32-bit mobile AMD Athlon XP2200+ CPU (1 core, 1 thread, external graphics), 1024x768 screen, with 32-bit WinXP Pro (installed)

- 64-bit Intel Atom N450 CPU (1 core, 2 threads, built-in graphics), 1024x600 screen, with 32-bit WinXP Home (installed), 32-bit Debian Jessie (liveUSB, Linux 4.3), 64-bit Linux Mint 17.1 (installed Linux 3.13 and liveUSB Linux 3.13)

- 64-bit Intel Core i5-6400 CPU (4 cores, 4 threads, built-in graphics), 1920x1080 screen, with 64-bit Win10 Home (installed)

Other reproduced:
- @Buovjaga with CPU Threads: 8; OS Version: Linux 4.5, 64-bit


Not reproduced:
- @raal with CPU Threads: 4; OS Version: Linux 4.2, 64-bit

- @Cor Nouws with CPU Threads: 2; OS Version: Linux 4.4, 32-bit


So that there could be some kind of HW dependency, but it's not the 64-bit by itself. It's not yet clear what it depends on. The PC age on my tests is also between 14 years (mobile AMD Athlon XP2200+ CPU) and 0 years (Intel Core i5-6400 CPU).
Comment 15 Markus Mohrhard 2016-08-04 23:39:20 UTC
Is this still an issue in 5.2? I think I fixed one or two performance issues around the filter popup for the 5.2 cycle.
Comment 16 Johnny_M 2016-08-05 11:27:13 UTC
(In reply to Markus Mohrhard from comment #15)
> Is this still an issue in 5.2? I think I fixed one or two performance issues
> around the filter popup for the 5.2 cycle.

Thanks for looking into this!

I've just checked the following and the issue is still there with all of those:

1) A year-2008 machine with AMD turion X2 processor, with 32-bit Windows Vista:
Version: 5.2.0.4
Build-ID: 066b007f5ebcc236395c7d282ba488bca6720265
CPU-Threads: 2; BS-Version: Windows 6.0; UI-Render: Standard;
Gebietsschema: de-DE (de_DE)

2) A year-2010 machine with Intel Atom N450, with:
a) 32-bit Debian on live USB:
Version: 5.3.0.0.alpha0+
Build ID: cb95159e79ee531f3908e30fefae73693672f0a6
CPU Threads: 2; OS Version: Linux 4.3; UI Render: default; 
TinderBox: Linux-rpm_deb-x86@71-TDF, Branch:master, Time: 2016-07-25_03:57:42
Locale: en-US (en_US.UTF-8); Calc: group

b) 64-bit Linux Mint 17.1 (based on Ubuntu 14.04.1) live USB:
Version: 5.3.0.0.alpha0+
Build ID: 39e300612c97b7742c8d8417d4dc6c0022cfa040
CPU Threads: 2; OS Version: Linux 3.13; UI Render: default; 
TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2016-08-05_01:22:59
Locale: en-US (en_US.UTF-8); Calc: group


I'm not sure why the issue couldn't be reproduced by @raal und @Cor Nouws. Maybe it somehow/sometime depends on the column where the filter is activated for the test, or the number of selected items in the filter. I tried several, all with the same results, but then (for reproducibility) started always using the same on the test step 2. So, here are the steps defined more specifically:

2. In the "Company" column filter, unselect "All", then select "LATECH S.R.L." (so that it's the only one active).
3. In the "Company" column filter, select "All" to deactivate the filtering.


For a rough regression point, please see the "SUMMARY" in the comment 8.
Comment 17 Johnny_M 2016-08-06 20:29:14 UTC
(In reply to Markus Mohrhard from comment #15)
> Is this still an issue in 5.2? I think I fixed one or two performance issues
> around the filter popup for the 5.2 cycle.

To prevent any confusion: Unlike the bug 91307, etc., this is not about the behavior of the "filter popup" (which behaves without problems for me). Rather, the issue appears/disappears after the popup gets closed using the OK button. (I missed adding the "... and click OK to confirm" part at the end of the test steps 2 and 3.)
Comment 18 Markus Mohrhard 2016-10-05 02:21:14 UTC
Looks like this one is related to online spelling checking.

When the range is filtered even the filtered range is considered part of the visible range. The spell check then goes through each cell and calls for each one a repaint.

There are several things we can do to improve the performance. The first one and most likely the easiest one is to skip hidden rows and columns. That one should already bring the performance to acceptable levels.

Finally I think it is insane to call a repaint for all the wrongly spelled cells so we might need to investigate how we can limit that to one repaint call.
Comment 19 Commit Notification 2016-10-08 17:44:04 UTC
Markus Mohrhard committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=eadd75ef212b4dd1b43aeacb34c8dd3ab40df369

skip hidden rows/columns for spellchecking, tdf#99607

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 20 Markus Mohrhard 2016-10-08 18:30:29 UTC
Review request for 5-2 in gerrit.
Comment 21 Commit Notification 2016-10-10 04:02:20 UTC
Markus Mohrhard committed a patch related to this issue.
It has been pushed to "libreoffice-5-2":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=64c25f120cd96cb6c1c232cc86661d9927840851&h=libreoffice-5-2

skip hidden rows/columns for spellchecking, tdf#99607

It will be available in 5.2.3.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.