Bug 76481 - Using autofilter pulldown on column with particular amount of records will freeze graphical mode, after few minutes normal state is restored
Summary: Using autofilter pulldown on column with particular amount of records will fr...
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
4.2.2.1 release
Hardware: All All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard: target:7.0.0 target:7.1.0 target:7.0.0.1
Keywords: perf
: 124450 131719 (view as bug list)
Depends on:
Blocks: AutoFilter
  Show dependency treegraph
 
Reported: 2014-03-22 15:12 UTC by oc
Modified: 2020-11-08 07:20 UTC (History)
16 users (show)

See Also:
Crash report or crash signature:


Attachments
File containing more that 220000 records in one column. (329.96 KB, application/vnd.oasis.opendocument.spreadsheet)
2014-03-22 15:12 UTC, oc
Details
spreadsheet with more than 13.000 columns (647.42 KB, application/vnd.oasis.opendocument.spreadsheet)
2014-09-19 09:42 UTC, Haico
Details

Note You need to log in before you can comment on or make changes to this bug.
Description oc 2014-03-22 15:12:16 UTC
Created attachment 96201 [details]
File containing more that 220000 records in one column.

Reproducing steps: Open attached file, open pull down menu on A1 cell, enjoy freezed system. 

No other actions is possible in graphical mode. Except opening terminal with keyboard shortcuts. Also switching to tty1 or whatever works.
System resumes normal state after few minutes. 
Reformatting column does not give any results. Also with few hundreds records bug is not reproducible.

Error was reproduced with LibreOffice versions
4.2.2.1
4.2.3.1

Was not reproduced with 4.1.x.x (whatever version comes by default with Mint 16)

On Ubuntu Gnome 13.10 64bit, Linux Mint 16 64bit on 3 different PCs.
Comment 1 oc 2014-03-23 10:47:16 UTC
Was able to reproduce this bug un Windows 7 64 system with LibreOffice 4.2.0.4 Calc using same xlsx file, after presing A1 pulldown autofilter button - calc freezes, and starts working again after few minutes. After that It starts to work very laggy and generaly is freezing after every simple action. Becomes unusable.
Comment 2 sophie 2014-03-24 13:30:43 UTC
I can reproduce on 4.3.2.1 on Ubuntu 13.10, freeze and take sometimes to come back. 
I do not reproduce with 4.1.5.3 under Debian 6. 
Set to New, change system because also reproduced under Windows, lower importance as 4.2.x is not production ready - Sophie
Comment 3 oc 2014-03-26 12:01:35 UTC
Reproduced on Manjaro 0.8.9 LibreOffice 4.2.2.1
Comment 4 oc 2014-04-07 11:40:42 UTC
Reproduced on 4.2.3.3 420m0 build3 ubuntu 13.10
Comment 5 Haico 2014-09-19 09:42:50 UTC
Created attachment 106543 [details]
spreadsheet with more than 13.000 columns
Comment 6 Haico 2014-09-19 09:45:06 UTC
Comment on attachment 106543 [details]
spreadsheet with more than 13.000 columns

Error was reproduced since the libroffice 4.2 series up to the latest nightly builds (19 sept. 4.3.3.). File contains more than 13.000 columns.
Using windows 8.1.

Didn't have this problem while trying it out on a windows 7 64 bit laptop.
Comment 7 Kevin Suo 2016-10-27 02:08:21 UTC
Version: 5.3.0.0.alpha1
Build ID: f4ca1573fcf445164c068c1046ab5d084e1b005f
CPU Threads: 4; OS Version: Linux 4.4; UI Render: default; VCL: gtk2; 
Locale: zh-CN (zh_CN.UTF-8); Calc: group

This system does not freeze when I open the drop down of A1.
However the system does freeze when I was trying to check a value in the list and click "OK".
Comment 8 ju0815nk 2017-04-03 12:35:28 UTC
*** Bug 102927 has been marked as a duplicate of this bug. ***
Comment 9 ju0815nk 2017-04-03 13:02:22 UTC
Same problem can be reproduced with v5.3.1 (see duplicate bug 102927 for details).
Comment 10 Xisco Faulí 2017-10-27 10:59:10 UTC
*** Bug 105406 has been marked as a duplicate of this bug. ***
Comment 11 Aaron Wang - TW 2018-09-15 06:10:13 UTC
Same as version 6.1.0.3 release
Comment 12 b. 2019-04-05 20:44:45 UTC
on fast xeon system win 7 (x64) ~ three seconds to open filter list on sheet with 5 columns and 219000 rows, 

Version: 6.3.0.0.alpha0+ (x64)
Build ID: 9956cf0692058414ef3efdb0e8058fbb0b39f6bc
CPU threads: 8; OS: Windows 6.1; UI render: default; VCL: win; 
TinderBox: Win-x86_64@42, Branch:master, Time: 2019-04-04_02:20:31
Locale: de-DE (de_DE); UI-Language: en-US
Calc: 

wfm?
Comment 13 berger 2019-09-10 16:56:28 UTC
Version: 6.2.6.2
Build ID: 1:6.2.6-0ubuntu0.16.04.1~lo1
CPU threads: 8; OS: Linux 4.4; UI render: default; VCL: gtk2; 
Locale: en-US (en_US.UTF-8); UI-Language: en-US
Calc: threaded

i7-4712MQ CPU @ 2.30GHz

Opening the autofilter UI takes only a second.

Note that there still might be/are other freezing issues with the autofilter (see bug #102927)

This bug -> Works for me.

I will reopen #102927.
Comment 14 perie_gut 2020-03-22 12:56:34 UTC
Confirming that the issue is still open under the latest stable and fresh release


Version: 6.3.5.2 (x64)
Build ID: dd0751754f11728f69b42ee2af66670068624673
CPU threads: 8; OS: Windows 10.0; UI render: default; VCL: win; 
Locale: en-PH (en_PH); UI-Language: en-US
Calc: threaded
Comment 15 V Stuart Foote 2020-03-22 13:59:54 UTC
*** Bug 124450 has been marked as a duplicate of this bug. ***
Comment 16 V Stuart Foote 2020-03-22 14:01:50 UTC
*** Bug 102927 has been marked as a duplicate of this bug. ***
Comment 17 V Stuart Foote 2020-03-22 14:06:49 UTC
There is this issue handling ODF spreadsheet (.ODS) and bug 123807 for OfficeOpen XML spreadsheet (.XMLS) --likely the same issue in filtering the lb treeview.
Comment 18 Roman Kuznetsov 2020-03-22 14:58:14 UTC
Noel, possibly you will be interested
Comment 19 V Stuart Foote 2020-03-22 15:01:11 UTC
still have long delay after making a string entry into the "Standard filter..." text box for a column with AutoFilter enabled. And, no means to escape from the filter action.

STR

attachment 96201 [details] here -- autofilter "PN" headed column using "00j" string

attachment 127796 [details] from dupe bug 102927  -- autofilter "Type" headed column using string "5Q"


On Windows 10 Home 64-bit en-US (1909) with
Version: 6.4.1.2 (x64)
Build ID: 4d224e95b98b138af42a64d84056446d09082932
CPU threads: 4; OS: Windows 10.0 Build 18363; UI render: GL; VCL: win; 
Locale: en-US (en_US); UI-Language: en-US
Calc: threaded

and with recent master/7.0.0alpha
Version: 7.0.0.0.alpha0+ (x64)
Build ID: 61d8d991a27c3bfe70e3b8d3b4ce4d8a41d18d2d
CPU threads: 4; OS: Windows 10.0 Build 18363; UI render: Skia/Vulkan; VCL: win; 
Locale: en-US (en_US); UI-Language: en-US
Calc: threaded
Comment 20 b. 2020-03-22 21:08:54 UTC
i was mislead when testing, didn't check #102927 which has more detailed description, autofilter and standardfilter as i normally use work fine, but!: 

there is a text box 'search items' underneath the link to 'Standard Filter' in the pulldown-box for autofilter, which - imho - hasn't anything to do with Standard Filter, 

i hadn't used it before, tested it now, while the list of available items is 'not too big' it's shrinking the list to items which contain the string in the text box, but above some threshold on lines or different items the calc autofilter is overstrained and  typing anything into this text-box sends calc to busy, unresponsive, and returns after long delay without any productive effect ... with very long lists already opening the autofilter-dropdown-box fails ... 

my system could handle 75.000 rows of the file provided with this bug, col A copied to col B,C,D and E, but freaks out with 100.000 rows. 

thus: there is a bug, 

versions tested: 

Version: 6.2.8.2 (x64)
Build ID: f82ddfca21ebc1e222a662a32b25c0c9d20169ee
CPU threads: 8; OS: Windows 6.1; UI render: default; VCL: win; 
Locale: de-DE (de_DE); UI-Language: en-US
Calc: 

and: 

Version: 7.0.0.0.alpha0+ (x64)
Build ID: 61d8d991a27c3bfe70e3b8d3b4ce4d8a41d18d2d
CPU threads: 8; OS: Windows 6.1 Service Pack 1 Build 7601; UI render: GL; VCL: win; 
Locale: de-DE (de_DE); UI-Language: en-US
Calc:
Comment 21 NISZ LibreOffice Team 2020-03-31 07:54:11 UTC
*** Bug 131719 has been marked as a duplicate of this bug. ***
Comment 22 b. 2020-05-18 12:56:29 UTC
this bugs file and the initial bug description working fine - open dropdown nopro, selecting value nopro - and thus leading to the idea 'solved', 

while the description to key something into the search box is hidden in plenty comments, 

and other bugs with better description are pulled away as duplicates, 

blocks getting in touch with the problem, 

thus i changed the title to  - !type in search box! - because that triggers the mentioned bug on my system,  

there are! other conditions and other freezes in similar and dup bugs, and hints on specific conditions as well in this thread, with 7.0.0.0.a1+ on win7 x64 i could only repro the search box issue, but that 'all the time'.
Comment 23 b. 2020-05-18 20:28:56 UTC
*** Bug 124080 has been marked as a duplicate of this bug. ***
Comment 24 Commit Notification 2020-05-21 14:02:47 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/f71557e958a8a626dfc1eef646b84b3c8b72569a

tdf#76481 speed up searching in autofilter pulldown

It will be available in 7.0.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 25 b. 2020-05-23 16:58:29 UTC
@Noel: don't see real better performance, neither with lin (master form today homebrew) nor with win (daily from today), 

@all: technically it is! possible to have the results of an instring search even with 100k random! cells within milliseconds, see standard filter and 'contains', 

thus why is it a problem for autofilter since 6+ years with 11+ duplicate and plenty similar and related bugs ...
Comment 26 Telesto 2020-05-23 18:42:27 UTC
(In reply to b. from comment #25)
> @Noel: don't see real better performance, neither with lin (master form
> today homebrew) nor with win (daily from today), 

No difference indeed
Version: 7.0.0.0.alpha1+ (x64)
Build ID: b587de60d4e6aa96238766272d94f1499b22f696
CPU threads: 4; OS: Windows 6.3 Build 9600; UI render: Skia/Raster; VCL: win; 
Locale: nl-NL (nl_NL); UI: en-US
Calc: CL
Comment 27 NISZ LibreOffice Team 2020-05-25 06:57:11 UTC
(In reply to Telesto from comment #26)
> (In reply to b. from comment #25)
> > @Noel: don't see real better performance, neither with lin (master form
> > today homebrew) nor with win (daily from today), 
> 
> No difference indeed

With the first attachment, filtering for a letter (tried W) in 220 000 lines in 

Version: 6.4.0.3 (x86)
Build ID: b0a288ab3d2d4774cb44b62f04d5d28733ac6df8
CPU threads: 4; OS: Windows 6.3 Build 9600; UI render: default; VCL: win; 
Locale: hu-HU (hu_HU); UI-Language: en-US
Calc: CL

took about 2.5 mins, now with latest nightly

Version: 7.0.0.0.alpha1+ (x64)
Build ID: 21875558f6c478f07d68ff39e025d7ffd451674f
CPU threads: 4; OS: Windows 6.3 Build 9600; UI render: Skia/Raster; VCL: win
Locale: hu-HU (hu_HU); UI: en-US
Calc: CL

it takes between 22-28 secs (I measured multiple letters with my phone stopwatch). 

I see real difference, but there should still be room for further improvement.
Comment 28 Commit Notification 2020-05-29 17:25:55 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/b81432a23c900329ece07854fd06a322225a97c1

fix tree disabled in autofilter pulldown, tdf#76481 related

It will be available in 7.1.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 29 Commit Notification 2020-05-30 11:24:41 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "libreoffice-7-0":

https://git.libreoffice.org/core/commit/98aced3625168e454679d3a14ebcdf55a67cbc18

fix tree disabled in autofilter pulldown, tdf#76481 related

It will be available in 7.0.0.1.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 30 Roman Kuznetsov 2020-05-31 18:58:25 UTC
20 sec when I try type a "W" letter in

Version: 7.1.0.0.alpha0+
Build ID: 2047a5978ac8188e61da9cd3b2f40d86df5570bb
CPU threads: 4; OS: Mac OS X 10.15.4; UI render: default; VCL: osx
Locale: ru-RU (ru_RU.UTF-8); UI: en-US
Calc: threaded
Comment 31 Kevin Suo 2020-11-08 07:20:11 UTC
There are (were) 2 different problems in the autofilter:

1. When a sheet contains may records, then it was slow when you open the autofilter dropdown list (without typing any text in the search box). 

This is what the original bug reporter has reported. And it was true in previous versions, but based on my test it has improved a lot. At least, on master branch under Fedora 32 I can open the 1st test doc attached to this bug (attachment 96201 [details]) within 4 seconds. This was reported to take several minutes to open up the autofilter dropdown. As a result, this issue was RESOLVED.

2. When a column contains many unique items, then it is very slow when you type in a char in the autofilter dropdown search box. It is only slow when there are many unique items (i.e., it is not slow to type in the search box if the column only contains several limited number of unique values)

This (the 2nd) bug behaviour is still reproducible on master builds as of today, and I suspect a lot of duplicated bug reports. One example was the one I have reported in bug 122419.

The bad thing is that, a lot of bugs with behaviour #2 (but not #1) was wrongly marked as duplicates of this bug (i.e., the bug I am now posting). For instance, bug 102927.

As a result, I will mark this bug as RESOLVED FIXED, and move those with behaviour #2 to bug 122419.