Bug 133804 - Autofilter is very slow when deselecting one item and click OK if the column headers are not defined
Summary: Autofilter is very slow when deselecting one item and click OK if the column ...
Status: RESOLVED DUPLICATE of bug 133835
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
5.4.0.3 release
Hardware: All All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard: target:7.3.0
Keywords: bibisected, bisected, haveBacktrace, perf, regression
Depends on:
Blocks: AutoFilter
  Show dependency treegraph
 
Reported: 2020-06-08 19:05 UTC by Telesto
Modified: 2021-11-25 13:26 UTC (History)
10 users (show)

See Also:
Crash report or crash signature:


Attachments
Bibisect log attempt (6.42 KB, text/plain)
2020-06-08 19:08 UTC, Telesto
Details
Example file (36.01 KB, application/vnd.oasis.opendocument.spreadsheet)
2020-06-09 09:08 UTC, Telesto
Details
Bibisect log (2.90 KB, text/plain)
2020-06-09 09:55 UTC, Telesto
Details
Perf flamegraph (71.77 KB, image/svg+xml)
2020-10-19 11:44 UTC, Buovjaga
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Telesto 2020-06-08 19:05:59 UTC
Description:
Autofilter performance regressed from 9 seconds to 27 seconds

Steps to Reproduce:
1. Open the attached file 161773
2. Click autofilter drop down column H
3. Uncheck 32132 and press OK

Actual Results:
27 seconds with 7.1
15 seconds with 4.2 master


Expected Results:
9 seconds as with 4.1


Reproducible: Always


User Profile Reset: No



Additional Info:
Found in
Version: 7.0.0.0.beta1+
Build ID: c344de1b9985b6ca10b354e24151d0bdf92dc20e
CPU threads: 2; OS: Linux 5.3; UI render: default; VCL: x11
Locale: en-US (en_US.UTF-8); UI: en-US
Calc: threaded

but not in
4.1
Comment 1 Telesto 2020-06-08 19:08:21 UTC
Created attachment 161778 [details]
Bibisect log attempt

Bibisect skipping hell. so only a partial bibisect... The bibisect is about the bump from 9 to 15 seconds.. so there must another point of failure
Comment 2 V Stuart Foote 2020-06-08 20:21:42 UTC
@Telesto, the attachment 161773 [details] seems to have had its formatting scrambled--the header row has been split. Not sure you've got a viable test document to work against. When I clean it up and get the header row back in place, and remove the multiple filters already applied (clearing the AutoFilter) I've no issue filtering against column the H "HONORAIROS CIRUJANO" values.

Also if that is actual real world data, there is a privacy issue as the document does not belong on the TDF BZ instance-- Stuart
Comment 3 Telesto 2020-06-08 21:19:32 UTC Comment hidden (obsolete)
Comment 4 Telesto 2020-06-08 21:23:00 UTC
(In reply to Telesto from comment #3)
> (In reply to V Stuart Foote from comment #2)
> > Also if that is actual real world data, there is a privacy issue as the
> > document does not belong on the TDF BZ instance-- Stuart
> 
> Source:
> https://bug-attachments.documentfoundation.org/attachment.cgi?id=119985
Bug 95346 actually..  If there need to remove it
Comment 5 Telesto 2020-06-09 09:08:11 UTC Comment hidden (obsolete)
Comment 6 Telesto 2020-06-09 09:12:02 UTC
NEW file
1. Open the attached file
2. Column H uncheck 23359 -> Wait
3. Click Drop down column D -> Wait again
Comment 7 Xisco Faulí 2020-06-09 09:16:12 UTC
(In reply to Telesto from comment #6)
> NEW file
> 1. Open the attached file
> 2. Column H uncheck 23359 -> Wait
> 3. Click Drop down column D -> Wait again

Wait for how long ? 27 seconds or forever ?
Comment 8 Telesto 2020-06-09 09:42:22 UTC
(In reply to Xisco Faulí from comment #7)
> (In reply to Telesto from comment #6)
> > NEW file
> > 1. Open the attached file
> > 2. Column H uncheck 23359 -> Wait
> > 3. Click Drop down column D -> Wait again
> 

Step 2: 19 seconds -> previously 4.1: 0,5 second
Step 3: 17 seconds -> previously 4.1: 0,5 second
Comment 9 Telesto 2020-06-09 09:55:02 UTC
Created attachment 161794 [details]
Bibisect log

Bisected to
author	Markus Mohrhard <markus.mohrhard@googlemail.com>	2017-04-09 22:13:24 +0200
committer	Markus Mohrhard <markus.mohrhard@googlemail.com>	2017-04-11 00:34:47 +0200
commit	eefe13c77d01be37c911e75af191717a944fedb3 (patch)
tree	1c4d7e9e698446e95ce9c63ecaf65474e0303913
parent	091a0dd416171393d939aac924433063a68967b0 (diff)
shrinking the DB area just causes problems, tdf#89546
Comment 10 V Stuart Foote 2020-06-09 12:32:58 UTC
(In reply to Telesto from comment #8)
> (In reply to Xisco Faulí from comment #7)
> > (In reply to Telesto from comment #6)
> > > NEW file
> > > 1. Open the attached file
> > > 2. Column H uncheck 23359 -> Wait
> > > 3. Click Drop down column D -> Wait again
> > 
> 
> Step 2: 19 seconds -> previously 4.1: 0,5 second
> Step 3: 17 seconds -> previously 4.1: 0,5 second

OK, see some delay...

Version: 7.1.0.0.alpha0+ (x64)
Build ID: 59939d2490726336546c7ad05082d23031074e12
CPU threads: 4; OS: Windows 10.0 Build 18363; UI render: Skia/Vulkan; VCL: win
Locale: en-US (en_US); UI: en-US
Calc: threaded

Seems happier (as in NO delay) if I simply populate the 1st row with column labels, and apply the 'AutoFiler' against the row with the lables--rather than the first row with data.

Then Inserting rows above the now column labeled row does not degrade--so issue seems to be with the autofilter applied to selected data rows without column headers.

LO has some sense of this status, as with AutoFilter set simply removing the rows with column header will prompt:

The range does not contain column headers.
Do you want the first line to be used as column header?

So, add the column headers back into the sample file and retest.

And, why would lack of column headers make the filter action so much slower?
Comment 11 b. 2020-06-09 14:40:31 UTC
a short look: 

the __anonymous_databaserange_ used in your file covers nearly all rows, "Sheet1.A1:Sheet1.I1048542", 

once you shrink this by deleting all rows below the data autofilter becomes fast again, 

if 4.1 was faster with exact this file it likely has better management of empty columns or adapting the database range or similar ... 

likely not a bug but 'use case'? 

Version: 7.0.0.0.alpha1
Build ID: 6a03b2a54143a9bc0c6d4c7f1...
CPU threads: 8; OS: Linux 4.19; UI render: default; VCL: gtk3; 
Locale: de-DE (en_US.UTF-8); UI: en-US
Calc: threaded
Comment 12 V Stuart Foote 2020-06-09 15:42:36 UTC
(In reply to b. from comment #11)
> a short look: 
> 
> the __anonymous_databaserange_ used in your file covers nearly all rows,
> "Sheet1.A1:Sheet1.I1048542", 
> 
> once you shrink this by deleting all rows below the data autofilter becomes
> fast again, 
> 
> if 4.1 was faster with exact this file it likely has better management of
> empty columns or adapting the database range or similar ... 
> 
> likely not a bug but 'use case'? 
> 

I guess that tracks--if I remove the Autofilter to add column labels and add then add it back it is recalculating the range. 

So probably not the presence/absence of the column labeling?
Comment 13 Buovjaga 2020-10-19 11:44:02 UTC
Created attachment 166502 [details]
Perf flamegraph

Arch Linux 64-bit
Version: 7.1.0.0.alpha0+
Build ID: ccdb78773ac6c9d19140e8084f37cc2c7f06240e
CPU threads: 8; OS: Linux 5.8; UI render: default; VCL: kf5
Locale: fi-FI (fi_FI.UTF-8); UI: en-US
Calc: threaded
Built on 18 October 2020
Comment 14 Buovjaga 2020-10-19 11:44:19 UTC
Someone can evaluate it.
Comment 15 Luboš Luňák 2021-11-23 17:33:46 UTC

*** This bug has been marked as a duplicate of bug 133835 ***
Comment 16 Commit Notification 2021-11-23 21:42:45 UTC
Luboš Luňák committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/990d32dad6bdf5d92a14d8bc02ae2853facd9097

allow matching of empty cells as svl::SharedString (tdf#133804)

It will be available in 7.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 17 Commit Notification 2021-11-25 13:26:46 UTC
Luboš Luňák committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/02f3157f75b654ef7648efdc3004b3f326d5af40

don't fetch cell string content for each query item (tdf#133804)

It will be available in 7.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.