Bug 83141 - Find & Replace extremely slow / freeze with large spreadsheet
Summary: Find & Replace extremely slow / freeze with large spreadsheet
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
4.2.5.2 release
Hardware: All All
: high critical
Assignee: Not Assigned
URL:
Whiteboard: BSA target:4.4.0
Keywords: bibisected, perf, regression
: 82569 (view as bug list)
Depends on:
Blocks: Find-Search mab4.3
  Show dependency treegraph
 
Reported: 2014-08-27 12:08 UTC by Nicolas R
Modified: 2022-03-30 13:40 UTC (History)
13 users (show)

See Also:
Crash report or crash signature:


Attachments
test file with 40000 anonymized adresses (1.18 MB, application/vnd.oasis.opendocument.spreadsheet)
2014-08-27 12:08 UTC, Nicolas R
Details
OSX backtrace (8.87 KB, text/plain)
2014-10-06 17:13 UTC, Matthew Francis
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Nicolas R 2014-08-27 12:08:28 UTC
Created attachment 105331 [details]
test file with 40000 anonymized adresses

Hi,

Since the news Search Results window in find replace function, this function has become extremely slow and even freezing Libo on large spreadsheets.

I often use Calc to treat large adresses files ( 5000 to 40000 rows / adresses) with large replace actions on some columns.

Try the following test with a real ( in size and number of rows , adresses have been anonymized) file join in attchment.
 
- Open the file
- Select the whole 'titre' column by clicking on 'D'
- Ctrl-H
- Find 'MME' Replace with 'MMES'
- Options validated : Entire cells, Current selection only, Search in Values and Search Direction Columns
- Then click 'Find All'

Old version 4.1.6 : not search reasults window, but you've got selection in less than 5 seconds. And replace all also takes less than 5 seconds

4.2.5 and recent 4.4.0 dev version (libo-master~2014-08-25_01.27.12_LibreOfficeDev_4.4.0.0.alpha0_Win_x86)
After few seconds you've got the search results windows with the first results, but Libo become unresponsive. you can only move Searc Results and Find Replace window. I've to stop / crash Libo after 35 minutes ...

Same problem with other dev version libo-43~2014-08-23_08.23.00_LibreOfficeDev_4.3.2.0.0_Win_x86 and libo-42~2014-08-22_11.48.56_LibreOfficeDev_4.2.7.0.0_Win_x86

So for this kind of work, new version totally unusable for me.

I think it's related with bug 79422 , but  Markus Mohrhard asked me to make a new bug report.
Comment 1 Nicolas R 2014-08-27 12:13:05 UTC
Some precisions : 

tested on win 7 pro 64 bits

And 
'standard install' LibO 4.2.5
other versions (4.1.6, 4.2.7, 4.3.2 and 4.4.0 )installed in parallel with LibreOffice Server Install gui
Comment 2 raal 2014-08-31 20:12:22 UTC
I can confirm that the bug is present in master.
Version: 4.4.0.0.alpha0+
Build ID: 37b9ea92ba81d74764a2345a9c75c65bfd272d2b
TinderBox: Linux-rpm_deb-x86_64@46-TDF, Branch:master, Time: 2014-08-26_09:48:30

No problem in LO 3.5, regression.
Comment 3 raal 2014-09-15 15:20:40 UTC
*** Bug 82569 has been marked as a duplicate of this bug. ***
Comment 4 Joel Madero 2014-09-23 22:38:32 UTC
In the future please don't add your own bugs (because of bias). If you think it belongs on the list, ping the QA mailing list and ask for a 2nd opinion. Thanks
Comment 5 Joel Madero 2014-09-23 22:41:16 UTC
A bibisect would be useful - should have been done before MAB add.
Comment 6 Nicolas R 2014-09-24 09:39:49 UTC
(In reply to comment #5)
> A bibisect would be useful - should have been done before MAB add.

What is a bibisect ?

This bug is a regression regarding previous versions (4.1.x and all versions without the "Find All results window"), and in some way all regressions are MAB if there are no workaround.

And there are few other bugs related to this problem I think, so I'm not the only one who's annoyed : bug 82569 , bug 79422 and perhaps bug 77515.

And sorry for MAB, but I don't know exactly who's allowed to add bug in this list.
Next time, I will post in French Q/A mailing list where I think somebody could adds a bug to MAB.
Comment 7 Joel Madero 2014-09-24 17:57:20 UTC
All regressions are not all MAB's - this is not the definition we use.

Bibisect Instructions: https://wiki.documentfoundation.org/QA/HowToBibisect
Comment 8 Joel Madero 2014-09-24 18:04:16 UTC
BTW - no worries, that's why I just included a comment. Anyone can add to the list but they should follow the guidelines and should not nominate their own bugs.

I need to update the wiki with clear guidelines....generally:
(1) don't nominate your own bugs, ask the QA list if you think your bug qualifies;
(2) Do as much work as possible before nominating - including narrowing down where the bug slipped in (easiest done by bibisecting using the instructions);
(3) Appropriately set the priority/importance according to guidelines here: https://wiki.documentfoundation.org/File:Prioritizing_Bugs_Flowchart.jpg (again be UNBIASED WHEN YOU DO IT) ;)
(4) When adding (by an unbiased person) make sure importance is set to Highest - this is preserved exclusively for MAB list.
(5) Leave comment on MAB tracker explaining the bug a bit.

While regression status does help a bug get to MAB list - it is not determinative (there are a couple hundred regressions floating around, many quite minor). More important to focus on workflow and the overall impact on users (if it's likely to affect a large portion of our users -- subjective analysis).
Comment 9 raal 2014-09-24 19:17:26 UTC
 83a62c1c1e8e259144e489d9a1f42611eba063c3 is the first bad commit
commit 83a62c1c1e8e259144e489d9a1f42611eba063c3
Author: Bjoern Michaelsen <bjoern.michaelsen@canonical.com>
Date:   Thu Oct 17 14:30:14 2013 +0000

    source-hash-022c54742e7997bf46a608f1ab0b500f2537f7f5
    
    commit 022c54742e7997bf46a608f1ab0b500f2537f7f5
    Author:     Tor Lillqvist <tml@iki.fi>
    AuthorDate: Tue Jun 25 07:19:41 2013 +0300
    Commit:     Tor Lillqvist <tml@iki.fi>
    CommitDate: Tue Jun 25 07:19:41 2013 +0300
    
        WaE: private field 'mrCells' is not used
    
        Change-Id: I0ab3fabb82c839f5194b0e20eb834dd86635a609

:100644 100644 4b10c5c8ddbedca0971e0839a8acc603792a447c 483b58760a06de929b32eafde25a67466c622502 M	ccache.log
:100644 100644 54c63dd94c275598f317bb54ddfdd27aaad5d8a1 fcfaf4eddaf5f8c7a66f90a052cbf2c7473cdc9b M	commitmsg
:100644 100644 e607019f9ceabe4513be6de63f5724c67ece57f9 3e023e83e964fd4b90d7bdf45eab489c7382956c M	dev-install.log
:100644 100644 2d16d57e331ca5fab2ec46ad12fe030528c544bb 47ead046b9af75e2384d8d8f51767edfa54d5dc8 M	make.log
:040000 040000 3aaab4081e7400904dc31731c74182db7e18493c 82a20807f2d069e8294cfa6e30778214a869a341 M	opt

 git bisect log
# bad: [423a84c4f7068853974887d98442bc2a2d0cc91b] source-hash-c15927f20d4727c3b8de68497b6949e72f9e6e9e
# good: [65fd30f5cb4cdd37995a33420ed8273c0a29bf00] source-hash-d6cde02dbce8c28c6af836e2dc1120f8a6ef9932
git bisect start 'latest' 'oldest'
# good: [e02439a3d6297a1f5334fa558ddec5ef4212c574] source-hash-6b8393474974d2af7a2cb3c47b3d5c081b550bdb
git bisect good e02439a3d6297a1f5334fa558ddec5ef4212c574
# bad: [4850941efe43ae800be5c76e1102ab80ac2c085d] source-hash-980a6e552502f02f12c15bfb1c9f8e6269499f4b
git bisect bad 4850941efe43ae800be5c76e1102ab80ac2c085d
# skip: [a043626b542eb8314218d7439534dce2fc325304] source-hash-9379a922c07df3cdb7d567cc88dfaaa39ead3681
git bisect skip a043626b542eb8314218d7439534dce2fc325304
# skip: [aba65c3e4c0df07e4909aeefb758cdb688242bf6] source-hash-827524abfb4b577d08276fde40929a9adfb7ff1a
git bisect skip aba65c3e4c0df07e4909aeefb758cdb688242bf6
# skip: [aba65c3e4c0df07e4909aeefb758cdb688242bf6] source-hash-827524abfb4b577d08276fde40929a9adfb7ff1a
git bisect skip aba65c3e4c0df07e4909aeefb758cdb688242bf6
# bad: [c81a8a0dcfc1ed095a80e4485c89dd0fcaf73f31] source-hash-c69ed33628ec0b7abf6296539cf280d6c4265930
git bisect bad c81a8a0dcfc1ed095a80e4485c89dd0fcaf73f31
# bad: [c81a8a0dcfc1ed095a80e4485c89dd0fcaf73f31] source-hash-c69ed33628ec0b7abf6296539cf280d6c4265930
git bisect bad c81a8a0dcfc1ed095a80e4485c89dd0fcaf73f31
# bad: [c81a8a0dcfc1ed095a80e4485c89dd0fcaf73f31] source-hash-c69ed33628ec0b7abf6296539cf280d6c4265930
git bisect bad c81a8a0dcfc1ed095a80e4485c89dd0fcaf73f31
# bad: [c81a8a0dcfc1ed095a80e4485c89dd0fcaf73f31] source-hash-c69ed33628ec0b7abf6296539cf280d6c4265930
git bisect bad c81a8a0dcfc1ed095a80e4485c89dd0fcaf73f31
# bad: [1d4980621741d3050a5fe61b247c157d769988f2] source-hash-89d01a7d8028ddb765e02c116d202a2435894217
git bisect bad 1d4980621741d3050a5fe61b247c157d769988f2
# bad: [ba096f438393091574da98fe7b8e6b05182a8971] source-hash-8499e78ca03c792f4fa2650e02b519094ba0baa8
git bisect bad ba096f438393091574da98fe7b8e6b05182a8971
# bad: [9daa289e178460daaafa4b3911031df5b8736218] source-hash-704292996a3731a61339b1a4a5c90c9403aa095f
git bisect bad 9daa289e178460daaafa4b3911031df5b8736218
# good: [69bf614869471f46413fe1d2af5976b2e6d85084] source-hash-76dea8b2db906156e77f78738a68f932a15afd4b
git bisect good 69bf614869471f46413fe1d2af5976b2e6d85084
# good: [502c05c771cd993b237febc2d8a20140fe589488] source-hash-462df4920ef50032c8f99a9db2ca34c9cc928657
git bisect good 502c05c771cd993b237febc2d8a20140fe589488
# bad: [567bfa79fb5ad4f9dfa05f0dea7666208d6129b2] source-hash-4d5fc661d37d03129b8054e494c03bed1933231d
git bisect bad 567bfa79fb5ad4f9dfa05f0dea7666208d6129b2
# good: [7d878017eaa2fc1d2eab72689a5e453622d474a2] source-hash-b139f6fedfcf3cbed0eadeb007e2155b576413d2
git bisect good 7d878017eaa2fc1d2eab72689a5e453622d474a2
# bad: [83a62c1c1e8e259144e489d9a1f42611eba063c3] source-hash-022c54742e7997bf46a608f1ab0b500f2537f7f5
git bisect bad 83a62c1c1e8e259144e489d9a1f42611eba063c3
# first bad commit: [83a62c1c1e8e259144e489d9a1f42611eba063c3] source-hash-022c54742e7997bf46a608f1ab0b500f2537f7f5
Comment 10 Matthew Francis 2014-10-06 17:13:57 UTC
Created attachment 107439 [details]
OSX backtrace

Following the reproduction instructions on a recent build of master compiled --enable-dbgutil results in death from the following assertion:
(assertions are fatal in dbgutil builds)

Assertion failed: (index >= 0 && static_cast<sal_uInt32>(index) < static_cast<sal_uInt32>(getLength())), function operator[], file /Users/asbel/Development/LibreOffice/core/include/rtl/ustring.hxx, line 421.

OSX backtrace attached

The same happens on the Linux 4.4 bibisect repository (which is also a dbgutil build) as far back as the 4.3 branch point, which is the earliest it covers. Unfortunately while there are earlier bibisect repositories, they aren't dbgutil so I can't immediately tell if this is a different symptom of the same bug or not.
Comment 11 Caolán McNamara 2014-10-08 15:57:41 UTC
http://cgit.freedesktop.org/libreoffice/core/commit/?id=1e721077b43de84edab2a3ed2f316ddcbec6e3ec

hopefully the above can be tested in tomorrows dailies to see if this actually solves the problem
Comment 12 Bryan Quigley 2014-10-09 14:36:16 UTC
This commit 1e721077 does make it much more reasonable speed wise with an internal test document:
Before regression - ~10-15 seconds
Before recent patch - >5 minutes / borderline not responsive
With current patch - ~50 seconds
Comment 13 Nicolas R 2014-10-09 14:50:42 UTC
I will test as soon as possible !

Is the commit included in this release : 
http://dev-builds.libreoffice.org/daily/master/Win-x86@42/2014-10-09_03.25.27/libo-master~2014-10-09_03.25.27_LibreOfficeDev_4.4.0.0.alpha0_Win_x86.msi ?

at Bryan Quigley, have you tested with the included attachement (real file anonymized ) ?

Thanks.
Comment 14 Bryan Quigley 2014-10-09 15:24:54 UTC
Just tested with "test file with 40000..."

latest build
MME -> MMES - 20seconds
MME -> nothing - 1 min

4.3.2.2 (w Ubuntu 14.04)
MME -> MMES - 20seconds to results window, then unresponsive for >2 minues (gave up waiting)
MME -> nothing 0 1 min to results window, then unresponsive for > 2 minutes (gave up waiting)

Looks good :)

@Nicolas That the same date build as the Ubuntu version I used which had the above fixes.
Comment 15 Nicolas R 2014-10-09 16:43:36 UTC
@Bryan thanks for your fast response.

So my tests with test_file in attachment :
Win 7 pro 64 bits I7 8Go Ram 

4.1.6 
MME -> MMES : find all 3 sec, then replace all 3.75 sec

latest build ( see comment 13 )

MME -> MMES : Find all : search results windows in 4.5 sec, then unresponsive during  around 3 sec then ok
 Replace all with the remaining / resulting selection : like find all => replace results window in 4.5 sec then unresponsive around 3 sec then ... ok

Tested with the original file (40000 real adresses) , same performances.

Perhaps an optional 'result windows' could still improve performance and would be 'the icing on the cake' ...
But for me ( and for my use), it's totally usable as is ( and result window could be useful)

Could this patch be backported to 4.3 branch ?

Good patch by Seyeong Kim, and good work by all the team !

Many thanks
Comment 16 pgoin 2014-11-14 11:45:11 UTC
same problem 4.3.4 release on win 7 PRO 64
search and replace freeze
Comment 17 Adolfo Jayme Barrientos 2014-11-18 10:22:35 UTC
Let’s close this as FIXED as per comment 12.
Comment 18 khagaroth 2014-11-18 16:11:48 UTC
I wouldn't call 5 times slower fixed.
Comment 19 andis.lazdins 2014-11-18 16:19:44 UTC
(In reply to khagaroth from comment #18)
> I wouldn't call 5 times slower fixed.
 
I agree to that. The source of the problem is unclear and so much worse result in compare to situation before needs  more attention.

However, thank you very much for this temporary solution!
Comment 20 Jean-Baptiste Faure 2014-11-19 05:33:02 UTC
Please, don't change version number. Set back to previous values.

That said it works very well for me with Version: 4.4.0.0.alpha2+
Build ID: 47659b4744467ffda504116a1284c7f096c1b326 built at home under Ubuntu 14.10 x86-64.

Best regards. JBF
Comment 21 pgoin 2014-11-19 09:04:21 UTC
Hi 
Work fine with Version: libreofficedev 4.4.0.0.alpha2
windows 7 64 bit
best regard
Comment 22 Yousuf Philips (jay) (retired) 2015-01-02 13:16:45 UTC
As this is on the 4.3 MAB list, was this also backported to 4.3 as asked by Nicolas in comment 15?
Comment 23 Robinson Tryon (qubit) 2015-12-17 08:34:02 UTC Comment hidden (obsolete)