Bug 46805 - large autocorrect database make LibO freeze when start typing
Summary: large autocorrect database make LibO freeze when start typing
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Linguistic (show other bugs)
Version:
(earliest affected)
3.5.0 release
Hardware: All All
: medium major
Assignee: Not Assigned
URL:
Whiteboard: target:3.6.0 target:3.5.4
Keywords:
Depends on:
Blocks: mab3.5
  Show dependency treegraph
 
Reported: 2012-02-29 22:52 UTC by tommy27
Modified: 2014-11-15 07:27 UTC (History)
5 users (show)

See Also:
Crash report or crash signature:


Attachments
large autocorrect database (393.08 KB, application/octet-stream)
2012-02-29 22:52 UTC, tommy27
Details
LibO start typing without acor_.dat (153.72 KB, video/bar)
2012-04-13 11:50 UTC, tommy27
Details
LibO start typing with acor_.dat (281.86 KB, video/bar)
2012-04-13 11:58 UTC, tommy27
Details
kcachegrind output ... (496.84 KB, image/png)
2012-04-26 03:18 UTC, Michael Meeks
Details

Note You need to log in before you can comment on or make changes to this bug.
Description tommy27 2012-02-29 22:52:58 UTC
Created attachment 57846 [details]
large autocorrect database

this is a different issue than:
https://www.libreoffice.org/bugzilla/show_bug.cgi?id=46765
that only affect LibO 3.5.0, while the current bug has been there since the OOo era...

it seems that large autocorrect databases male LibO freeze when start typing.
 
try this: enter the autocorr subfolder in your user profile (in Windows is under: User\LibreOffice 3\user\autocorr)
and put the attached acor_.dat file (backup yours first) which has 64000 entries inside it.
 
start LibO and open a blank writer file.
digit "test" and hit space...
 
you will see a first 2-3 second freeze after "tes" the the cursor moves to "test" and another longer freeze (6 seconds maybe) happens before you see the "space"
 
so there are 2 freeze moments, the first one is shorter and unrelated to autocorrection (bug 46765... it's present even with no autocorrect database) while the second is longer and depends on autocorrection.

both freeze moment happen just at first start, so it will annoy you only once when you first launch LibO, but give a bad impression about LibO performace
 
these tests were done on LibO 3.5.0
Windows Vista 64bit SP1
IntelCore2 Duo CPI P8400 @2.26 GhZ, 4GB RAM

but can be reproduced even on Windows XP.
Comment 1 Szabolcs Dézsi 2012-03-01 04:44:18 UTC
Hello!

I tested it with the patch which reduced loadtime of autocorrection tables. The first freeze occured just as you described (although it was 1 sec max), there was no freeze before space.
Comment 2 tommy27 2012-03-01 13:20:49 UTC
great!!! as I hoped, your path fixed both things!!

thank you so much... I'm struggling with those  autocorrect issues since years!!!

I'm going to mark this bug as fixed
Comment 3 tommy27 2012-03-01 22:26:20 UTC
just forgot to say that Dezsi's patch fixes this old OOo issue I opened in May 2009:

Issue 101726 - Speed up autocorrect replacement table loading time
https://issues.apache.org/ooo/show_bug.cgi?id=101726

Great to see that LibO folks solved an issue that OOo never seriously tried to fix.
Comment 5 tommy27 2012-04-09 06:56:29 UTC
mmhhh.... it seems that the "huge database" related typing freeze is not gone yet... I still experience this in LibO 3.5.1 and 3.5.2...

I thought that dezsi's patch which fixed the old "OOo autocorrect replacement table loading time issue", had an impact on the current issue as well...

he told in Comment 1 that it did, so I closed the issue...
but now I'm going to reopen since it's still here...

if I put the previously attached huge acor_.dat file in my "User\LibreOffice 3\user\autocorr" folder, the freeze happens

if I remove it, the freeze does not happen again.

tested with LibO 3.5.2 on Vista 64bit.
Comment 6 tommy27 2012-04-13 11:50:52 UTC
Created attachment 59932 [details]
LibO start typing without acor_.dat

this is a fresh install of portable LibreOffice 3.5.2 
(downloaded from http://www.winpenpack.com/main/download.php?view.1338)

the user profile is clean
no autocorrect file database is present

once an empty Writer document is opened, there's a short freeze while typing the first word (i.e. "test")

the freeze happens between "te" and "st"

following words are written with no freeze

this is Bug 46765
Comment 7 tommy27 2012-04-13 11:58:21 UTC
Created attachment 59934 [details]
LibO start typing with acor_.dat

this is the same fresh install of portable LibreOffice 3.5.2 as before

the user profile is the same except for the presence of a large autocorrect file database (acor_.dat - see first attachment - containing more than 64000 entries)

same test as before, after the first small freeze between "te" and "st", you will see a much longer freeze after "test" and before the cursor moves to the "space" while following words are again written with no freeze.

this shows that at document start the first access to a large autocorrect database has impact on LibO performance
Comment 8 tommy27 2012-04-13 12:03:24 UTC
the previous tests were done on Windows Vista 64bit but have been also reproduced in Windows XP 32 bit.

I ask for Linux and Mac users to try reproducing the scenario using the attached large autocorrect database (acor_.dat) whose Windows path is: 

User\LibreOffice 3\user\autocorr
Comment 9 tommy27 2012-04-13 12:09:14 UTC
I report here a private mail from Michael Meeks about this bug, giving instructions to debug:

So - then we need a profile of the piece that is slow; do you have a
Linux machine you can repeat this on ?

        If so, can you run libreoffice under callgrind like this:

  export OOO_DISABLE_RECOVERY=1
  valgrind --tool=callgrind --simulate-cache=yes --dump-instr=yes ./soffice.bin -writer --splash-pipe=0

         Select your dictionary (beware it runs 80x slower than normal or so ;-).

         Then you'd want to gzip up the callgrind.out.... and attach it to the
 bug - that'll help the developers ~immediately fix the problem, and save
 them lots of time.

The fix is prolly only a few minutes if we have that data, and it's
 more likely to get done. A linux VM would be fine to do the profiling.

..........

unfortunately I have no experience with Linux so I can not run this text.
is there any "penguin" right here that may help?
Comment 10 Korrawit Pruegsanusak 2012-04-25 11:15:23 UTC
I tried valgrind as in comment 9, but the callgrind.out.XXXX is empty (0 bytes) after exit libo. Am I missing something?

Ubuntu 64 bit libreoffice 3.5.2.2 installed from ppa:libreoffice/ppa, with libreoffice-dbg installed.
I tried, and the empty file callgrind.out.XXXX created every times (4 times currently), even if I just started Writer and close, doing nothing. Console window (Terminal) says 'Segmentation fault' after exit libo.
Comment 11 tommy27 2012-04-25 11:27:25 UTC
hi Korrawit. thanks 4 testing.

I'm just curious to know if you experience on Linux the freeze I described on Windows
Comment 12 Michael Meeks 2012-04-26 01:19:49 UTC
> I tried valgrind as in comment 9, but the callgrind.out.XXXX is
> empty (0 bytes) after exit libo. Am I missing something?

Thanks for trying that ! you're sure you used 'soffice.bin' and not 'soffice' ? do you have write access to the directory that soffice.bin is in (prolly best to give that to yourself and run it from there).

Also - you need to ensure no other soffice.bin is running or the factory behaviour will defeat you so perhaps 'pkill -9 -f soffice' beforehand (?).

What else ? Can you run './soffice.bin' successfully yourself without valgrind ? if not perhaps we need to run 'soffice' with --trace-children=yes as another argument to valgrind.

Thanks so much for trying to get a trace !
Comment 13 Korrawit Pruegsanusak 2012-04-26 02:03:29 UTC
Forgot to mention in comment 10 that I use ubuntu 10.04 64bit.

@tommy27: Yes, I see the freeze on ubuntu too, so set platform to All.

@michael: Thanks for your suggestions :) Anyway, I just try again with ubuntu 12.04 32bit, and it seems to work. So I don't know which part(s) that trigger zero-length-file problem.
Please see next comment for the callgrind log.
Comment 14 Korrawit Pruegsanusak 2012-04-26 02:13:57 UTC
Well, attachment is too large! So, http://www.mediafire.com/?6mi1gypc666rinr

Hope it'll be useful, and feel free to blame me if I've done something wrong. :)
Comment 15 Michael Meeks 2012-04-26 03:18:55 UTC
Created attachment 60609 [details]
kcachegrind output ...

Korrawit - you're a hero :-) this is exactly what we need. It's a nice test, 55% of the measured time is in this code, so it's easy to see; and I attach a photo of the calltrace where all the time is being consumed.

Hopefully a fix will just drop out of that.
Comment 16 Michael Meeks 2012-04-26 04:01:36 UTC
I have a rather simple prototype patch; just re-building to test it.
Comment 17 Michael Meeks 2012-04-26 04:58:23 UTC
The patch fixes the N^2 in the insert/sort - unfortunately that only saves around 1/2 the time - the rest seems to be some diffuse wasteage of a reasonably normal and fairly unbelievable kind (harder to fix).

This takes me down from 1200ms to load the list to 700ms or so - which should be better. Give it some testing, and if you're happy I'll get some review / cherry-picking action to -3-5.

Ultimately this code needs a re-write to use STL anyway so ... :-) I'd prefer to wait for / encourage that: perhaps adding this specific instance of:

editeng/source/misc/svxacorr.cxx and it's 'SvxAutocorrWordList' to the STL porting EasyHack might help if you'd like to dig that out.
Comment 18 Not Assigned 2012-04-26 05:01:03 UTC
Michael Meeks committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=b269dced445494891a8e6e8d1d62b931a31dddbd

fdo#46805 - special-case appending items to autocorrect lists
Comment 19 tommy27 2012-04-26 07:32:11 UTC
thanks Micheal. I like to test the build with the fix. where can I download it?

P.S. do you think is safe to be cherry-picked in 3.5.3 or 3.5.4?
Comment 20 Korrawit Pruegsanusak 2012-04-26 07:54:10 UTC
Thanks Michael too :)

tommy27: You're on Windows, right? If yes, try one in <http://dev-builds.libreoffice.org/daily/Win-x86@6-fast/master/>, but you'll have to wait for today builds.

/me also clear out irrelevant target in whiteboard
Comment 21 tommy27 2012-04-26 08:27:32 UTC
thanks Micheal. I like to test the build with the fix. where can I download it?

P.S. do you think is safe to be cherry-picked in 3.5.3 or 3.5.4?
Comment 22 tommy27 2012-04-27 23:32:34 UTC
ok, I downloaded the:
master~2012-04-27_21.25.23_LibO-Dev_3.6.0alpha0_Win_x86_install_en-US.msi

and made comparative testings.

- start LibO
- open empty Writer document, italian language
- digit "test" and hit "space"

here's the results.

1 large autocorrect database (acor_.dat)
time to see the white space

LibO    3.5.2 --> 9 seconds
LibOdev 3.6.0 --> 5 seconds

2 large autocorrect databases (acor_.dat ; acor_it-IT.dat)
time to see the white space

LibO    3.5.2 --> 14 seconds
LibOdev 3.6.0 -->  9 seconds


as Michael said, his fix reduced the freeze time about 50% which is a good result after all.

maybe it's too late for 3.5.3.
could it be considered for 3.5.4?
Comment 23 Not Assigned 2012-05-01 05:46:07 UTC
Michael Meeks committed a patch related to this issue.
It has been pushed to "libreoffice-3-5":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=a617eecdb6b2f8d6bb1b20674f8be8ce4d60f2d1&g=libreoffice-3-5

fdo#46805 - special-case appending items to autocorrect lists


It will be available in LibreOffice 3.5.4.
Comment 24 tommy27 2012-05-01 06:34:53 UTC
just wanted to say a final thank you to Micheal and Korrawit
Comment 25 Michael Meeks 2012-05-01 06:45:32 UTC
> just wanted to say a final thank you to Micheal and Korrawit

No problem - Korrawit was the man with the profile ;-)

Having said that - we should build another profile, of the code with the fix; and try to work out where/why it is still quite so bad, there is simply no excuse for a modern CPU to take -that- long doing anything; but it needs some staring at kcachegrind & playing with it I think.

Why we try to keep these autocorrection entries in this specific order, I have no idea - I would have thought a hash-table would have been by far the best - at least until we have to present a list of them in the GUI.
Comment 26 tommy27 2012-10-02 19:58:49 UTC
mmmhhh... it seems that performance in LOdev 3.7 declined versus current 3.6 stable releases... so I'm REOPENING this bug.

my tests say that the freeze time after typing the first word in a Writer document become longer in LOdev 3.7

--------------------
LibO 3.6.1 
--------------------
acor_.dat (65K entries)
5 seconds

acor_.dat 65K entries + acor_it-IT.dat 55K entries
9 seconds

--------------------
master 3.7.x
--------------------
acor_.dat 65K entries 
13 seconds 

acor_.dat 65K entries + acor_it-IT.dat 55K entries
20 seconds 

acor_.dat 65K entries + acor_it-IT.dat 119 entries *
26 seconds 

* this test can't be done in 3.6.1 since only master currently supports autocorrect databases larger than 65K


I suspect that the longer freeze of LOdev 3.7 is a side effect of the new "32-bit autocorrect databases" introduced by this committ: 

http://cgit.freedesktop.org/libreoffice/core/commit/?id=78a39502de36c32d7aaf6e5bbde1f8df80fdd21f

so I'm adding Tomaz Vajngerl to the CC List, since he's the man who's doing a lot of work fixing autocorrect performance issues, and maybe he's could find a solution in this area he already knows very well

@ Tomaz

if you read Comment 17 and Comment 25 from Micheal Meeks, you'll see that he saw room for improvement for his partial 3.5.4 fix
Comment 27 Korrawit Pruegsanusak 2012-10-03 04:41:32 UTC
(In reply to comment #26)
> mmmhhh... it seems that performance in LOdev 3.7 declined versus current 3.6
> stable releases... so I'm REOPENING this bug.

That sounds bad!

> I suspect that the longer freeze of LOdev 3.7 is a side effect of the new
> "32-bit autocorrect databases" introduced by this committ:

So, could you please check whether this is real ;) I mean, do you have daily builds before and after this commit? If yes, you should check if the commit actually decrease speed. Or the decrease comes from another commit.
Comment 28 Michael Meeks 2012-10-03 10:33:28 UTC
Guys - this is a most-annoying bug for 3.5.

I would strongly prefer not to mangle the history of this bug, and the stats for 3.5 by moving all these around. If there is a -new- regression in 3.7 - please create a new bug for that, and link to this one rather than re-opening an old one and confusing the version picture :-)

As a one-off, just for you I've filed bug#55570 - which is also tracked vs. the 3.7 regression tracker. It'd be great if someone could check that that has all the right links, and information necessary to work on this.