Bug 93433 - incomplete display of long autocorrect list in the replacement table and possible data loss when saving
Summary: incomplete display of long autocorrect list in the replacement table and poss...
Status: RESOLVED WORKSFORME
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: UI (show other bugs)
Version:
(earliest affected)
4.4.0.3 release
Hardware: Other Windows (All)
: high normal
Assignee: Not Assigned
URL:
Whiteboard:
Keywords: regression
Depends on:
Blocks: AutoCorrect-Complete
  Show dependency treegraph
 
Reported: 2015-08-14 18:19 UTC by tommy27
Modified: 2018-05-27 12:33 UTC (History)
3 users (show)

See Also:
Crash report or crash signature:


Attachments
long autocorrect list (more than 126K entries) (1.17 MB, application/zip)
2015-08-14 18:19 UTC, tommy27
Details
screenshot (32.73 KB, image/png)
2015-08-14 18:23 UTC, tommy27
Details
Kontol (deleted)
2017-10-03 17:55 UTC, Abenk
Details
Screenshot of Autocorrect dialog (49.08 KB, image/png)
2017-11-01 10:40 UTC, Buovjaga
Details
even longer autocorrect list (more than 217K entries) (4.17 MB, application/zip)
2017-11-03 04:55 UTC, tommy27
Details
wrong display of autocorrect list just moving up/down the scrollbar (383.95 KB, image/jpeg)
2017-11-04 05:41 UTC, tommy27
Details

Note You need to log in before you can comment on or make changes to this bug.
Description tommy27 2015-08-14 18:19:00 UTC
Created attachment 117921 [details]
long autocorrect list (more than 126K entries)

tested under Windows 7 x64 and Windows 8.1 x64

STEPS TO REPRODUCE

1- put the attached long autocorrect list (acor_it-IT.dat which has more than 126000 entries) inside your user profile under ...User\LibreOffice 4\user\autocorr

2- then click “Tools/AutoCorrect/AutoCorrect Options/Replace” and select Italian language 

3- scroll down to the bottom of the list... the last entry should be “zzampe → zampe” 

4- then click OK and repeat it over and over again 
  
  
CURRENT BEHAVIOUR  

LibO 4.3.5.2 
Always loads the whole “A to Z” list. 
Editing entries and adding new ones works fine.

LibO 4.4.0.3 → 4.4.5.1 
Inconstantly loads the whole list and sometimes (1 out of 5 tries in my experience) only entries from “A to I” are displayed (see screenshot). If you edit an entry or add a new one and click OK, only the displayed entries will be saved and the undisplayed entries (L to Z) will be lost in next sessions (you can also see the acor_it-IT.dat filesize shrinking because of the data loss).

Workaround is to enlarge the replacement table dragging its bottom right corner before adding new items... after that you'll see the incompletely loaded list refreshing and displaying the missing items. 
Editing and saving after this workaround will cause no data loss.

LibO 5.0.0.4 
Inconstantly loads the complete list as 4.4.x but saving items in an incompletely loaded list won't cause data loss... the undisplayed items will be preserved and accessible on the next session.

LibO 5.1.0.0 alpha 
never displays the “A to Z” list and always shows items from “A to I”.
anyway, like in 5.0.x, there's no data loss after editing or adding a new entry. 
Even the undisplayed items will be still present.


BOTTOMLINE

There's a regression bug in the display of long autocorrect list... which is inconstant in 4.4.x and 5.0.x but constant in 5.1.x

The 4.4.x is also affected by a data loss bug when editing an incompletely loaded list.

Luckily this data loss bug is not present in 5.0.x and 5.1.x
Comment 1 tommy27 2015-08-14 18:23:01 UTC
Created attachment 117922 [details]
screenshot

screenshot showing normal and abnormal loading of the same long autocorrect list
Comment 2 Buovjaga 2015-09-08 09:46:46 UTC
I get to zampe just fine and even to ømassimo.
However, if I first go to Italy (Switzerland) and then to Italy (Italy), LibO hangs.

Win 7 Pro 64-bit, Version: 5.0.1.2 (32-bit)
Build ID: 81898c9f5c0d43f3473ba111d7b351050be20261
Locale: fi-FI (fi_FI)
Comment 3 tommy27 2015-09-09 07:56:44 UTC
you have to reload the list multiple times consecutively since the incomplete loading is inconstant

anyway I still reproduce the bug with LibO 5.0.1.2 under Win7x64
Comment 4 tommy27 2015-10-27 06:25:29 UTC
(In reply to tommy27 from comment #0)
> ....
> 
> BOTTOMLINE
> 
> There's a regression bug in the display of long autocorrect list... which is
> inconstant in 4.4.x and 5.0.x but constant in 5.1.x
> 
> The 4.4.x is also affected by a data loss bug when editing an incompletely
> loaded list.
> 
> Luckily this data loss bug is not present in 5.0.x and 5.1.x

I'm sorry to tell that data loss affects LibO 5.0.3.1 as well.
It doesn't happens so frequently as in LibO 4.4.5.2 but I was able to reproduce a data loss at least 3 times.
Comment 5 tommy27 2015-11-06 06:38:13 UTC
a new symptom of this is bug is that when you enter the replacement table list and you notice that after scrolling down the list is not fully loaded, if you then scroll up and down fast multiple times you see different degrees of list loading... 

sometime you see the whole list, sometimes some more entries and sometimes fewer entries...

this is different from the "resize table" workaround described in comment 0

it seems there's a problem in populating the table in a constant matter
Comment 6 tommy27 2015-11-22 11:12:28 UTC
just to tell that the data loss problem has happened a few times weven after adding a new autocorrect entry from the right click autocorrect suggestions menu without entering the replacement table

cannot find a consistent way to reproduce it, but it happened even if less frequently than the replacement table data loss
Comment 7 tommy27 2015-12-10 06:31:52 UTC
CC'ing Marina
can you please try to reproduce this one under Linux and/or Windows?
Comment 8 Valter Mura 2015-12-20 13:17:27 UTC
Hi Tommaso

have you checked in mailing list if there is a program's size limit for this file?
Comment 9 tommy27 2015-12-20 15:42:19 UTC
no there's not a size limit problem
if that was the case the data loss would be constant after a certain number of autocorrect entries

I'm doing constant backups of that files and once it's incompletely saved I restore the latest version and keep adding new entries to that one, then after some time I get a new data loss and the thing keeps repeating over and over again...

the bug happens randomly... let's say I notice it 2-3 times a week

it seems to me that there's a filesave issue of the autocorrect list.
sometimes it's saved incompletely, don0t know why
Comment 10 tommy27 2016-04-20 06:04:50 UTC
still happening with LibO 5.1.2.2
Comment 11 tommy27 2016-10-05 07:36:23 UTC
still present in LibO 5.1.5.2
the data loss issue however happens less frequently than in previous releases.
in the past it happened frequently (2 times a week) but now 1 or 2 times in a month.
Comment 12 tommy27 2016-11-19 08:31:12 UTC
additional infos from last time

bug is still present in LibO 5.2.3.3

it happens more frequently under Win8.1 x64 rather than Win 7 x64 (I have the same cloned user profile running on both machines... both have powerful processors and SSD disks so it's not a matter of hardware power).

sometimes I noticed that the loss of autocorrect entries happens just after adding a new entry from right click suggestions menu and the document was auto-saved soon after... so maybe there's some sort of collision between auto-save and storing of the changes of the autocorrect .dat file.
Comment 13 tommy27 2017-04-22 12:18:56 UTC
still affecting LibO 5.3.2.2 as well
Comment 14 tommy27 2017-07-15 13:52:29 UTC
still present in LibO 5.3.4.2
still more frequent under Win8.1 x64 rather than Win7 x64, even using the same user profile on both machines.
Comment 15 Xisco Faulí 2017-07-15 14:23:14 UTC
Could you try whether it has been fixed by https://cgit.freedesktop.org/libreoffice/core/commit/?id=94c7a401583200cf5982594b1b043ad1a5e3cd38 with a master build from http://dev-builds.libreoffice.org/daily/master/ ?
Comment 16 tommy27 2017-07-17 12:10:34 UTC
tried latest LibO 6.0.0.0 alpha build.

the AutoCorrect replacement table now correctly displays the entire list... anyway there's now a more important regression..

see my report here at Bug 109156
Comment 17 Buovjaga 2017-07-17 12:16:57 UTC
Thanks, setting to fixed.
Comment 18 tommy27 2017-07-17 12:33:52 UTC
well, the "display" issue is certainly fixed in 6.0.x but I can't tell if the "data loss" issue is resolved as well.

because of bug 109156 I can't save new entries so I cannot tell if there's data loss or not... please also note that the "data loss" thing didn't happen all the time.
Comment 19 Buovjaga 2017-07-17 12:36:37 UTC
Ok, let's wait some more, then.
Comment 20 Xisco Faulí 2017-07-17 12:49:34 UTC
(In reply to tommy27 from comment #18)
> well, the "display" issue is certainly fixed in 6.0.x but I can't tell if
> the "data loss" issue is resolved as well.
> 
> because of bug 109156 I can't save new entries so I cannot tell if there's
> data loss or not... please also note that the "data loss" thing didn't
> happen all the time.

Since this bug is about the displaying, it's correct to close it as RESOLVED WORKSFORME. The data loss is addressed in bug 108835
Comment 21 tommy27 2017-07-17 13:14:17 UTC
(In reply to Xisco Faulí from comment #20)
> (In reply to tommy27 from comment #18)
> ...
> Since this bug is about the displaying, it's correct to close it as RESOLVED
> WORKSFORME. The data loss is addressed in bug 108835

@Xisco
no... the bug 108835 reported by Cor was a completely different issue than the one I reported here.

anyway, let's keep this one as RESOLVED regarding the "incomplete display" thing...

I'll open a new clean bug report about the "data loss" I experienced and described in my initial report after more testing with 5.4.x and 6.0.x
Comment 22 tommy27 2017-07-29 04:55:57 UTC
setting it back to UNCONFIRMED

apparently something changed in latest builds since in 5.4.1.x and 6.0.0.x updated right now the incomplete list display list is back to the original buggy behaviour.
Comment 23 tommy27 2017-07-30 12:13:24 UTC
incomplete display of long autocorrect list still happens in:
LibO 5.4.1.0.0+ (x64)
Build ID: b78c398817940bbbe0b9ebe848a923cef1856758
CPU threads: 4; OS: Windows 6.29; UI render: default; 
TinderBox: Win-x86_64@62-TDF, Branch:libreoffice-5-4, Time: 2017-07-28_09:02:44
Locale: it-IT (it_IT); Calc: group

even the random data loss issue is still reproducible.
Comment 24 tommy27 2017-08-03 05:31:27 UTC
bugs exists in LibO 5.4.0.3 as well.
Comment 25 tommy27 2017-09-23 12:13:12 UTC
incomplete display of autocorrect items is present in LibO 6.0.0.0.alpha0+
Build ID: 31919b8909fa7b34412dd52c3d4dff17bc5b6fab
CPU threads: 4; OS: Windows 6.29; UI render: default; 
TinderBox: Win-x86@42, Branch:master, Time: 2017-09-22_00:51:46
Locale: it-IT (it_IT); Calc: group

tested under Win8.1 x64
Comment 26 Abenk 2017-10-03 17:55:47 UTC Comment hidden (obsolete)
Comment 27 tommy27 2017-10-04 04:51:47 UTC Comment hidden (obsolete)
Comment 28 Buovjaga 2017-10-16 18:42:39 UTC Comment hidden (obsolete)
Comment 29 Ivan 2017-10-30 21:22:15 UTC
Thank you for reporting the bug. I can not reproduce the bug in  5.4.3.1 (x64).
Comment 30 tommy27 2017-10-31 03:05:12 UTC
did you use the long replacement table I attached?
I can still reproduce it in 5.4.1 and 5.4.2 too under Win7x64
Comment 31 Buovjaga 2017-10-31 19:11:34 UTC
(In reply to tommy27 from comment #30)
> did you use the long replacement table I attached?
> I can still reproduce it in 5.4.1 and 5.4.2 too under Win7x64

Back in comment 2 I could not repro with Win 7. Now I tried with Win 10 and repeated the loading 10 times, each time closing and restarting LibO: not reproduced.
The last one displayed is always ømassimo.

Version: 6.0.0.0.alpha1+ (x64)
Build ID: 7e03c4eed72452fdfb87341214a21956c08ba969
CPU threads: 4; OS: Windows 10.0; UI render: default; 
TinderBox: Win-x86_64@42, Branch:master, Time: 2017-10-26_00:58:29
Locale: fi-FI (fi_FI); Calc: group
Comment 32 tommy27 2017-10-31 19:43:51 UTC
(In reply to Buovjaga from comment #31)
> (In reply to tommy27 from comment #30)
> > did you use the long replacement table I attached?
> > I can still reproduce it in 5.4.1 and 5.4.2 too under Win7x64
> 
> Back in comment 2 I could not repro with Win 7. Now I tried with Win 10 and
> repeated the loading 10 times, each time closing and restarting LibO: not
> reproduced.
> The last one displayed is always ømassimo.
> 
> Version: 6.0.0.0.alpha1+ (x64)
>  ...

I think you reproduced the data loss part of the bug.
the whole autocorrect list should end with "zzampe" not "ømassimo".

this implies that the filed failed to load properly and probably was cut in the middle with data loss when reloaded.  now since the file got smaller you don't reproduce the incomplete load.

try comparing the file size of the original  attachment 117921 [details] size and the autocorrect file size you have now in the user profile.
Comment 33 Buovjaga 2017-10-31 19:57:00 UTC
(In reply to tommy27 from comment #32)
> try comparing the file size of the original  attachment 117921 [details]
> size and the autocorrect file size you have now in the user profile.

It's the same size, same date..
zzampe comes before ømassimo, nothing is cut off.
Comment 34 tommy27 2017-11-01 02:52:28 UTC
I still reproduce the issue under Win8.1 x64 using 6.0.0.0.alpha1+ 
nothing changed in my environment from the original submission from august 2015

Build ID: 8ea346b87c8f62d93bec283515abae8db36a08ed
CPU threads: 4; OS: Windows 6.3; UI render: default; 
TinderBox: Win-x86@42, Branch:master, Time: 2017-10-31_23:37:25
Locale: it-IT (it_IT); Calc: group(In reply to Buovjaga from comment #33)


> (In reply to tommy27 from comment #32)
> > try comparing the file size of the original  attachment 117921 [details]
> > size and the autocorrect file size you have now in the user profile.
> 
> It's the same size, same date..
> zzampe comes before ømassimo, nothing is cut off.

there's something strange about your result...
check my screenshot in attachment 117922 [details] 

the last entry is "zzampe"
the entry list is showed in alphabetical order so how can it be "ømassimo" ? 

please post a screenshot too.
Comment 35 Buovjaga 2017-11-01 10:40:34 UTC
Created attachment 137413 [details]
Screenshot of Autocorrect dialog
Comment 36 tommy27 2017-11-01 21:02:55 UTC
very strange... how can the same file be sorted in a different alphabetic order in 2 different computers?

maybe you have a different locale setting in which the ø is shown after the z...

in my computer the "ø" entries are at the the top of the list and the last are the "z" ones.
Comment 37 tommy27 2017-11-03 04:55:11 UTC
Created attachment 137490 [details]
even longer autocorrect list (more than 217K entries)

I'm attaching an heavier version of my huge autocorrect list (more than 217K entries) that may help to further stress the autocorrect display engine.

the first entry should be "___z"
and the last one "zzzoticoniii"

this was tested with this LibO interface locale setting (it-IT).
Comment 38 Buovjaga 2017-11-03 13:26:24 UTC
(In reply to tommy27 from comment #37)
> Created attachment 137490 [details]
> even longer autocorrect list (more than 217K entries)
> 
> I'm attaching an heavier version of my huge autocorrect list (more than 217K
> entries) that may help to further stress the autocorrect display engine.
> 
> the first entry should be "___z"
> and the last one "zzzoticoniii"
> 
> this was tested with this LibO interface locale setting (it-IT).

Yep I see zzzoticoniii and then the ø entries start.
Comment 39 tommy27 2017-11-04 05:41:04 UTC
Created attachment 137513 [details]
wrong display of autocorrect list just moving up/down the scrollbar

In reply to Buovjaga from comment #38)
> (In reply to tommy27 from comment #37)
> > ....
> > 
> > the first entry should be "___z"
> > and the last one "zzzoticoniii"
> > 
> > this was tested with this LibO interface locale setting (it-IT).
> 
> Yep I see zzzoticoniii and then the ø entries start.

please tell me which is your "locale setting" there's obviously a different alphabetical sorting method between my PC and yours.

moreover check my screenshots... this is how the top and bottom of the list are displayed just moving the scrollbar up/down from top to bottom multiple times...

sometimes you don't see the correct end of the list and sometimes neither the top of if.

did you try doing this up/down thing many times in your PC of did you only try once to scroll to bottom? 

this was tested with latest daily build:
6.0.0.0.alpha1+
Build ID: 8ce49eab696d830d420fbf48b22ac151167bbd62
CPU threads: 4; OS: Windows 6.3; UI render: default; 
TinderBox: Win-x86@42, Branch:master, Time: 2017-11-03_23:12:37
Locale: it-IT (it_IT); Calc: group
Comment 40 Buovjaga 2017-11-04 13:34:56 UTC
Well, now I tried scrolling many times and it was OK.

Locale is fi-FI (fi_FI) like seen in comment 31.
Comment 41 tommy27 2017-11-06 06:53:51 UTC
strange... I easily and constantly reproduce the issue on 4 different computers (3 Win7 x64 and 1 Win8.1 x64 machine).

the other thing that I don't understand is the different position of the entries...

how alphabetical order works in Finland? I mean, in the italian locale symbols like "ø" are shown at the top of the list and then all the "a-z" entries.
Comment 42 tommy27 2017-11-12 16:29:19 UTC
@Buovjaga

please check this .xml file (https://www.dropbox.com/s/9gypljvqyrqbenc/DocumentList.xml?dl=0 ) ; it contains the whole autocorrect replacement list which is zipped together with other files in the .dat autocorrect file I attached before as attachement 137490

In this .xml the first entry is "___z" and the last one is "zzzoticonii"

it is the same order I see in the LibO GUI when I load the replacement table.

would you please check the .xml file in your computer and tell me if the internal content is the same? just check first and last entry...

If you see a different order in the GUI you should see the same in the .xml too.
Comment 43 Buovjaga 2017-11-12 17:01:25 UTC
(In reply to tommy27 from comment #42)
> would you please check the .xml file in your computer and tell me if the
> internal content is the same? just check first and last entry...
> 
> If you see a different order in the GUI you should see the same in the .xml
> too.

Check with what? Naturally it will be in the order you mentioned in a text editor no matter what computer I use to open it.
Comment 44 tommy27 2017-11-12 17:39:36 UTC
I suspect that in your locale the order is different because LibO changes the alphabetical order according to your locale.

maybe I'm wrong but I want to be sure about that.
Comment 45 Buovjaga 2017-11-12 17:57:49 UTC
(In reply to tommy27 from comment #44)
> I suspect that in your locale the order is different because LibO changes
> the alphabetical order according to your locale.
> 
> maybe I'm wrong but I want to be sure about that.

The order can be changed only by software like LibreOffice that parses the XML structure and does things based on it. A text editor will always display it in the original form (most of it in one huge line).
Comment 46 tommy27 2017-11-13 06:43:26 UTC
I know it diplays the list in a single huge line.
but according to my tests, if I edit the .xml file adding an entry which is not in alphabetic order and then load the file in LibO, it will be shown in correct alphabetic order and then if I add in the GUI a new entry and save it, then the whole .xml will be processed by LibO and saved in correct alphabetic order.

so if I reopen the .xml in a text editor the initial "out of place" entry will be now in correct place even in the text editor.

so please, add a new entry in LibO replacement table then send your .dat file to me at my email barta@quipo.it 
I'll do myself the inspection of the .xml file.

another question... how finnish grammar works? in your language is correct to show "ø" after "z" entries?

In italian all symbols are shown before "a" not after "z".
Comment 47 Buovjaga 2017-11-13 17:32:27 UTC
(In reply to tommy27 from comment #46)
> another question... how finnish grammar works? in your language is correct
> to show "ø" after "z" entries?

I don't know.

Sorry, but I don't think this sorting is relevant and this needs more Italian testers instead (I sent out a plea for testers).
Comment 48 tommy27 2017-12-27 07:33:07 UTC
incomplete load issue still present in LibO 6.1.0.0.alpha0+
Build ID: 075b5ad5bdd230738ee30b0ea17a7fa9502c218b
CPU threads: 4; OS: Windows 6.3; UI render: default; 
TinderBox: Win-x86@42, Branch:master, Time: 2017-12-22_00:40:12
Locale: it-IT (it_IT); Calc: group threaded
Comment 49 tommy27 2018-03-14 14:28:37 UTC
@ driss.zoubeir@gmail.com

this is not FIXED 
is still happens in 6.0.2 under Windows

reverting to UNCONFIRMED
Comment 50 tommy27 2018-05-27 12:33:52 UTC
retested with LibO 6.0.4.2

finally I can tell that the incomplete display of the autocorrect list is gone if you scroll down at normal speed (it happens again if you croll up & down at crazy speed but it's not a normal user scenario)

most important, the random data loss when saving new autocorrect items seems finally gone.

this was very frequent under Win8.1 x64 using 5.3.7.2 but has never happened again with latest 6.0.4.2 which I tested in the last day.

so finally, RESOLVED WORKSFORME