Bug 52022 - CONFIGURATION: Part of data in userdir is lost on 3.5 -> 3.6 upgrade
Summary: CONFIGURATION: Part of data in userdir is lost on 3.5 -> 3.6 upgrade
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Installation (show other bugs)
Version:
(earliest affected)
3.6.0.1 rc
Hardware: Other All
: high critical
Assignee: Stephan Bergmann
URL:
Whiteboard: target:3.7.0 target:3.6.3
Keywords:
: 53049 53447 53496 53587 53638 54590 (view as bug list)
Depends on:
Blocks: 43489 mab3.6
  Show dependency treegraph
 
Reported: 2012-07-12 17:30 UTC by Rainer Bielefeld Retired
Modified: 2012-11-24 08:52 UTC (History)
14 users (show)

See Also:
Crash report or crash signature:


Attachments
List of files affected by this issue (2.90 KB, text/plain)
2012-08-30 10:02 UTC, Mirosław Zalewski
Details
part of registrymodifications.xcu that has not been migrated from 3.5 to 3.6 (190.09 KB, text/plain)
2012-08-30 10:04 UTC, Mirosław Zalewski
Details
tar.xz archive with userdirs from 3.5 and 3.6 (see README inside and comment #9 for details) (2.29 MB, application/x-xz)
2012-08-30 10:05 UTC, Mirosław Zalewski
Details
userdir of version 3.5.7 that is not upgraded to 3.6.3.3 (259.08 KB, application/x-zip)
2012-11-02 12:16 UTC, Winfried Donkers
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Rainer Bielefeld Retired 2012-07-12 17:30:19 UTC
I found that with normal Installation of  "LibreOffice 3.6.0.1 rc  English UI/Locale [Build-ID: 73f9fb6] on German WIN7 Home Premium (64bit). In Menu 'Tools -> Macros > Organize -> Basic' The folder where my macros were stored in 3.5.5.3 stored is missing. 

Not a big problem, but irksome
Comment 1 Rainer Bielefeld Retired 2012-08-02 04:56:21 UTC
*** Bug 53049 has been marked as a duplicate of this bug. ***
Comment 2 Rainer Bielefeld Retired 2012-08-02 04:56:45 UTC
OS=All due to Dup
Comment 3 Winfried Donkers 2012-08-06 11:21:42 UTC
I confirm the problem with 3.6.0.4 Dutch (upgrading from 3.5.5.3) on Dutch Windows XP.

Probably related to this issue:
When upgrading from 3.5.5.3 to 3.6.0.4 imported BASIC macro libraries are not transferred, i.e. they have to be imported again.
Comment 4 gte.ejecutivo 2012-08-10 15:54:07 UTC
Upgrade from 3.5.5 to 3.6.0.4 (Win7 professional 64 bit) erases all "My Macros". As a consecuence, all my customization is lost. I would say it is a major issue.
Comment 5 Rainer Bielefeld Retired 2012-08-16 10:14:51 UTC
Again saw that effect for update from 3.5.6.2 to "LibreOffice 3.6.1.1  German UI/Locale [Build-ID:  4db6344] on German WIN7 Home Premium (64bit)
Comment 6 manj_k 2012-08-17 22:43:26 UTC
Installed LibO 3.6.1.1 rc (Build ID: 4db6344) over
LibO 3.5.6.2 (Build ID: e0fbe70-5879838-a0745b0-0cd1158-638b327)
on WinXP 32b.

My customized 'Module1.xba' has been reset to default 'Module1.xba';
severe loss of data in 'My Macros'.

[Path: ...\LibreOffice\3\user\basic\Standard\Module1.xba]
Comment 7 manj_k 2012-08-17 22:45:33 UTC
*** Bug 53638 has been marked as a duplicate of this bug. ***
Comment 8 ted bug 2012-08-19 09:19:01 UTC
i can confirm this happened to me too.
on mac osx 10.7.4 
upgrading to libreoffice 3.6.0.4, from i forgot what version (i always upgrade when there is a new stable version)

when i tried to use my macros in my toolbar, i had an error message saying

the following basic script could not be found
library: Standard
module: Module1
method: <mymethod>
location: application

i think this is a CRITICAL problem, a 23KB Module1.xba was replaced with a default 300Bytes Module1.xba
the toolbar icons for the macros i use were intact and not replaced.

luckily i had my backups so i restored Module1.xba and all was back to normal.
as a workaround/safety, I renamed copied all the standard module1.xba to a new library/module afterwards.
Comment 9 Mirosław Zalewski 2012-08-30 10:02:09 UTC
In fact, this issue is larger than just macros.

LibreOffice 3.6, when upgraded from 3.5, reverts some files in userdir with default ones. I am attaching list of affected files (files-that-differ.txt). There might be more than that, though.

Most configuration is held in registrymodifications.xcu file. On upgrade, LO does not migrate all settings from this file. Among lost ones are:
- language of UI
- user data (name, address etc.)
- measure unit setting (see #53587)
- info about default template
I am attaching file that shows which information were in 3.5, but are not migrated to 3.6 (removed-in-3.6.txt). Again, there might be more than that.

Finally, I am attaching tar.xz archive (LO-3.6-erases-user-data.tar.xz) that contains my userdir from 3.5 and from 3.6. See README file in archive for details. It might be useful while reproducing this bug and checking fixes (just copy content of user-3.5 into your userdir and launch LO 3.6).

BUT at least part of data is lost only on initial launch (first launch after upgrade). If I copy config/standard.soc from 3.5 to userdir after 3.6 was launched at least once, then LO will not overwrite my settings with defaults. This is most likely related to LastCompatibilityCheckID key in /org.openoffice.Setup/Office property (registrymodifications.xcu)

To sum things up:
- LO fails to migrate some user settings on upgrade from 3.5 to 3.6;
- this causes loss of data of various importance
- software upgrade MUST NOT cause loss of user data
Comment 10 Mirosław Zalewski 2012-08-30 10:02:54 UTC
Created attachment 66325 [details]
List of files affected by this issue
Comment 11 Mirosław Zalewski 2012-08-30 10:04:00 UTC
Created attachment 66326 [details]
part of registrymodifications.xcu that has not been migrated from 3.5 to 3.6
Comment 12 Mirosław Zalewski 2012-08-30 10:05:41 UTC
Created attachment 66328 [details]
tar.xz archive with userdirs from 3.5 and 3.6 (see README inside and comment #9 for details)
Comment 13 Mirosław Zalewski 2012-08-30 10:10:51 UTC
*** Bug 53587 has been marked as a duplicate of this bug. ***
Comment 14 Rainer Bielefeld Retired 2012-08-30 10:26:40 UTC
@Stephan:
I think this one might be related to some update registration problems you fixed during the last weeks?
Comment 15 Roman Eisele 2012-09-06 12:18:27 UTC
*** Bug 53447 has been marked as a duplicate of this bug. ***
Comment 16 Roman Eisele 2012-09-06 12:21:55 UTC
*** Bug 53496 has been marked as a duplicate of this bug. ***
Comment 17 Roman Eisele 2012-09-06 12:24:08 UTC
*** Bug 54590 has been marked as a duplicate of this bug. ***
Comment 18 Roman Eisele 2012-09-06 12:26:19 UTC
(In reply to comment #17)
> *** Bug 54590 has been marked as a duplicate of this bug. ***

Hint:
Please note that bug 54590 contains some additional information/attachments which could be _helpful_ for fixing this bug, especially a copy of the user’s "Module1.xba" file _before_ and another copy of his "Module1.xba" file _after_ the update to 3.6.
Comment 19 Craig 2012-09-10 02:46:26 UTC
Comment on attachment 66325 [details]
List of files affected by this issue

not sure if this is the right bug issue, but when I installed 3.6 over 3.5 I lost all of my macros.
Comment 20 Geoff 2012-09-11 23:07:02 UTC
Totally agree with comment #9. This is a MAJOR issue!

In my own case it caused the loss of 40+ macros I've built up over years of using OOo/LO, plus reset basic things like page setup (my default A4 went back to Letter, Centimetres to Inches, etc.) Fortunately I have backups and downgraded back to 3.5, but my advice for now would be "DON'T UPGRADE!"

I'm warning users too: http://is.gd/NFrK6D

To repeat Miroslaw's (#9's) final comment:
"Software upgrade MUST NOT cause loss of user data"
Comment 21 tommy27 2012-09-12 07:46:03 UTC
@ Geoff

In my experience, it was enough to copy the "basic" subfolder twice from the 3.5 user profile to the 3.6 to make the macros be recognized again.

It seems that first copy doesn't work while if you overwrite again old macros reappear
Comment 22 Roman Eisele 2012-10-04 16:09:20 UTC
Let me be brave (or rude, just as you want to see it ;-) and change the severity of this issue from “major” to “critical”. I know that this does not help anything to fix a bug; and I am not a fan of setting every bug to highest priority, either. But this bug

a) may cause important data loss for users who do not backup their user profile;
   this is IMHO critical;

b) gives us bad press, cf. e.g.

http://blogs.pcworld.co.nz/pcworld/tux-love/2012/09/be_wary_of_libreoffice_36.html

I know this is “only” a blog, but nevertheless I think headlines like “Be wary of LibreOffice 3.6!” are not good for the FLOSS world. By such own goals we help both AOO and MS Office.

If I can do anything to help with fixing this issue, please let me know; but I am just a simple-minded bug wrangler, and bound to Mac OS X (and Win XP), so I fear my possibilites are limited ...
Comment 23 Stephan Bergmann 2012-10-04 20:44:07 UTC
This is due to an unfortunate chain of things:

1  For LO 3.5, ec4f69493b50c15861b0cdcc290ecedd00efb51d "removed Simple Handles option" and 6ea8ea456cf5df267284278ecda42aa9b089a682 "Remove Large Handles option" remove use of some SimpleControlPoints and LargeControlPoints configuration properties.  However, the way those changes are made in sw/source/ui/config/usrpref.cxx, SwLayoutViewConfig::Commit when called will write nil values for those properties into user/registrymodifications.xcu.  (Only bbd638350fb83af2cadd85cdac2900de80bf7401 "remove gaps in options and reduce by two" later fixed that for LO 3.6.)

2  For LO 3.6, e8bb827571f540ac4af2247cb11239bb96876669 "Fixed cppheader.xsl nillable treatment" changes the nil-ability of many configuration properties that should never be nil in practice (i.e., their schemas already supply a non-nil default value) from the poorly chosen default "nil-able" to "not nil-able," as that makes using the newly introduced autogenerated C++ configuration access code less error-prone.

3  When configmgr reads an xcu file and reads a nil value for a property that cannot be nil, it abandons reading that file, effectively ignoring the rest of the file's data.

4  Whether a user/ directory has already been populated with default files is controlled by configuration property /org.openoffice.Setup/Office/ooSetupInstCompleted.  If that property is not true, LO will copy files from presets/ to user/, even overwriting already existing files (and then set the property to true in user/registrymodifications.xcu; as items are listed lexicographically in user/registrymodifications.xcu, /org.openoffice.Setup/... comes near the end, after all the /org.openoffice.Office/... stuff).

5  Now, if (1) has ever caused a LO 3.5 user/registrymodifications.xcu to contain nil values for those Simple/LargeControlPoints, migration to LO 3.6 will cause LO to stop reading user/registrymodifications.xcu once it sees such a nil value.  This means that LO does not read the value "true" for ooSetupInstCompleted, thinks the user/ directory has not yet been populated, and overwrites various existing files in user/ with defaults from presets/.

The quickest fix is to make those Simple/LargeControlPoints properties (which are no longer used anyway) nil-able again.
Comment 24 Not Assigned 2012-10-04 20:54:12 UTC
Stephan Bergmann committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=a51833bf79eed21a7cddd6da45dc9d3dc07b4983

fdo#52022: Simple/LargeControlPoints actually can have nil values



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 25 Roman Eisele 2012-10-05 07:06:50 UTC
@ Stephan Bergmann:
Thank you very much for your research into this issue and for the fix!

(And also for the detailed explanation. Very interesting! How little changes can have such a big impact ...)
Comment 26 Not Assigned 2012-10-05 08:17:35 UTC
Stephan Bergmann committed a patch related to this issue.
It has been pushed to "libreoffice-3-6":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=b619f4e67290c6c299e44927b1616a60b5dcf34d&g=libreoffice-3-6

fdo#52022: Simple/LargeControlPoints actually can have nil values


It will be available in LibreOffice 3.6.3.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 27 Michael Meeks 2012-10-05 09:56:51 UTC
> 3  When configmgr reads an xcu file and reads a nil value for a property that 
> cannot be nil, it abandons reading that file, effectively ignoring the rest of
> the file's data.

That seems excessively intolerant and sounds like an ongoing nightmare for this sort of issue. Seems like picking a consistent (but essentially random) default value for these cases, would be a much less damaging outcome. That is unless there is some good way of ensuring that this bug-type never recurrs ?

Is there a plan there ? or should I file a separate bug ?
Comment 28 Stephan Bergmann 2012-10-05 10:09:58 UTC
(In reply to comment #27)
> > 3  When configmgr reads an xcu file and reads a nil value for a property that 
> > cannot be nil, it abandons reading that file, effectively ignoring the rest of
> > the file's data.
> 
> That seems excessively intolerant and sounds like an ongoing nightmare for
> this sort of issue. Seems like picking a consistent (but essentially random)
> default value for these cases, would be a much less damaging outcome. That
> is unless there is some good way of ensuring that this bug-type never
> recurrs ?
> 
> Is there a plan there ? or should I file a separate bug ?

With an immediate fix in place now, there's more in this fragile chain to think about, yes.  One is to ignore bad content in an .xcu file more selectively (e.g., ignore just a single item, instead of effectively ignoring the remainder of the file; the current strategy was mostly in place for poor extensions' xcu files, but doesn't make too much sense for problems like we see here).  Another is to have copying from presets/ to user/ not overwrite any already existing files.  Yes, I'm planning on addressing those (and planned to do so in the context of this issue).
Comment 29 Michael Meeks 2012-10-09 10:37:27 UTC
Great news - thanks Stephan ! :-)
Comment 30 Winfried Donkers 2012-11-02 08:14:07 UTC
When upgrading from 3.5.7 to 3.6.3.2 on Windows7-64 with LibO_3.6.3_Win_x86_install_multi.msi, all user information is lost/not transferred to version 3.6 (last used documents, custom colours, macros, etc.).

I have more Windows machines to upgrade (approx. 25), with WinXP, Win7-32 and Win7-64. One some of these machines I could experiment/perform tests to help solving this problem.
Comment 31 Stephan Bergmann 2012-11-02 09:55:43 UTC
(In reply to comment #30)
> I have more Windows machines to upgrade (approx. 25), with WinXP, Win7-32
> and Win7-64. One some of these machines I could experiment/perform tests to
> help solving this problem.

Can you make available the pre-upgrade snapshot of a userdir (or at least its registrymodifications.xcu) for which upgrading fails?
Comment 32 Winfried Donkers 2012-11-02 12:16:52 UTC
Created attachment 69442 [details]
userdir of version 3.5.7 that is not upgraded to 3.6.3.3

Attached zipped file contains all directories/files in C:\Users\username\AppData\Roaming\LibreOffice of a machine (Windows7-64) that still has version 3.5.7 .
Customised setting are (at least) 
last used documents
first, last name, abbreviation of name
7 colours (e.g. 'DCI ziek')

I'm willing to be of further assistence (but I can't build on Windows).
Comment 33 Stephan Bergmann 2012-11-02 12:48:53 UTC
(In reply to comment #32)
> Created attachment 69442 [details]
> userdir of version 3.5.7 that is not upgraded to 3.6.3.3

Quick experimentation with that registrymodifications.xcu at least does not show any new signs of the problem that was the original cause for this issue, namely nil values for non-nilable properties.
Comment 34 Winfried Donkers 2012-11-02 13:05:02 UTC
(In reply to comment #33)
 
> Quick experimentation with that registrymodifications.xcu at least does not
> show any new signs of the problem that was the original cause for this
> issue, namely nil values for non-nilable properties.

Do you want me to create a new bug, or do you think -given that the symptoms are very much alike- that this bug could be extended with this problem as well?
Comment 35 Stephan Bergmann 2012-11-02 14:37:00 UTC
(In reply to comment #30)
> When upgrading from 3.5.7 to 3.6.3.2 on Windows7-64 with
> LibO_3.6.3_Win_x86_install_multi.msi, all user information is lost/not
> transferred to version 3.6 (last used documents, custom colours, macros,
> etc.).

Running Windows LO 3.6.3 with the attached 3.zip:

last used documents:  indeed LO does not show any, but the contained 3/user/registrymodifications.xcu also does not contain any /org.openoffice.Office.Histories data, so there would be nothing to show; after loading new documents, the list does fill up as expected

custom colours:  I do see various "DCI ziek", etc. ones in "Tools - Options... - LibreOffice - Colors"

macros:  there are 3/user/basic/EngLibrary and 3/user/basic/UBC directories with some content, but "Tools - Macros - Organize Macros - LibreOffice Basic... - My Macros" indeed only lists "Standard" and an empty "UBC" (and no "EngLibrary" at all)

So to me this looks like problems migrating Basic macros and I'd suggest you move this to a new issue.  (There might of course be specific userdirs different from the attached one that exhibit different problems.  So it might be best if you archived each userdir before migration.)
Comment 36 Winfried Donkers 2012-11-02 17:01:57 UTC
(In reply to comment #35)
 
> So to me this looks like problems migrating Basic macros and I'd suggest you
> move this to a new issue.  (There might of course be specific userdirs
> different from the attached one that exhibit different problems.  So it
> might be best if you archived each userdir before migration.)

I will do so, after collecting useful data (like the usedirs).
(I restored the status of this bug to resolved/fixed).
Stepha, thank you for your help!
Comment 37 Cor Nouws 2012-11-24 08:52:41 UTC
(In reply to comment #36)
> (In reply to comment #35)
>  
> > So to me this looks like problems migrating Basic macros and I'd suggest you
> > move this to a new issue.  (There might of course be specific userdirs
> > different from the attached one that exhibit different problems.  So it
> > might be best if you archived each userdir before migration.)
> 
> I will do so, after collecting useful data (like the usedirs).
> (I restored the status of this bug to resolved/fixed).
> Stepha, thank you for your help!

Pls see bug 55005
Might either be a representative of this 52022 issue or the new one you are looking for?