Bug 94804 - Cross references break on reload.
Summary: Cross references break on reload.
Status: VERIFIED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Writer (show other bugs)
Version:
(earliest affected)
4.4.4.1 rc
Hardware: All All
: highest critical
Assignee: Oliver Specht (CIB)
URL:
Whiteboard: target:5.1.0 target:5.0.4 target:4.4.6
Keywords: bibisected, dataLoss, regression
: 94548 94836 (view as bug list)
Depends on:
Blocks:
 
Reported: 2015-10-05 20:43 UTC by Taylorh140
Modified: 2016-10-25 19:17 UTC (History)
12 users (show)

See Also:
Crash report or crash signature:


Attachments
Document with error example. (102.94 KB, application/vnd.oasis.opendocument.text)
2015-10-05 20:43 UTC, Taylorh140
Details
cross-reference-numbered paragraph (13.54 KB, application/vnd.oasis.opendocument.text)
2015-10-07 09:51 UTC, Orwel
Details
bibisect log (3.50 KB, text/plain)
2015-10-07 13:01 UTC, Niklas Johansson
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Taylorh140 2015-10-05 20:43:40 UTC
Created attachment 119328 [details]
Document with error example.

I added cross references to the Items listed in the attached file and after i reload the file using FILE-> Reload the cross references are broken.
File should look like this:

1  References
1.1  Build Specifications
[1.0] Tester
[1.1] x64 BK Driver
[1.2] SystemInstaller



a) Build Tester by right clicking and selecting build.
b) Build x64 BK Driver by right clicking and selecting build.
c) Build SystemInstaller by right clicking and selecting build. If additional media is requested obtain it from referenced sources.

but instead looks like this:

1  References
1.1  Build Specifications
[1.0] Tester
[1.1] x64 BK Driver
[1.2] SystemInstaller



a) Build Tester by right clicking and selecting build.
b) Build Error: Reference source not found by right clicking and selecting build.
c) Build Error: Reference source not found by right clicking and selecting build. If additional media is requested obtain it from referenced sources.
Comment 1 Cor Nouws 2015-10-06 12:18:10 UTC
Hi Taylo,

Thanks for your report.
I confirm the problem...

1. Open test file
2. Ctrl+F2 > Cross references > Numbered paragraphs ..
   insert one or more
3. Save, Close, Reopen
   > references are shown in a flash, but then it is destroyed.

TODO
Check duplicates
Check version (bibisect)
Comment 2 Cor Nouws 2015-10-06 12:31:52 UTC
No problem in 4.4.3.2, problem in all later versions
Comment 3 Cor Nouws 2015-10-06 21:56:47 UTC
*** Bug 94836 has been marked as a duplicate of this bug. ***
Comment 4 Orwel 2015-10-07 09:51:11 UTC
Created attachment 119383 [details]
cross-reference-numbered paragraph
Comment 5 Orwel 2015-10-07 09:52:00 UTC
The same behavior in LO 5.0.1 and 5.0.2 and also if you use cross-reference directly to numbered paragraph - see attached file (open and update fields).
This is a critical bug, because every .odt file with cross-references is broken after re-opening it.
Comment 6 Niklas Johansson 2015-10-07 13:01:16 UTC
Created attachment 119387 [details]
bibisect log
Comment 7 Niklas Johansson 2015-10-07 13:04:18 UTC
See bibisect log for details.
Comment 8 Oliver Specht (CIB) 2015-10-07 14:06:54 UTC
Each time a new cross reference is inserted a new bookmark is put around the source. These duplicated bookmarks are removed by the patch of 90808 

Such duplicates should not be created. 
The reason for the duplicates is that the inserted mark is of type MarkType::CROSSREF_NUMITEM_BOOKMARK but after save&reload this is converted to a MarkType::CROSSREF_HEADING_BOOKMARK.
Now the next insertion cannot find the CROSSREF_NUMITEM_BOOKMARK at the given position and inserts a new one.
Comment 9 Peter Moon 2015-10-09 21:53:44 UTC
I am seeing this extensively in Writer for Windows, all versions of LO5 and it was in later releases of 4 as well. About a third of cross-references typically affected.

Interestingly, there can be several references to one target, and some fail while others work properly.

Also, if you construct a cross-reference of several parts (eg "see clause 26.1(a)(i)" where each part of the clause number is individually referenced) you can get partial failure eg the above reference could appear as:

"See clause 26.1[Error-not found!](i)"

If you delete and recreate a broken reference, it will fail again every time. But a reference that survives the first reload is stable forever.

Normally, if you open the affected file in LO for Mac, there's no problem. The Windows version is far worse affected.

This is an important one. In our case, it makes LO unusable in a law office.
Comment 10 Yvan Rose 2015-10-10 13:04:39 UTC
The only way I could fix the problems was:

1) Revert back to LO 3.4.7
2) Completely remove the faulty cross-references from the document
3) Close the document; quit LO
4) Reopen everything
5) Insert the x-references anew

Conclusion from these observations: x-references have been erratic all through LO 4 and 5 (thus far). 

Thank you big time for addressing that.
Comment 11 Michael Stahl (allotropia) 2015-10-12 23:23:16 UTC
thanks Oliver for fixing the root cause.

regression from commit 679faffc68bb854af0f55d0f218698e2f372f00b

looks to me there are 2 problems here, firstly the merge problem above
causing heading-reference to be created instead of numbering-references,
second is that in 4.4.x the heading-references stopped working in
this case.

commit 9d0c51daea67104349cac26de9839afa8baeb099
Author:     Michael Stahl <mstahl@redhat.com>
AuthorDate: Fri May 15 23:57:04 2015 +0200

    tdf#90808: sw: fix detection of duplicate CrossRefBookmarks
    
    The check for existing bookmark positions is using the wrong position.
    It must use the Start of the pam, because the constructor of
    CrossRefBookmark only uses the Start of the pam.

so the versions since 4.2.0 or so would create these documents
where there can be more than one "__RefHeading__" bookmarks on a single
paragraph: every time you try to insert a new CROSSREF_NUMITEM_BOOKMARK
it will find that there is no CROSSREF_NUMITEM_BOOKMARK on the paragraph
and then wrongly insert another CROSSREF_HEADING_BOOKMARK on it.

then with the fix for bug 90808, we will import only the first of these
CROSSREF_HEADING_BOOKMARK in ODF import, and throw the rest of them away
because they are duplicates.

if you're lucky then the bookmark that survives is the one that is
referenced by all the fields; if not, then you get lots of
"Error: Reference source not found" once you update fields.

i'm afraid we'll have to add some horrible hack to load all these corrupted documents.

i've added something sufficiently ugly on master now.
Comment 12 Commit Notification 2015-10-12 23:24:52 UTC
Oliver Specht committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=85eb4d9514858c71fb22c5ed93d651ea24bc6c05

tdf#94804: prefix to save cross reference bookmark fixed

It will be available in 5.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 13 Commit Notification 2015-10-12 23:24:58 UTC
Michael Stahl committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=7c3c3006deaaaf1bb3f2f4eeeaf11da3bcebe53c

tdf#94804 sw: allow duplicate heading cross reference bookmarks

It will be available in 5.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 14 Commit Notification 2015-10-12 23:47:50 UTC
Oliver Specht committed a patch related to this issue.
It has been pushed to "libreoffice-5-0":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=6e0638338d5c587d6bbb9c349d1b25382fbdbfce&h=libreoffice-5-0

tdf#94804: prefix to save cross reference bookmark fixed

It will be available in 5.0.4.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 15 Commit Notification 2015-10-12 23:58:19 UTC
Oliver Specht committed a patch related to this issue.
It has been pushed to "libreoffice-4-4":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=bdc5aab39be149b2fc7d37520414ae251341bc14&h=libreoffice-4-4

tdf#94804: prefix to save cross reference bookmark fixed

It will be available in 4.4.7.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 16 Cor Nouws 2015-10-13 06:36:11 UTC
thanks a lot!

Hardly dare to ask, but since 4.4.6.2 and 5.0.3.2 are still to be run..
Could this be more or less safely done with testing of ..?
Comment 17 Commit Notification 2015-10-14 08:05:21 UTC
Michael Stahl committed a patch related to this issue.
It has been pushed to "libreoffice-4-4":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e95b9fd796e3a52afbb52bc142019115653c6099&h=libreoffice-4-4

tdf#94804 sw: allow duplicate heading cross reference bookmarks

It will be available in 4.4.7.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 18 Commit Notification 2015-10-14 08:07:19 UTC
Michael Stahl committed a patch related to this issue.
It has been pushed to "libreoffice-5-0":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=c5f55acd89ebc57d43e4b92f29800ea4cf12ae31&h=libreoffice-5-0

tdf#94804 sw: allow duplicate heading cross reference bookmarks

It will be available in 5.0.4.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 19 Jean-Baptiste Faure 2015-10-14 09:20:24 UTC
Verified fixed on LibreOffice 5.0.4.0+ built at home under Ubuntu 15.04 x86-64 with gcc 5.1. Thank you very much.

Best regards. JBF
Comment 20 Commit Notification 2015-10-14 22:07:17 UTC
Oliver Specht committed a patch related to this issue.
It has been pushed to "libreoffice-4-4-6":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e59c048a589db0aabd6a066be8bb73f9859d1375&h=libreoffice-4-4-6

tdf#94804: prefix to save cross reference bookmark fixed

It will be available in 4.4.6.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 21 Commit Notification 2015-10-14 22:08:44 UTC
Michael Stahl committed a patch related to this issue.
It has been pushed to "libreoffice-4-4-6":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=7750ea00a88aa6c7acd0a99c9596f16910ab13ba&h=libreoffice-4-4-6

tdf#94804 sw: allow duplicate heading cross reference bookmarks

It will be available in 4.4.6.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 22 Orwel 2015-10-20 13:42:04 UTC
I confirm fixing the bug in tested version:  master~2015-10-18_04.55.50_LibreOfficeDev_5.1.0.0.alpha1 in Win7.

But if i can ask, PLEASE(!), release this bug already in the 5.0.3 version and ASAP. I do not know if you understand, how CRITICAL this bug is. I work in a small office using LO, but almost every document we use contains Cross-references, which caused we CAN NOT WORK on existing documents in LO. Downgrading LO is not a solution, because the upgrade from 4.X to 5.X caused loss of all added personal icons in the Toolbars which was horrible, so we are now back on MS Office (using one LO 4.3 and saving everything as .doc) and waiting for fixing the bug in a stable version. My propose for the future - such (from the viewpoint of a user) CRITICAL bugs which causes LO NOT WORKING (because we really can not use the Writer now) should be fixed in a stable version ASAP although in the way the new stable version contains only this one fix...
Comment 23 Peter Moon 2015-10-21 04:04:21 UTC
I second Orwel's plea.  I am a business technology journalist, and I have raved about LO in the media, so I felt obliged to issue a warning in the national Australian press when this bug hit.  It's that damaging to business users
Comment 24 Yvan Rose 2015-10-21 11:22:03 UTC
I'm on-side too. But at the same time want to thank you all involved for helping with that nasty bug.
Comment 25 Commit Notification 2015-10-22 15:53:08 UTC
Oliver Specht committed a patch related to this issue.
It has been pushed to "libreoffice-5-0-3":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=88675fdfb6498c3203c0e8f5540b610d3963074a&h=libreoffice-5-0-3

tdf#94804: prefix to save cross reference bookmark fixed

It will be available in 5.0.3.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 26 Orwel 2015-10-23 09:55:08 UTC
Thank you very much!
Comment 27 Orwel 2015-10-30 09:48:54 UTC
I just installed the 5.0.3.2 (RC2) and the bug is still present although Comment 25 (should be not present yet). Tested on x64 version (Win7) with the attachment 119383 [details] (Comment 4).
Could you take a look at this, because it is not fixed as mentioned.
Comment 28 Yvan Rose 2015-10-31 18:05:41 UTC
I confirm this from my end as well. The bug is still present in the fresh versions of the application.
Comment 29 Yvan Rose 2015-11-01 13:57:36 UTC
Looking further into this, it may be that references created with older (buggy) versions are the ones that get corrupted. If this is the case, and all we need to do is recreate them one last time, this could be a decent compromise (to many, I hope). However, I cannot find information about this anywhere. Thank you for your further input about this.
Comment 30 Orwel 2015-11-02 08:54:49 UTC
I confirm the bug occurs on documents of older versions. A manual delete of a cross reference and setting a new one solve the problem. The problem is not solved, if you only re-change the cross-reference (this means, double click it and set it one more time, you have to delete it completely before).

I am not sure if recreating the CR one last time suggested by Yvan Rose is meant as an automatically or manually process.  It is a good solution if the process will be done automatically by LO. If you suggest to make it manual, this is a horrible vision for me and my colleagues.

Imagine, there are hundreds of documents created/re-saved in older LO Version. Some of these documents have more than 10 sites. Doing it manually would cost us hours and hours of work, you have to read all the documents through and think where the cross reference should point to.

Sorry for my next words as they are not directly concerned with the solution of the case:
I was a big fan of LO, trying to support the LO team mainly with my free time (tests of beta versions, reporting bugs...) and sometimes also with small contributions, but this BUG (present for more than a month) is so terrible that I slowly loose my faith into this product, really. Understand, there are people using LO as the sole office suite. I do not think each bug has the same "weight". There are CRITICAL bugs (causing LO is NOT WORKING in its basic functions properly) and other bugs. This kind of bug is CRITICAL (in my eyes) and makes LO really unworkable with (at least in our small office where almost each document contains cross references and we often go back to old documents).

The problem with cross references is long-continuing (let see the bug reports of the past), I remember problems with cross-references in LO 3.5. And the problem occurs repeatedly (solved and than again occurred) in many versions. I think a new Version of LO should be released after more testing. I can not believe this kind of CRITICAL bug where CROSS REFERENCES ARE NOT WORKING could be overlooked and not solved for a such long time.

I stopped to use Microsoft Office many years ago because it drove me crazy with repeated bugs of formatting. I feel myself now the same.

One more time sorry for this post, but I felt I have to write this words hoping some developers will read it and discuss it in the higher level of Document foundation...
Comment 31 Michael Stahl (allotropia) 2015-11-02 13:03:20 UTC
there were 2 fixes for this bug, one to fix the creation of new cross references, and one to fix the import of existing documents written by older versions that contain the bad cross references.

unfortunately only the first fix got sufficient reviewers to be included
in time for the 5.0.3 release, while the second fix will only be in the 5.0.4
release.

since this obviously causes confusion i will remove the 5.0.3 from the bugzilla whiteboard.

if you are affected by the bug then use LO 4.4.6 or wait for LO 5.0.4.
Comment 32 Cor Nouws 2015-11-02 15:05:13 UTC
(In reply to Michael Stahl from comment #31)

> if you are affected by the bug then use LO 4.4.6 or wait for LO 5.0.4.

Thanks for clarifying, Michael!
Comment 33 Cor Nouws 2015-11-02 15:22:14 UTC
checked in 4.4.6.3
Both this issue and 94063 are fixed there.
Comment 34 Yvan Rose 2015-11-02 22:42:43 UTC
Adding my thanks here. 

I have been a supporter of LO for years now, providing my yearly donation and encouraging my students to join in. 

On the other hand, I too have been experiencing a little bit of what Orwel was voicing earlier, about the quality control side of things. Given my experience as software developer (designer --I don't to programming), I am fully aware that innovation typically comes at the cost of instability. But perhaps The Document Foundation could work on strengthening the testing side of things a little more. If I could anything to help in that respect, for example in terms of voicing my opinion to the right people, I certainly would. (Thank you for any input there, if any.)

In the meantime, I am grateful for all the work that everyone has poured in LO, by far the best office suite out there.
Comment 35 Michael Stahl (allotropia) 2015-11-04 15:07:03 UTC
*** Bug 94548 has been marked as a duplicate of this bug. ***
Comment 36 Commit Notification 2015-11-27 08:49:46 UTC
Varun Dhall committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=4bd0739b5b2110b428180c270f5f5f8722ff0b0b

Added Test for tdf#94804 cross ref bookmarks created with wrong prefix

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 37 Robinson Tryon (qubit) 2015-12-17 10:33:52 UTC Comment hidden (obsolete)
Comment 38 Cor Nouws 2016-03-29 07:08:10 UTC
verified in 5.1.2.1 on Ubuntu 32 bits