Bug Hunting Session
Bug 39484 - All Outline groups of sheet lost while editing
Summary: All Outline groups of sheet lost while editing
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
3.5.7.2 release
Hardware: All All
: highest critical
Assignee: Not Assigned
URL:
Whiteboard: target:4.2.0 target:4.1.2 target:4.0.6
Keywords:
Depends on:
Blocks: mab4.0
  Show dependency treegraph
 
Reported: 2011-07-22 14:25 UTC by Marcus
Modified: 2013-09-25 15:56 UTC (History)
11 users (show)

See Also:
Crash report or crash signature:


Attachments
First (9.33 KB, application/vnd.oasis.opendocument.spreadsheet)
2011-07-29 14:59 UTC, Marcus
Details
Second (10.42 KB, application/vnd.oasis.opendocument.spreadsheet)
2011-07-29 14:59 UTC, Marcus
Details
Third (10.81 KB, application/vnd.oasis.opendocument.spreadsheet)
2011-07-29 15:00 UTC, Marcus
Details
Third (corrected) (10.38 KB, application/vnd.oasis.opendocument.spreadsheet)
2011-07-29 15:17 UTC, Marcus
Details
calc document that suffers grouping/notice loss (100.20 KB, application/vnd.oasis.opendocument.spreadsheet)
2013-02-18 07:12 UTC, Winfried Donkers
Details
sample of document with outlining before breakage (186.33 KB, application/vnd.oasis.opendocument.spreadsheet)
2013-03-02 10:36 UTC, koukasio
Details
sample of document with outlining after breakage (186.60 KB, application/vnd.oasis.opendocument.spreadsheet)
2013-03-02 10:38 UTC, koukasio
Details
sample2 of document with outlining before breakage (71.09 KB, application/vnd.oasis.opendocument.spreadsheet)
2013-05-01 10:34 UTC, koukasio
Details
sample2 of document with outlining after breakage (63.97 KB, application/vnd.oasis.opendocument.spreadsheet)
2013-05-01 10:35 UTC, koukasio
Details
reproduction_sample (292.34 KB, application/vnd.oasis.opendocument.spreadsheet)
2013-08-17 00:34 UTC, koukasio
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Marcus 2011-07-22 14:25:34 UTC
Several times, while editing one of several spreadsheets, I have found the outline groups which I had recently created, to have disappeared. I believe this sometimes happened while the file was open; more often, I have noticed the outlines' absence only after re-opening a file which I had previously edited. Sometimes the outlining disappear from multiple sheets, sometimes just from one.

When I discover the outline groups to be gone, the rows (or columns) which had been collapsed, are now hidden. I can reveal them by selecting a range including those rows or columns, right-clicking, and choosing Show. But there seems to be no way to recover the outlining I had created, other than to re-create it from scratch.

Is there anyone else who has noticed, or is able to replicate, this behavior? Because it happens inconsistently, I'm not yet sure how to reliably reproduce it, and the files which it affects are full of personal information, so I'm reluctant to share them. If necessary, I suppose I could try to craft a simple file where the behavior might be replicated. (Any tips are welcome.)
Comment 1 Jeffrey 2011-07-24 20:28:19 UTC
I think it would be best if you give a case (even if it isn't consistently reproducible) in which the bug occurs. That way there are some attributes that people can examine which might be the cause for the failure to record the change.

There was a case reported in bug 39479 in which calc does not save changes in cell's widths and other attributes because the spreadsheet was imported as a .doc file. Perhaps that might be the cause of the bug?

Anyway, keep digging,but any sample document or reproducible behavior would be great help.
Comment 2 Marcus 2011-07-25 07:48:42 UTC
(In reply to comment #1)
> I think it would be best if you give a case (even if it isn't consistently
> reproducible) in which the bug occurs. That way there are some attributes that
> people can examine which might be the cause for the failure to record the
> change.
> 
> There was a case reported in bug 39479 in which calc does not save changes in
> cell's widths and other attributes because the spreadsheet was imported as a
> .doc file. Perhaps that might be the cause of the bug?
> 
> Anyway, keep digging,but any sample document or reproducible behavior would be
> great help.

Thanks for the reply. In fact, I believe all of the documents in question originated as PlanMaker files which I saved as Excel and then opened in LibreOffice. When I was editing them in PlanMaker, besides other bugs (which eventually led me to switch to LibreOffice), sometimes the outlines were lost or corrupted (certain levels or elements of them were lost). I thought it was strange that LibreOffice, a completely different program, would have a similar bug, where outlines could be corrupted or lost. Perhaps, as you suggest, it's the files themselves which are corrupted, and LibreOffice only wasn't able to detect or fix the corruption when importing them.

I'll experiment with this and post more.
Comment 3 Marcus 2011-07-29 14:59:03 UTC
Created attachment 49730 [details]
First

See post for details
Comment 4 Marcus 2011-07-29 14:59:36 UTC
Created attachment 49731 [details]
Second

See post for details
Comment 5 Marcus 2011-07-29 15:00:04 UTC
Created attachment 49732 [details]
Third

See post for details
Comment 6 Marcus 2011-07-29 15:01:26 UTC
Since my last post, I tried copying all the sheets of one of the spreadsheets in question to a new file (both ODS), to see if the outlines would still be occasionally lost. Today I opened that file to find, indeed, all the outlining gone.

Jeffrey, if you have a hint for how to copy the data to a new file without bringing along whatever corruption might be causing the outlines to disappear, please let me know.

For now, I post three files, which I've just created, in case anyone can examine them and see what differences might lead one or more of them to lose its outlines. I haven't yet observed that behavior with any of these files, but if I do, I'll post again.

All files are ODS. The first file I simply created from scratch in LibreOffice. I named it "Outlining bug test (created in LibreOffice).ods".

The second file I created in PlanMaker 2010 (by copying & pasting the range of text cells from LibreOffice, then duplicating the outlining by hand in PlanMaker), and saved in PMD (PlanMaker's native format), then saved again in XLS, then opened the XLS in LibreOffice and saved it as ODS. I named it "Outlining bug test (created in PlanMaker 2010 as xls).ods".

For the third file, I used LibreOffice to open the original PMD I had created in PlanMaker, and then saved it as ODS. I named this file "Outlining bug test (created in PlanMaker 2010 as pmd).ods".

For the second and third files, I adjusted some of the column widths, as they didn't all convert accurately. For all three files, I collapsed the outlines to level two before saving (i.e. levels one and two are visible, level three is collapsed).

All three files, therefore, should have these rows, where parentheses indicate rows which are collapsed, and brackets indicate rows which are simply hidden (and aren't part of the outline):

1
-2
-3
-4
5
-6
-7 (--8)
-9
-10
11 [12 13 14]
15
-16 (--17)
18
-19 (--20 --21)
-22
-23

And these columns:
A
-B
-C
-D (--E)
-F

In my experience, when the outlines disappear, those rows/columns which had been collapsed, the last time they were viewed, are now simply hidden, and there's no evidence of outlining.

I hope these files are helpful.
Comment 7 Marcus 2011-07-29 15:17:16 UTC
Created attachment 49733 [details]
Third (corrected)

See post for details
Comment 8 Björn Michaelsen 2011-12-23 12:23:07 UTC
[This is an automated message.]
This bug was filed before the changes to Bugzilla on 2011-10-16. Thus it
started right out as NEW without ever being explicitly confirmed. The bug is
changed to state NEEDINFO for this reason. To move this bug from NEEDINFO back
to NEW please check if the bug still persists with the 3.5.0 beta1 or beta2 prereleases.
Details on how to test the 3.5.0 beta1 can be found at:
http://wiki.documentfoundation.org/QA/BugHunting_Session_3.5.0.-1

more detail on this bulk operation: http://nabble.documentfoundation.org/RFC-Operation-Spamzilla-tp3607474p3607474.html
Comment 9 Florian Reisinger 2012-08-14 13:58:54 UTC
Dear bug submitter!

Due to the fact, that there are a lot of NEEDINFO bugs with no answer within the last six months, we close all of these bugs.

To keep this message short, more infos are available @ https://wiki.documentfoundation.org/QA/NeedinfoClosure#Statement

Thanks for understanding and hopefully updating your bug, so that everything is prepared for developers to fix your problem.

Yours!

Florian
Comment 10 Florian Reisinger 2012-08-14 14:00:07 UTC
Dear bug submitter!

Due to the fact, that there are a lot of NEEDINFO bugs with no answer within the last six months, we close all of these bugs.

To keep this message short, more infos are available @ https://wiki.documentfoundation.org/QA/NeedinfoClosure#Statement

Thanks for understanding and hopefully updating your bug, so that everything is prepared for developers to fix your problem.

Yours!

Florian
Comment 11 Florian Reisinger 2012-08-14 14:04:47 UTC
Dear bug submitter!

Due to the fact, that there are a lot of NEEDINFO bugs with no answer within the last six months, we close all of these bugs.

To keep this message short, more infos are available @ https://wiki.documentfoundation.org/QA/NeedinfoClosure#Statement

Thanks for understanding and hopefully updating your bug, so that everything is prepared for developers to fix your problem.

Yours!

Florian
Comment 12 Florian Reisinger 2012-08-14 14:06:54 UTC
Dear bug submitter!

Due to the fact, that there are a lot of NEEDINFO bugs with no answer within the last six months, we close all of these bugs.

To keep this message short, more infos are available @ https://wiki.documentfoundation.org/QA/NeedinfoClosure#Statement

Thanks for understanding and hopefully updating your bug, so that everything is prepared for developers to fix your problem.

Yours!

Florian
Comment 13 Winfried Donkers 2013-01-24 13:02:55 UTC
I confirm this bug.
With version 3.6.4.3 we had several cases of disappearing groups (and the rows/columns remaining hidden).
Worse, still, all notes were gone too!

All files were originally created with Excel 2000 and in october 2011 converted to ods (by openeing and saving as ods) in a batch.
We have only noticed this behaviour since 3.6.4.3, i.e. we have no reports before december 2012, and 3 sofar since. Currently we are checking all ods files changed since out update to 3.6.4.3, so we might find more.
Comment 14 Winfried Donkers 2013-01-24 13:04:41 UTC
As data gets lost, I have upgraded the severity of this bug
Comment 15 Winfried Donkers 2013-01-29 08:20:04 UTC
(In reply to comment #13)

We have finished out investigations:
of the thousand+ of ods-files that have been converted from xls to ods, two files have suffered loss of all groups and all notes on one or more tabs (not on all tabs). One file had this phenomenom more than once, i.e. at one date the groups and notes of one tab were lost, at another date the groups and note of another tab were lost.
We have only checked files that have been modified in the last 2 months.
One of the files lost its groups and notes on one tab before the upgrade from version 3.5 to 3.6.4.3, the other incidents happened with version 3.6.4.3. We have not been able to establish whether all incidents originated from one user/one installation.
We cannot reproduce the problem. Should we be able to reproduce the problem or similar behaviour, I will update this bug inmediately.
Comment 16 koukasio 2013-02-16 15:44:52 UTC
(In reply to comment #15)
> (In reply to comment #13)
> 
> We have finished out investigations:
> of the thousand+ of ods-files that have been converted from xls to ods, two
> files have suffered loss of all groups and all notes on one or more tabs
> (not on all tabs). One file had this phenomenom more than once, i.e. at one
> date the groups and notes of one tab were lost, at another date the groups
> and note of another tab were lost.
> We have only checked files that have been modified in the last 2 months.
> One of the files lost its groups and notes on one tab before the upgrade
> from version 3.5 to 3.6.4.3, the other incidents happened with version
> 3.6.4.3. We have not been able to establish whether all incidents originated
> from one user/one installation.
> We cannot reproduce the problem. Should we be able to reproduce the problem
> or similar behaviour, I will update this bug inmediately.

I have this problem with EVERY ods file I use. I've tried to narrow down what could the fault be, but I get lost in the possibilities.I feel lost, as a I am engaged to free software and it seems impossible to work without manual grouping and outlining. I tried these with no success
http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=26168
http://webcache.googleusercontent.com/search?q=cache:M_ja5LV3JxcJ:www.oooforum.org/forum/viewtopic.phtml%3Ft%3D67596+&cd=5&hl=en&ct=clnk&client=ubuntu
I am currently working on a 10MB,260 sheet ods file. The problem is exactly as explained above. The ods file does not come from xls conversion. It sometimes happen on one sheet, sometimes on all sheets. The result is devastating on a 260 sheet file. I've always managed to workaround the various bugs from various releases, but this one is the most critical. I am worrying what I am going to do, if I don't find a workaround in the near future. I currently use kubuntu 12.04 , had the same problem with 11.04, I am considering of trying libreoffice4 at some point or another distro. I am also trying to find out how to report the bug since it is the firt time I report or comment a bug.
In the meanwhile I would appreciate any directions on how to deal with it. Going to ms excel is no option, I intend to deal with it.
Comment 17 Winfried Donkers 2013-02-18 07:12:16 UTC
Created attachment 75030 [details]
calc document that suffers grouping/notice loss

Attached document has suffered loss of grouping and notes in entire tabsheet(s) more than once, all with version 3.6.4.3 on Windows 7.
Tab "Bediendeel P4238_4241" was most often the victim of the bug.

Document has been saved as fods to remove client's name and then resaved as ods.
This document was originally created in MS Office 2000 and in 2011 opened in LibreOffice, saved as ods and kept as ods.

When investigating the damaged file (saved as fods to investigate), Grouping and notes where simply not there at all. If need be, I could upload such a damaged file too (after removing the client's name).
Comment 18 Petr Mladek 2013-02-25 10:13:02 UTC
First, thanks a lot for moving this bug forward. Unfortunately, I am not able to reproduce the problem.
I tried the following with both LO-3.6.5.2, and 4.0.1.1:

   1. Opened fdo39484-sample.ods from the comment #17
   2. Saved as fodt and reloaded
   3. Saved as ods and reloaded

Result: The grouping was still there.

The above test was inspired by the description:
--- cut ---
Document has been saved as fods to remove client's name and then resaved as ods.
This document was originally created in MS Office 2000 and in 2011 opened in LibreOffice, saved as ods and kept as ods.
--- cut ---

So,  I do not know how reproduce this and how to debug this. It even looks that the document could get corrupted when removing client's name in the fods form.

Could you please, provide clear steps how to reproduce this? It would help if you start with a good document, do some steps and end with broken document. The following style helps to provide good description:

1. Open document X in application Y
2. Do this
3. Do that
4. ...

Result:

Expected Result:

I am sorry but this bug can't be blocker in this state. It is a data loss but the comment 15 indicates that it affects only very limited set of documents. It seems to be pretty old but I do not see any duplicates or many people in CC which means that it probably affects only very limited group of people => reducing severity a bit.

It even does not belong into MABs in this state because nobody is able to start debugging it right now. I will leave it there because it is a data loss and I guess that you will provide the scenario within next few hours or days. I feel that we are very close :-)
Comment 19 Winfried Donkers 2013-02-25 11:29:21 UTC
(In reply to comment #18)
> Could you please, provide clear steps how to reproduce this? It would help
> if you start with a good document, do some steps and end with broken
> document. The following style helps to provide good description:

No, to my frustration I cannot. We have so far not been able to reproduce the data loss on purpose. We know that one document (attachment 75030 [details]) seems to be more susceptible, but losing the grouping (which is very annoying) and losing the notes (which is more than annoying) seems to happen unexpectedly.
Koukasio (comment #16) also reports seemingly random, unexpected and not (yet) reproducible happening of the bug
What I can say is that we did not have any of this with version 3.5 and earlier.
Because of problems with openening Access files, we only updated to 3.6 when 3.6.4 came out.
When I get a report from someone regaridng this grouping/notes-loss, I try to look at the damaged file, but it seems as if the grouping and notes are simply not saved for that tab sheet.

As our problems look like the orignal reported bug, I reopened it, but my (and koukasio's) problems may be different from the one reported in 2011. I don't know.

> I am sorry but this bug can't be blocker in this state. It is a data loss
> but the comment 15 indicates that it affects only very limited set of
> documents. It seems to be pretty old but I do not see any duplicates or many
> people in CC which means that it probably affects only very limited group of
> people => reducing severity a bit.
> 
> It even does not belong into MABs in this state because nobody is able to
> start debugging it right now. I will leave it there because it is a data
> loss and I guess that you will provide the scenario within next few hours or
> days. I feel that we are very close :-)

Changing the severity and reporting it as MAB was triggered by the loss of data, it not happening only once or with only one user and the confirmation of koukasio of the bug.

I'm willing to help wherever I can. On the coding part, I have no experience with the file saving aspects of calc, but I could add traces to the code - if only we know how to reproduce th bug ...
Comment 20 koukasio 2013-02-25 13:38:42 UTC
I have tried to reproduce the bug, but I can't. I have even made notes with the possibilities, but I get lost in them. My basic problem of reproducing the bug is that it happens in an unfocused sheet and there seems to be a random aspect. 
The closest I've come to it, was when once the +- buttons on horizontal grouping were visible but not responding to clicks or shortcuts for hide/show. After switching sheet and back again , buttons were gone and grouping was lost.
I have some examples, because I save instances of the document often, but it is a 260 sheet 10mb document and it is difficult to track changes...
Comment 21 Petr Mladek 2013-02-28 14:46:09 UTC
I understand your pain. Please, try to understand the other side as well. If it is hard to reproduce, it is also hard to fix. The LO source code is really huge and the bug might be in many locations. Some complicated bugs takes days to fix even when they are easily reproducible. This one is harder because there is no scenario.

The bug has been reported 1.5 year ago. I understand that it is data loss and more people are affected. But still there are only 4 people in CC and no duplicate. I do not understand why it should suddenly start blocking bug fix release with other useful fixes and improvements.

Please note that changing severity from blocker to critical is not that big change. It only means that this bug will not block the release. It has the highest priority, the second highest severity, so it is still on top of the queries. It is listed in MABs, so heavily advertised. I have also nominated it as "Hard Hack", so it might get even more attention soon, see https://wiki.documentfoundation.org/HardHacks

Please continue in monitoring this problem when you work with LO and when you read other bugs or mails. It is kind of detective work. You might meat a new indicia that could help to create a scenario and help to fix it.

I guess that there is an operation that breaks a data structure in memory and the information about the outline groups gets lost. It might help to remember what operations you did when this bug happened and compare them with operations that you do the other days and you do not see this bug.
Comment 22 koukasio 2013-02-28 18:08:39 UTC
(In reply to comment #21)
Thanks for your assistance and cooperation..!
   I understand. I've always managed to work around the bugs, and keep on doing my job, but this one is the first time I feel blocked.
   Considering that I have noticed this problem for over a year now without reporting(not sure what to do), and considering that I feel being a quite experienced user, I can imagine that there are a lot more people who have this problem but do nothing, or run to MSoffice. Going to MSoffice is not an option to me, but I know that people don't have to think the same way.
   I know now, that it is impossible to work without outlining. After failure for a long time, I've decided to sacrifice my time and outline normally (it always breaks in all my multisheet files) until that indicia comes, but I feel a bit wasted at times (it's been a MONTH of everyday failure, not days). 
   I need help.
Comment 23 Winfried Donkers 2013-03-01 06:48:54 UTC
(In reply to comment #21)

I 100% agree with you, I work on the code. But as this is a 'hard hack', it's too complex for me to fix ;)
Given the difficulty in reproducing the problem and the, blocker is not correct.
I've taken the liberty to change it to critical.

I will provide as much information as I can gather about this bug, in order to assist the hard hackers.
Comment 24 Rainer Bielefeld Retired 2013-03-01 17:51:24 UTC
I gambled a little around with 3.4.5 (WIN) and "fdo39484-sample", as expected without quick success. Really a hare nut, thanks to Petr no nominate this one as  hard nut.

I would like to do a "brute-force test" to try to make this one reproducible, but there still are some questions to be clarified before in before

a) is it 100% clear when the loss happens?
a1) you see outlines missing after FILEOPEN, but do not know whether might
    have been lost during FILESAVE, because you did not try to open with
    other software before edits?
a2) definitivly FILESAVE, groups lost for all other software when FILEOPEN
a3) definitivly FILEOPEN, I tried with an other software and still saw the 
    missing group(s)

b) Am I right that normally only some groups are lost and replaced by hiding
   the rows / columns, but normally not all?
b1) yes
b2) no, but ...

c) do we have any document set showing a before/after loss. I would prefer something like Winfried Donkers' sample what is simple enough for quick tests (might be with macro aid)
c1) yes
c2) no, but ...

d) due to "fdo39484-sample.ods" contains hundreds of ODF errors. Are sample documents with valid ODF 1.2 (strict) available?
d1) yes
d2) no
Comment 25 koukasio 2013-03-02 10:23:46 UTC
(In reply to comment #24)
> I gambled a little around with 3.4.5 (WIN) and "fdo39484-sample", as
> expected without quick success. Really a hare nut, thanks to Petr no
> nominate this one as  hard nut.
> 
> I would like to do a "brute-force test" to try to make this one
> reproducible, but there still are some questions to be clarified before in
> before
> 
> a) is it 100% clear when the loss happens?
> a1) you see outlines missing after FILEOPEN, but do not know whether might
>     have been lost during FILESAVE, because you did not try to open with
>     other software before edits?
> a2) definitivly FILESAVE, groups lost for all other software when FILEOPEN
> a3) definitivly FILEOPEN, I tried with an other software and still saw the 
>     missing group(s)
> 
I see outlines missing during a workflow, not affected by FILEOPEN, not sure if affected by FILESAVE. If the document is saved with correct outlining, the outlining will be there in after FILEOPEN. If the outlining breaks during the workflow and saved, it will not be there after FILEOPEN.
I have not tried other software. I use LO (linux).

> b) Am I right that normally only some groups are lost and replaced by hiding
>    the rows / columns, but normally not all?
> b1) yes
> b2) no, but ...
> 

The outlining breaks a single sheet or more sheets, not parts of sheets. There are times that I have noticed some sheets broken, while others remain unaffected, although it seems certain that they will break at some point during the workflow, they always break . Most times I've noticed all sheets broken.
My basic problem of reproducing the bug is that it happens in an unfocused sheet and there seems to be a random aspect. 
The closest I've come to it, was when once the +- buttons on horizontal grouping were visible but not responding to clicks or shortcuts for hide/show. After switching sheet and back again , buttons were gone and grouping was lost. 

> c) do we have any document set showing a before/after loss. I would prefer
> something like Winfried Donkers' sample what is simple enough for quick
> tests (might be with macro aid)
> c1) yes
> c2) no, but ...
> 
My samples might not be "simple enough for quick tests" as they are 10MB 260sheet, but I can send you two samples in case you want to check out.

> d) due to "fdo39484-sample.ods" contains hundreds of ODF errors. Are sample
> documents with valid ODF 1.2 (strict) available?
> d1) yes
> d2) no
I've always had the setting ODF format version: 1.2 Extended (recommended) in the Options menu.
Comment 26 koukasio 2013-03-02 10:36:24 UTC
Created attachment 75775 [details]
sample of document with outlining before breakage
Comment 27 koukasio 2013-03-02 10:38:30 UTC
Created attachment 75776 [details]
sample of document with outlining after breakage
Comment 28 Rainer Bielefeld Retired 2013-03-02 12:51:06 UTC
@koukasio (In reply to comment #25):
Thank you, that helps a lot. I will do some tests and research with your documents asap (unfortunately rather busy currently).

@all:
Can you please submit separate bugs for all similar problems what are not "loss of all groups during workflow" weth "related to this one"? Although all these problems might have the same roots, dividing the problem into smaller parts might ease tests and finding those roots.
Comment 29 Rainer Bielefeld Retired 2013-03-03 08:54:35 UTC
I did some first investigation in contents.xml of sample "2013-03-02 10:36 UTC, koukasio@gmail.com " and related broken document. I see that the groups (not unexpectedly) really are lost, compare:

CONTENT.XML OK
------------------
<table:table-cell table:number-columns-repeated="1015"/>
</table:table-row>
<table:table-row-group>
<table:table-row-group>
<table:table-row table:style-name="ro1">
<table:table-cell table:style-name="ce6"/>
<table:table-cell table:style-name="ce15" office:value-type="string">

BROKEN same area (missing group definition):
-------------------------------------------
<table:table-cell table:number-columns-repeated="1015"/>
</table:table-row>
<table:table-row table:style-name="ro1">
<table:table-cell table:style-name="ce6"/>
<table:table-cell table:style-name="ce15" office:value-type="string">


@koukasio / all
How often do you see such problems? all 100 edits? all 100000 edits? Only to get an idea how much edits might be needed to have a chance to see something interesting
Comment 30 Winfried Donkers 2013-03-04 07:09:17 UTC
(In reply to comment #24)
> I gambled a little around with 3.4.5 (WIN) and "fdo39484-sample", ...
We have not noticed grouping/notes losses with versions prior to 3.6.4 (all Windows), except possibly once (with version 3.5.something) in November 2012.

> a) is it 100% clear when the loss happens?
In our cases, all losses happened when the document was saved and was noticed when openened the next time. That is to say, grouping (sometimes notes as well) gets lost on a complete sheet, which need not necessarily be the sheet in view. So in some cases, the loss of grouping/notes was only noticed some time later.
The groupings (notes) are simply not saved for that sheet, I examinated several backups and studied the xml-content.

> b) Am I right that normally only some groups are lost and replaced by hiding
>    the rows / columns, but normally not all?
It may be that I do not understand your question, but when it happens _all_ grouping (rows and columns) are gone on a sheet. I cannot say whether it happened on one sheet only or or one or more sheets, but it did not happen on all sheets. The sheet(s) with lost grouping (notes) were often not the selected/visible sheets.
 
> c) do we have any document set showing a before/after loss. I would prefer
> something like Winfried Donkers' sample what is simple enough for quick
> tests (might be with macro aid)
Yes, I have several documents before and after as we have a lot of backups. I cannot upload them as they are (the documents include information from customers) to freedesktop. I can upload one or more fods with sensitive information removed, I can also send them (ods) as an attachment to Rainer or Petr directly on the condition that they are used discretely.
 
> d) due to "fdo39484-sample.ods" contains hundreds of ODF errors. Are sample
> documents with valid ODF 1.2 (strict) available?
The documents with which we experienced the grouping/notes loss were all xls-docuemnts (Excel 2000) that were converted to odt in the autumn of 2010 and used with LibreOffice since. So yes, ODF-errors can be expected in the documents.

I include 'notes' when I say grouping, as the loss of notes occurs together with the loss of grouping (in the same tab). I cannot say with certainty that grouping loss did not occur without notes loss, but the other way round (notes have not been lost without groups being lost) is definitivelt true.

We have 8 people editing a total of 20-100 of calc-documents like the fdo39484-sample. Some are edited several time a day, some only once a  year. We only noticed the data loss since December, 2012 and only found two documents which suffered data loss (we checked more than 50 of these documents that were modified in November/December 2012 and January 2013). One of these two documents, suffered data loss several times. My suspect is that it is a combination of document-content (ODF-error?) and FILESABE handling of LibreOffice. The people that edit these two documents regularly have downgraded to version 3.5.7. Since then we have not noticed grouping/notes losses.
As it looks to me that it is document-related, the ratio with which the bug happens could be less than 1/1000 for documents in general or 1/10 (a guess) for susceptible documents.
Comment 31 koukasio 2013-03-04 16:30:59 UTC
(In reply to comment #29)

> @koukasio / all
> How often do you see such problems? all 100 edits? all 100000 edits? Only to
> get an idea how much edits might be needed to have a chance to see something
> interesting
When trying to notice, I keep checking back an forth for any losses. At some point I find it really frustrating to keep checking but there comes a point of breakage. So I can tell you that it is more than 100. 100000 sounds too much. Maybe a class of 100-10000.
Comment 32 Petr Mladek 2013-03-05 08:48:00 UTC
@koukasio: Ah, you set the blocker severity again. I understand that it is a blocker for you but it is not blocker for most other users who do not use this feature. The broader view is important in the time based release process, see https://wiki.documentfoundation.org/Release_Criteria. I am sorry to say but this bug could not and will not block 4.0.1 release and any further bugfix releases. I leave the severity as is because I am not able to persuade you and do not have possibility to make it read only. Any shufling just create mess and is waste of time. The current severity does not reflect reality and will be ignored in the release process decisions.
Comment 33 koukasio 2013-03-05 08:50:51 UTC
(In reply to comment #32)
I did nothing, I expect your apology...
Comment 34 koukasio 2013-03-05 09:19:10 UTC
(In reply to comment #33)
> (In reply to comment #32)
> I did nothing, I expect your apology...
Apology is all mine, I was changing the severity of the bug accidentally, maybe something to do with the refreshing of the page in my browser. I am sorry for the mess I've caused.
Comment 35 Michael Meeks 2013-04-30 20:42:48 UTC
Winfried - does this continue with LibreOffice 4.0 ? - I had a go at reproducing it; cranked down the undo level to only 2 steps to try to provoke things some more [ no joy ], did a number of random hide/show / column deeletion etc. The sample sheet is rather beautiful ! Thanks for that koukasio ! otherwise, I'm a bit stuck here as others have been; how annoying !
Comment 36 Michael Meeks 2013-04-30 20:47:04 UTC
Diffing -u -w the flat-odf of before/after it is interesting to see things like:

@@ -1804,8 +1804,6 @@
      </table:table-cell>
      <table:table-cell table:number-columns-repeated="1015"/>
     </table:table-row>
-    <table:table-row-group>
-     <table:table-row-group>
       <table:table-row table:style-name="ro1">
        <table:table-cell table:style-name="ce6"/>
        <table:table-cell table:style-name="ce15" office:value-type="string" calcext:value-type="string">

ie. the data appears un-changed itself - the only omission is the table:table-row-group elements - of which there are exactly zero in the broken file vs. ~400 in the original file.
Comment 37 Michael Meeks 2013-04-30 21:04:17 UTC
There are several instances of:

@@ -10545,8 +10603,7 @@
      <table:table-cell table:style-name="Default" table:number-columns-repeated="6"/>
      <table:table-cell table:number-columns-repeated="1015"/>
     </table:table-row>
-    <table:table-row-group table:display="false">
-     <table:table-row table:style-name="ro1" table:visibility="collapse">
+    <table:table-row table:style-name="ro1">
       <table:table-cell table:style-name="Default"/>
       <table:table-cell office:value-type="string" calcext:value-type="string">
        <text:p>z/(m)=</text:p>
@@ -10557,7 +10614,7 @@
       <table:table-cell table:style-name="Default" table:number-columns-repeated="6"/>
       <table:table-cell table:number-columns-repeated="1015"/>
      </table:table-row>
-     <table:table-row table:style-name="ro1" table:visibility="collapse">
+    <table:table-row table:style-name="ro1">
       <table:table-cell table:style-name="Default"/>
       <table:table-cell office:value-type="string" calcext:value-type="string">
        <text:p>⇒z/(m)=</text:p>

The broken document has zero instances of a collapsed table visibility in parallel with it's lack of table-row-group elements; and independent of them.
Comment 38 Winfried Donkers 2013-05-01 07:24:44 UTC
(In reply to comment #35)
> Winfried - does this continue with LibreOffice 4.0 ? 

Hi Michael,
I have a script running daily that checks changed ods files for reduction in size; these files are then checked manually if grouping/notes are missing.
So far, the problem has not reoccurred, which is forunate for us (reoccurrence is deadly for the confidence in libreffice of a group of users who still want to go back to MS Office), but not so fortunate for bug fixing ;)
Currently, the department using the kind of ods file where the problems occurred use versions 3.5.7 (mainly) and some 3.6.5. Not yet 4.0 as bug 59405 blocks necessary functionality for us (haven't tested 4.0.3 yet though).

Should we encounter grouping/notes loss, I will report asap, with version and if possible fods files before and after. (The original ods files cannot be provided to a public site because of client names.)
Comment 39 koukasio 2013-05-01 10:34:17 UTC
Created attachment 78710 [details]
sample2 of document with outlining before breakage

There has been loss of outlining before in the document history. For what I recall, this one happened after I added some columns and messed up with some undo stuff.
Comment 40 koukasio 2013-05-01 10:35:06 UTC
Created attachment 78711 [details]
sample2 of document with outlining after breakage

There has been loss of outlining before in the document history. For what I recall, this one happened after I added some columns and messed up with some undo stuff.
Comment 41 Michael Meeks 2013-05-01 10:56:50 UTC
koukasio - did that happen recently - and if so with what version of LibreOffice ?

If it is before 3.6, and given that Winfried can't reproduce this in six weeks of work with more recent versions; I'm inclined to close this as 'worksforme' and hope we nailed it in newer versions (though where I don't know).

I wonder - could it be related to cut/paste/undo of grouped columns/rows somehow ? :-)
Comment 42 koukasio 2013-05-01 12:32:57 UTC
(In reply to comment #41)
> koukasio - did that happen recently - and if so with what version of
> LibreOffice ?
> 

This happened just yesterday! I keep using this feature (even though the dissapointment is big at points) for debugging purposes, in case of an enlightenment... I am still having it in all multisheet files which become a little complex...
There has been loss of outlining before in the attached document history. For what I recall, this one happened after I added some columns and messed up with some undo stuff.

> If it is before 3.6, and given that Winfried can't reproduce this in six
> weeks of work with more recent versions; I'm inclined to close this as
> 'worksforme' and hope we nailed it in newer versions (though where I don't
> know).
> 
I am the "stable" guy who doesn't update distros or software easily. It seems more productive to me to stay focused in my work rather than hunting the "edge" of things...
I will update to 4.0.2 for debugging purposes, but I will be working on the files i've started with 3.5.4.2 . Would that mean anything to you?

> I wonder - could it be related to cut/paste/undo of grouped columns/rows
> somehow ? :-)

I am very awkwardly suspicious about the "undo" thing. There's other things that I've noticed when "undo"ing , I wonder if I should report them as bugs. 
When I want to copy the exact contents of a cell without using absolute references (which I do a lot), I have 3 choices:
1)(cellInsert→Ctrl+A→Ctrl+C→Esc→...→Paste
Minor note: This last paste action cannot be undone?!..is never undone, I have to delete the paste manually if I have to... 
I have this action as a macro assigned to shortcut but it has the same behaviour as the manual one. The behaviour is right when I Ctrl+Shift+V(Paste_special)→Unformatted_Text...
This action is only useful for copying contents of a single cell
2)(cell1)...Ctrl+X→Ctrl+Z→(cell2)...Ctrl+V
Problem: The reference (cell1) is cut and pasted to cell2 even though I have undone the cut action. That's the reason I don't use this action.
3)(cell1)...Ctrl+X→(cell2)...→Paste→Ctrl+C→Ctrl+Z→Ctrl+Z→(cell2)...→Ctrl+V
This action works (for multiple cells), even though I've noticed at awkward points that the reference is cut and pasted as well but I can't reproduce it right now...
This action is useful for copying contents of a single or multiple cells but cannot write as a macro...
I begun writing this cut/paste/undo comment before you post your question..!
Should I report any of these?
Comment 43 koukasio 2013-05-03 17:09:37 UTC
I just had another incident with 4.0.2. Unfortunately I tried to reproduce it but failed. It happened while I was working on sample2. The steps that took place were Edit→Sheet→Select→Ctrl+a(all) and then I did insert columns, merge cells, undo stuff. I was checking the outlining. At some point I pointed with the mouse at the "+" sign of the outlining but was not pressable. I switched sheets back and forth and outlining was lost on several sheets but not all.
Comment 44 David Greaves 2013-05-27 09:43:01 UTC
Maybe things have changed but I found a very minimal test case

Version 4.0.3.3 (Build ID: 400m0(Build:3)) from Debian

Open new instance of localc

New sheet
highlight column D
Press F12 -> grouping appears
Save as /tmp/grptst2.xlsx
close sheet
open sheet

No grouping visible


For background I hit this bug on sheets generated by
  http://search.cpan.org/~jmcnamara/Excel-Writer-XLSX-0.67/

That creates sheets which *do* have grouping and which is preserved in rows, but not columns afaict
Comment 45 Winfried Donkers 2013-05-27 10:42:43 UTC
(In reply to comment #44)
> Maybe things have changed but I found a very minimal test case

Well, this may help  find the cause, which would be very nice.
I confirm the behaviour of the minimal test case with openSUSE version 3.6 and with master (on openSUSE 12.3).
The way to reproduce the problem may be different, but at least it is reproduceable.
Comment 46 koukasio 2013-05-27 13:17:23 UTC
(In reply to comment #44)
I confirm the behaviour with 4.0.2.2 on Kubuntu 12.04
Comment 47 Winfried Donkers 2013-05-28 05:58:42 UTC
(In reply to comment #45)

I forgot to mention:
I confirm the behaviour of the minimal test case with version 4.0.3 on Windows.

Now we can only hope that this _is_ related to the bug and not a separate anomaly ;)
Comment 48 Michael Meeks 2013-05-28 16:57:50 UTC
Hi David; I rather suspect that your test case just highlights the lack of grouping support in the XLSX export - which is a different bug.

Some random code pointers: SID_OUTLINE_MAKE applies outline grouping - (not to be confused with any SID_*GROUP foo).

It looks like that all ends up in sc/source/ui/docshell/olinefun.cxx - MakeOutline and RemoveOutline; hmm ...
Comment 49 Winfried Donkers 2013-06-10 07:10:46 UTC
We just found out another case of grouping loss on one sheet of an ods document.
Last change was made on a Windows 7 machine with version 3.5.7 (machine was downgraded to avoid this problem as we suspected the problem started with version 3.6 - which assumption was clearly wrong).
This time the notes were not lost, only the grouping of one sheet (other sheets in the document were still with grouping and notes).

The document is available in version with grouping and without (i.e. before and after the occurrance of the problem). Given the content I cannot make them attachments of this public bug, but I can send them directly to the developer working on this bug. I can however upload fods files with the client information removed, allthough I fear that both files will be valid and the difference will be that one file has grouping setting for n sheets and the other has grouping settings for n-1 sheets (the settings are simply not there, they don't seem corrupted).
Comment 50 tommy27 2013-08-12 13:13:07 UTC
@Winfried
do you still reproduce this bug with recent 4.0.4 or 4.1.0 releases?
Comment 51 Winfried Donkers (retired) 2013-08-13 06:56:31 UTC
(In reply to comment #50)
> @Winfried
> do you still reproduce this bug with recent 4.0.4 or 4.1.0 releases?

AFAIK we haven't encountered this bug with version 4.0 (4.1 isn't ready for company use yet).
The bug occurs very rarely, but -given the data loss- it is a bad one.

My guess is that it is connected to certain (long existing) files. With two files it has occurred multiple times, with hundreds of other files is has never occurred (and these files all have the same base file from which they originate, first an xls-file, now an ods file).
Comment 52 tommy27 2013-08-13 07:43:58 UTC
so are you saying that it has never reoccurred yet in 4.0.x or that it still happens rarely? or that it happened rarely in 3.6.x...
Comment 53 Winfried Donkers (retired) 2013-08-13 08:26:07 UTC
(In reply to comment #52)
> so are you saying that it has never reoccurred yet in 4.0.x or that it still
> happens rarely? or that it happened rarely in 3.6.x...

The bug hasn't occurred with version 3.5.5 or earlier (we think), it seemed to start with 3.6, but we know of at least one instance when the bug occurred with version 3.5.7.
We upgraded from 3.5.7 to 4.0.4 and so far, I have not received reports of the bug.

(BTW we are currently moving the data from these calc-documents to our ERP-platform, which means that we are likely to experience the bug even less, as these calc-documents become read-only once the data is moved. I will so be of limited 'use' with respect to this bug.)
Comment 54 tommy27 2013-08-13 13:21:22 UTC
Ok Winfried.
I'm gonna set status as WORKSFORME since you said it's not happening anymore in 4.0.x even if we don't know what fixed it.

feel free to REOPEN this report if you will reproduce it again in 4.0.x or 4.1.x
Comment 55 koukasio 2013-08-13 23:41:25 UTC
(In reply to comment #46)
> (In reply to comment #44)
> I confirm the behaviour with 4.0.2.2 on Kubuntu 12.04

I have confirmed that the bug exists with 4.0.X with newly created files(not old ones)...  I have been updating to the latest releases of libreoffice to help debugging. The problem is not fixed. I am still trying to figure out what is happening
I don't want to change any status bymyself but I think that it is wrong to state as resolved. I will keep trying to help
Comment 56 tommy27 2013-08-14 07:32:16 UTC
@koukasio
ok. I reopen the bug and move it to the mab4.0 list.
it would be very important to obtain a test case and instruction to reproduce.
Comment 57 Ralf 2013-08-15 12:46:16 UTC
I want to start with an introductory paragraph, which contains the most important information of this comment according to my non-technical judgment:

The bug occured repeatedly after changing cell formatting globally in the sheet. The cause for global change of cell formatting can be deletion of a user-defined format code, change of the default settings in "Styles and Formatting" in the "Format" menu or similar changes.

And now the long version:

The reported bug causes a siginificant annoyance in my work with LO Calc for more than a year. I had not reported the bug because I cannot reproduce it. Today I found this bug report accidentally and realised that other users encounter the same problem and that reproducing the bug is not easy for them either.

I encounter the bug in a rather big and complicated spreadsheet, which I had started setting up with MS Excel 2007. To migrate from MS Excel to LO Calc I had converted the sheet by saving it as XLS with MS Excel, loading it with LO Calc and then saving it as ODS with LO Calc. Since then I kept developing the spreadsheet considerably, and the bug occurs from time to time.

The spreadsheet contains 18 worksheets, the cells contain several thousand links to other cells, and the spreadsheet size is 2 MB in ODS file format. I cannot share the file for confidentiality reasons.

The bug occured with LO 3.5 and LO 3.6. I remember to have encountered the bug with LO 3.4 as well, but I am not absolutely sure about the occurrence in LO 3.4. I have not switched to LO 4.0 yet, because the bug had occured in my spreadsheet after version changes (i.e. when opening the sheet with the new version), so that I am very conservative with updating LO program versions. Besides, LO 4.0 contains another unrelated bug in LO Impress, which is significant for my work and which does not occur with LO 3.6. Currently I work with LO 3.6.7.2 under Windows 7 with 32-bit architecture (x86).

If the bug occurs, I lose the groupings of columns and rows on all worksheets. However, I am not absolutely sure, if the bug might have occured by affecting only some worksheets in the spreadsheet. I think that the bug was not related to saving or opening the spreadsheet. Again I am not sure about it. I lost the comments in cells of the worksheet as well. Therefore, I gave up using comments. I can work on the spreadsheet without using comments, but I do not get along without grouping columns and rows. Therefore, my experience with the bug concentrates on loss of grouping columns and rows.

Since I had found out that the bug occured mainly when changing cell formats globally (see introductury remark of this comment), I could minimise its frequency of occurrence. Nevertheless, the bug can still occur unexpectedly.

There is a high probability that the bug occurs in my spreadsheet if I do a series of global cell format manipulations. I tried to reproduce it with a simple spreadsheet, which I had converted from XLS or XLSX to ODS. I was not able to produce the bug with a simple spreadsheet.

I am not a developer. I assume that a comment like this one is technically of limited use. Nevertheless, I want to share my experience with this bug. Maybe my description helps though it is not extremely precise. Moreover, it documents that there are more users out there, who struggle considerably with the effects of this bug.

A big thank you to everyone who has tried and will try to hunt down this bug considering its complexity and the difficulty to reproduce it with a simple spreadsheet.
Comment 58 koukasio 2013-08-17 00:34:32 UTC
Created attachment 84160 [details]
reproduction_sample

HALLELUJAH!!!!!!!!
I have managed to reproduce the bug!!!! 
http://www.youtube.com/watch?v=YrLk4vdY28Q
Let's get serious...
I am attaching the file
The steps to follow are:

go to sheet named '1'
select row 15
Insert→Rows
select row 16
cut
paste to row 15
undo
Voilà!!
Go to sheet named '2'... grouping/outlining is lost!!
I hope this is useful, this thing has been such a pain for so long!!
Comment 59 koukasio 2013-08-17 00:49:39 UTC
I started building the document yesterday on lo 4.1.0.4 on Ubuntu 12.04 , 08/16/2013, 15:41:44 .
I started it from scratch , no M$ windows, office etc.
The template I am using was started a year before on something like lo3.5. Would you like me to attach my template as well?
Comment 60 Winfried Donkers (retired) 2013-08-19 07:30:42 UTC
(In reply to comment #58)
> Created attachment 84160 [details]
> reproduction_sample
> I hope this is useful, this thing has been such a pain for so long!!

I confirm the getting lost of the grouping in sheet 2, using version 4.0.4.2 on Windows 7 and using version 4.2.0.0.alpha0+ on openSUSE 12.3

Unfortunately, I am not familiar with in this part of the calc code, so I can't try to fix it.
Comment 61 tommy27 2013-08-19 07:47:19 UTC
I add Kohei Yoshida to CC list. maybe he can help.
Comment 62 Winfried Donkers (retired) 2013-08-19 07:48:30 UTC
(In reply to comment #60)
> (In reply to comment #58)
> > Created attachment 84160 [details]
> > reproduction_sample
> > I hope this is useful, this thing has been such a pain for so long!!
> 
> I confirm the getting lost of the grouping in sheet 2, using version 4.0.4.2
> on Windows 7 and using version 4.2.0.0.alpha0+ on openSUSE 12.3
> 
> Unfortunately, I am not familiar with in this part of the calc code, so I
> can't try to fix it.


I can reproduce the loosing of grouping in other calc documents as well:
-select sheet before sheet with grouping
-select row n
-paste somewhere
-undo : the grouping in th enext sheet is lost
I have not tried other combinations, the steps in comment 58 and above should be a sufficient starting point for a developer.
Comment 63 Michael Meeks 2013-08-19 10:57:53 UTC
koukasio - brilliant :-) thanks so much for the reproduction steps; it's fantastic to finally have a good way to reproduce this.
Comment 64 koukasio 2013-08-19 11:42:19 UTC
(In reply to comment #63)
> koukasio - brilliant :-) thanks so much for the reproduction steps; it's
> fantastic to finally have a good way to reproduce this.

It is the lo-people to thank, I am just a humble user :-) I hope I am being helpful...
Comment 65 Winfried Donkers (retired) 2013-08-20 10:16:18 UTC
(In reply to comment #62)
> I can reproduce the loosing of grouping in other calc documents as well:
> -select sheet before sheet with grouping
> -select row n
> -paste somewhere
> -undo : the grouping in th enext sheet is lost
> I have not tried other combinations, the steps in comment 58 and above
> should be a sufficient starting point for a developer.

Instead of rows, columns can also be used to reproduce the loosing of grouping in the next tab sheet. 
I have not been able yet to loose the comments with these steps.
Comment 66 Michael Meeks 2013-08-20 15:50:44 UTC
I did a bit of work on this; the undo object is ScUndoPaste:

ScUndoPaste::DoChange (this=0xae48eb8, bUndo=true) at /data/opt/libreoffice/master/sc/source/ui/undo/undoblk.cxx:896

Seems to do some rather odd things - it appears eg. to touch other sheets than the current one (on which the operation is done) - no idea why:

937	        for (size_t i = 0, n = maBlockRanges.size(); i < n; ++i)
938	        {
939	            ScRange aCopyRange = *maBlockRanges[i];
940	            aCopyRange.aStart.SetTab(0);
941	            aCopyRange.aEnd.SetTab(nTabCount-1);
942	            pDoc->CopyToDocument( aCopyRange, nUndoFlags, false, pRedoDoc );
943	            bRedoFilled = true;

Which of course only fills the redo data - but appears to copy every sheet rather than the current one [ perhaps that has something to do with it ] - still thinking ;-)
Comment 67 Ralf 2013-08-20 18:20:32 UTC
(In reply to comment #65)
> I have not been able yet to loose the comments with these steps.

I lose the the comments as well if I use koukasio's sheet from comment 58 and follow the described steps. After opening koukasio's sheet, the cursor is on sheet "2" in cell A12. I insert a comment in this cell. It is lost after following the steps in comment 58.

I use Windows 7 with 32-bit architecture (x86).

I assumed in comment 57 that the loss of grouping can be related with global changes of cell formatting. I could not verify this suspicion with koukasio's sheet. My assumption of comment 57 seems to be wrong.
Comment 68 Ralf 2013-08-20 18:25:38 UTC
(In reply to comment #67)
> I use Windows 7 with 32-bit architecture (x86).

Forgot to mention: LO 3.6.7.2
Comment 69 Winfried Donkers (retired) 2013-08-21 06:20:06 UTC
(In reply to comment #66)
> I did a bit of work on this; the undo object is ScUndoPaste:
> 
> ScUndoPaste::DoChange (this=0xae48eb8, bUndo=true) at
> /data/opt/libreoffice/master/sc/source/ui/undo/undoblk.cxx:896
> 
> Seems to do some rather odd things - it appears eg. to touch other sheets
> than the current one (on which the operation is done) - no idea why:
> 
> 937	        for (size_t i = 0, n = maBlockRanges.size(); i < n; ++i)
> 938	        {
> 939	            ScRange aCopyRange = *maBlockRanges[i];
> 940	            aCopyRange.aStart.SetTab(0);
> 941	            aCopyRange.aEnd.SetTab(nTabCount-1);
> 942	            pDoc->CopyToDocument( aCopyRange, nUndoFlags, false,
> pRedoDoc );
> 943	            bRedoFilled = true;
> 
> Which of course only fills the redo data - but appears to copy every sheet
> rather than the current one [ perhaps that has something to do with it ] -
> still thinking ;-)

@Michael: when you've thought it out ;-) , but haven't got the time to code and test, I am willing to help fixing this. It is a very annoying bug at our company. Having said that, I won't be able to do any coding in the first half of September.
Comment 70 Michael Meeks 2013-08-21 18:03:02 UTC
Not that it's news but this:

void ScUndoPaste::DoChange(bool bUndo)
...
        for (size_t i = 0, n = maBlockRanges.size(); i < n; ++i)
        {
            ScRange aRange = *maBlockRanges[i];
            ScMarkData::iterator itr = aMarkData.begin(), itrEnd = aMarkData.end();
            for (; itr != itrEnd && *itr < nTabCount; ++itr)
            {
                aRange.aStart.SetTab(*itr);
                aRange.aEnd.SetTab(*itr);
**              pUndoDoc->UndoToDocument(aRange, nUndoFlags, false, pDoc);
            }
        }

line - when uncommented avoids the issue occuring; I guess that's the core of Undoing pastes though ;-)

Interestingly though, the *itr is 4 for this - and the inner loop only occurs once; 4 seems a plausible value for nTab called '1' which I'm editing; which looks correct.
Comment 71 Michael Meeks 2013-08-21 19:10:10 UTC
void ScDocument::UndoToDocument(const ScRange& rRange,
...
...
    if (nTab2 < static_cast<SCTAB>(maTabs.size()))
        CopyToDocument( 0,0,nTab2+1, MAXCOL,MAXROW,maTabs.size(), IDF_FORMULA, false, pDestDoc, pMarks );

Appears to do a giant copy to undo a paste - of all the formulae in the document - which seems a tad steep. Either way - disabling that un-breaks the groups too.
Comment 72 Michael Meeks 2013-08-22 09:34:51 UTC
void ScTable::CopyToTable(
...
        pDestTab->SetOutlineTable( pOutlineTable );     // auch nur wenn bColRowFlags

Disabling this line stops us loosing the grouping; so - I guess that pOutlineTable is NULL in the undo information - which is interesting - but why only for this one sheet ? why don't we loos more :-)

The hunt continues ...
Comment 73 Winfried Donkers (retired) 2013-08-22 09:44:23 UTC
(In reply to comment #72)
> void ScTable::CopyToTable(
> ...
>         pDestTab->SetOutlineTable( pOutlineTable );     // auch nur wenn
> bColRowFlags
> 
> Disabling this line stops us loosing the grouping; so - I guess that
> pOutlineTable is NULL in the undo information - which is interesting - but
> why only for this one sheet ? why don't we loos more :-)
> 
> The hunt continues ...

Please keep hunting ;)

BTW, we did have a loss of grouping and of grouping_and_comments (all cell comments on a sheet). I can't tell if we experienced loss in more than one sheet at a time, especially as we didn't know what caused the loss and loss in more sheets could have been caused by more than one undo action.
Comment 74 Michael Meeks 2013-08-22 11:13:15 UTC
Another interesting data-point is that this is exclusive to cut/paste, copy/paste doesn't suffer from the same issue; so back up to:

bool ScViewFunc::PasteFromClip( sal_uInt16 nFlags, ScDocument* pClipDoc,

The more I read this the more I marvel at what is going on - it looks rather fragile. It appears that the code wants to preserve formulae on ~all sheets so we can undo the effects of cut/paste.

Unfortunately, the CopyToDocument method has two modes of operation: one where every tab is present (when it really copies the range you pass), and another when tabs are not present (when it only copies to existing tabs).

First we try to CopyToDocument with a limited set of (the selected) tabs - to just copy that data for undo; so we only copy the outline data for sheets that are active.

         // all sheets - CopyToDocument skips those that don't exist in pUndoDoc
         SCTAB nTabCount = pDoc->GetTableCount();
+        if ( bCutMode )
+        {
+            // fdo#39484 - needs to be done before the CopyToDocument to preserve
+            //             outline & other data on sheets that we're not viewing.
+            pUndoDoc->AddUndoTab( 0, nTabCount-1 );
+        }

fixes that; but of course it then starts to copy far too much.

Then unfortunately, we do another copy into the pUndoDoc here:

            pRefUndoDoc->CopyToDocument( 0,0,0, MAXCOL,MAXROW,nTabCount-1,
                                            IDF_FORMULA, false, pUndoDoc );

Which is presumably intended to preserve formulae, however this was again not constructed for all tabs, so it now overwrites the outline data with NULL dummy data.

So ... it seems that the basic problem here is a mis-understanding of how CopyToDocument interacts with outlining. We pass the IDF_FORMULA flag, but the code then operates on far more than just formulae.

Having fiddled a fair bit here; I think the best sol'n is to add an IDF_OUTLINE - or IDF_NOOUTLINE bit to add in there, to allow the IDF_FORMULA mask to only mutate formulae. Or worse - add some IDF_FORMULAONLY - which would suck ;-) I'll try to come up with something pleasant.
Comment 75 Michael Meeks 2013-08-22 16:52:24 UTC
I just pushed a prototype fix; quite possibly it severely harms the rest of calc [ I hope not, but you never know ], and I'm no calc engineer. Looking forward to some testing of copy / paste / move etc. with outlines around - AFAIK there are no unit tests for this lot (anyone wanting to write them would be more than welcome). Things to look out for when testing:

the code involved tries to restore formulae to the state they were in before the cut/paste - so worth checking some cases of this to make sure they are. Anyhow - I hope it will be in some master builds to play with in due course, and at some stage we can back-port it :-)

Thanks so much for the reproducible test-case; it looks like this issue has been there for years.
Comment 76 Commit Notification 2013-08-22 16:56:12 UTC
Michael Meeks committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e851ea0ed30e9bb95c273a29aeab7f48f606145f

fdo#39484 - don't loose outlines while trying to undo formulae changes.



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 77 Michael Meeks 2013-08-23 11:08:51 UTC
Eike seemed happy with it so it's:

https://gerrit.libreoffice.org/5592

for -4-1. If that doesn't cause any concern after a release, we could back-port it to 4-0 I imagine in a month or so.

Closing for now; do ping in a month if you're desperate for a -4-0 backport.
Comment 78 Winfried Donkers (retired) 2013-08-23 11:49:21 UTC
(In reply to comment #77)
> Eike seemed happy with it so it's:
> 
> https://gerrit.libreoffice.org/5592
> 
> for -4-1. If that doesn't cause any concern after a release, we could
> back-port it to 4-0 I imagine in a month or so.
> 
> Closing for now; do ping in a month if you're desperate for a -4-0 backport.

Thanks a lot for your fast -and informative- digging and fixing!
I will test with master if I can reproduce the problem with the comments getting lost (I hope I can't).
Comment 79 Commit Notification 2013-08-24 04:34:58 UTC
Michael Meeks committed a patch related to this issue.
It has been pushed to "libreoffice-4-1":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=4205ca8003bc5c62a10620c7e54bdf9afb9d86bf&h=libreoffice-4-1

fdo#39484 - don't loose outlines while trying to undo formulae changes.


It will be available in LibreOffice 4.1.2.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 80 Ralf 2013-08-25 17:42:00 UTC
According to my user checks things work fine including comments.

Thank you very much to all contributors for having removed this nasty bug! The bug used to be a real annoyance in my work with LO Calc. I am impressed about the process how it could be reproduced and removed.
Comment 81 Commit Notification 2013-08-26 14:59:59 UTC
Michael Meeks committed a patch related to this issue.
It has been pushed to "libreoffice-4-0":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=07a69320a7b475e8b09b2faca8b6f728a62372ec&h=libreoffice-4-0

fdo#39484 - don't loose outlines while trying to undo formulae changes.


It will be available in LibreOffice 4.0.6.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 82 koukasio 2013-09-12 18:47:07 UTC
I think I have found a workaround for those who use an affected version of lo. If you want to copy exactly one cell/row/column [→element]
copy element
paste special
Shift down/right
cut new element (leaves blank element)
paste
delete blank element (Ctrl+delete)
This way you don't use the undo command. It seems to be working fine but I have not tested this thoroughly. I hope it helps
Comment 83 Commit Notification 2013-09-25 15:56:24 UTC
Markus Mohrhard committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=6f2957969bd72308ddf79cb2befa2373f2dc1dbe

compare against IDF_HARDATTR, fdo#39484



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.