Bug 66969 - PIVOTTABLE: lost data when refreshing
Summary: PIVOTTABLE: lost data when refreshing
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
3.6.0.4 release
Hardware: Other All
: medium major
Assignee: Kohei Yoshida
QA Contact:
URL:
Whiteboard: target:4.3.0 target:4.2.0.1 target:4.1.5
Keywords: regression
Depends on:
Blocks:
 
Reported: 2013-07-16 16:42 UTC by Petr Mladek
Modified: 2014-01-20 20:05 UTC (History)
4 users (show)

See Also:
Crash report or crash signature:


Attachments
Screenshot after refresh. (55.55 KB, image/png)
2013-12-17 01:48 UTC, m.a.riosv
Details
Sample Spreadsheet that (still) exhibits the faulty behavior (123.82 KB, application/vnd.oasis.opendocument.spreadsheet)
2014-01-20 19:05 UTC, Sebastian Goetze
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Petr Mladek 2013-07-16 16:42:01 UTC
I have a document when the pivottable get malformed when you refresh it.

Steps to reproduce:

    1. Open the test document "Pivot table with data groups.ods" from the bug #63998.
        You can get it at https://bugs.freedesktop.org/attachment.cgi?id=78555

    2. Press the right mouse button in the cell [A5] and select the menu entry "Refresh"

   Result: The pivottable gets squashed to 3 rows and 2 colums. It does not show any data.

   Expected Result: The pivottable stay as is.


Observation: 

    + I see the problem with LO 4.1.0.3 and 4.0.4.2 on Linux x86_64 => it is already in 4.0

    + I do not see the problem when I do any other action with the pivot table before refreshing;
       for example, when I change the months filtering in the cell [B7] before doing the refresh
Comment 1 Petr Mladek 2013-07-16 16:46:25 UTC
I see the problem also with 3.6.6.1 build but not with 3.5.6.2 => it is a relatively old regression.
Comment 2 Jorendc 2013-07-26 20:23:31 UTC
With Linux Mint 15 x64 with LibreOffice Version: 4.2.0.0.alpha0+
Build ID: 7fd81244c21ad54a8b9766902fd7c34e8055b165 I now have the invert effect. When refreshing cell A5 -> all information expands.
Comment 3 Eike Rathke 2013-07-26 21:24:15 UTC
I think that is because the original filter was lost thus expands to all, selecting a filter before refreshing works. Not sure if anything can be done there, Cc'ing Kohei on the SuSE address that might work better ;-)
Comment 4 Kohei Yoshida 2013-07-29 13:35:24 UTC
Don't bother with bibisect on this one. Bibisect doesn't help much with heavily refactored code which the pivot table code is.
Comment 5 Kohei Yoshida 2013-08-13 19:31:33 UTC
So, Eike's comment in comment 3 pretty much nails it.  We just need to figure out why we are not loading the page field filtering info correctly.
Comment 6 Sebastian Goetze 2013-09-21 17:08:08 UTC
I also have a sheet with a filtered pivottable, losing the filter every time I open it. It's fairly complex and contains private (financial) info, that's why I was hesitant to try to simplify it and check for and file a bug. This one sounds a lot like my problem, though, so I just chime in:
I tried to narrow down, where the problem is and created 2 fairly identical pivot tables, (1) one in the middle of a sheet, using a fixed range of cells as a source, (2) one on a 'new sheet' using a 'named range'. Slightly different results...

* When I open the file, in both cases the filter is 'damaged': (3 conditions)
(1) In the first two conditions the fields are replaced by the very first field of the table, the third condition stays correct
(2) The first two conditions just like in case (1), the 3rd condition is 'empty'. When trying to correct the field, it seems to have lost the 'named range', only the first field is available to choose from, all other field are gone. When I 'edit' the pivot table, the named range is still correctly displayed, though. After re-selecting the named range, the fields finally are available again, to correct the filter.

So we might have 2 bugs here: Losing the filter and losing the named range...
2 bugs? I never noticed a problem refreshing a pivot table with a named range, though...

Hope that helps. Forgive my rambling, it's my first contribution to a bug report.
Comment 7 Commit Notification 2013-12-10 23:43:21 UTC
Kohei Yoshida committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=2e1b90a4272defb917b23e2e360e171114d6fa4d

fdo#66969: Set selected page name after building all dimension members.



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 8 Commit Notification 2013-12-10 23:43:38 UTC
Kohei Yoshida committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=b3977983e9f662392426f581516d86d7034ad0fd

fdo#66969: Reset group dimension data from all referencing pivot objects.



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 9 Commit Notification 2013-12-10 23:45:59 UTC
Kohei Yoshida committed a patch related to this issue.
It has been pushed to "libreoffice-4-2":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=a0fd13c0651acf0e620a58eca248b3a12ea43632&h=libreoffice-4-2

fdo#66969: Set selected page name after building all dimension members.


It will be available in LibreOffice 4.2.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 10 Commit Notification 2013-12-10 23:46:16 UTC
Kohei Yoshida committed a patch related to this issue.
It has been pushed to "libreoffice-4-2":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=61a91e69eb627a86a7358e3d65a0892ac8dc4d9d&h=libreoffice-4-2

fdo#66969: Reset group dimension data from all referencing pivot objects.


It will be available in LibreOffice 4.2.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 11 Commit Notification 2013-12-11 01:54:12 UTC
Kohei Yoshida committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e8fff12af2c0bc3172c3db830b5f6a59869e5be0

fdo#66969: Add test to ensure we import page field's visibility correctly.



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 12 Kohei Yoshida 2013-12-11 01:56:45 UTC
I'll mark this fixed.
Comment 13 Commit Notification 2013-12-11 16:11:09 UTC
Kohei Yoshida committed a patch related to this issue.
It has been pushed to "libreoffice-4-1":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=5618ac8ccddbff67b4b27a9c08f97d3bf2bcec3f&h=libreoffice-4-1

fdo#66969: Set selected page name after building all dimension members.


It will be available in LibreOffice 4.1.5.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 14 Commit Notification 2013-12-11 16:11:25 UTC
Kohei Yoshida committed a patch related to this issue.
It has been pushed to "libreoffice-4-1":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=b8339a06050fcbff25e3c990e1ff8ca02dc7bad5&h=libreoffice-4-1

fdo#66969: Reset group dimension data from all referencing pivot objects.


It will be available in LibreOffice 4.1.5.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 15 Sebastian Goetze 2013-12-16 22:47:51 UTC
I tried the following two builds with identical (bad) results:
libo-42~2013-12-14_23.15.05_LibreOfficeDev_4.2.0.0.beta2_Win_x86
libreoffice-4-1~2013-12-14_03.37.58_LibreOfficeDev_4.1.5.0.0_Win_x86

I had tried previous build last week, but wanted to make sure that the patch is in...

Results (both the same):
Open the file, Pivot Table is expanded (from previous edit)
Hit 'Refresh', Pivot Table collapses.
Check Filter: Filter 'gone' (Fieldname in 1st and 2nd condition reset to first field in table, 3rd condition stayed fine. Now no record fulfills filter...)
Correct Filter (fill in correct fields), Pivot Table expands to show 'unpaid invoices' as it should.
Save in new Version.
Close.
Open again.
Pivot table is expanded, hit 'refresh', Filter 'disappears' again, PT collapses.


IN ADDITION THERE's A REGRESSION:
De-selecting entries in my 'Page Field' doesn't work anymore.
It IS possible to select *new* ones, until you have 'all' selected.
But you never get rid of them again.

Should I try and simplify/narrow down my file?
Would you like screenshots?

BTW, this error appeared in my sheet with version 3.6, so I'm pretty sure, it's the one the original poster also noticed.

IMHO this bug is not yet fixed...

Sebastian
Comment 16 Kohei Yoshida 2013-12-16 23:49:14 UTC
Try the next build then. This one *is* fixed & it takes more than just a few days for the fix it make it into a public build.
Comment 17 Sebastian Goetze 2013-12-16 23:53:03 UTC
Sorry Kohei,

just want to be helpful...
I was going by your 24-48 hrs timeline.

I'll try later this week and report back.

Thanks for your hard work  :-)
Comment 18 Kohei Yoshida 2013-12-17 00:09:44 UTC
I can only go by git tags, and the latest tag on the 4.1 line points to

http://cgit.freedesktop.org/libreoffice/core/commit/?id=0a0440ccc0227ad9829de5f46be37cfb6edcf727 (libreoffice-4.1.4.2)

which is dated 2013-12-10, one day before my commit.

And as for 4.2.0 beta2 

http://cgit.freedesktop.org/libreoffice/core/commit/?id=1a27be92e320f97c20d581a69ef1c8b99ea9885d (libreoffice-4.2.0.0.beta2)

which is dated 2013-12-03.

I'm not sure if the builds you've tried are built on those tags, but if they are, clearly they don't include my fixes.

I don't track daily builds too closely to know whether they are built from the latest commits in each branch.  I wish there was an easier way to see which build was built from which commit.  The About dialog used to contain the last commit hash to make cross-checking of this kind easier, but I guess we've decided to remove that at some point. :-/
Comment 19 Sebastian Goetze 2013-12-17 00:22:55 UTC
Build ID from the 4.2 version is:
Version: 4.2.0.0.beta2+
Build ID: d663228bd348c844f38914c9e2167ef01fadf3b3
TinderBox: Win-x86@42, Branch:libreoffice-4-2, Time: 2013-12-14_23:15:0

That's why I assumed that your patch from 4 days earlier was in...
Comment 20 Kohei Yoshida 2013-12-17 00:39:30 UTC
Ok. That commit should contain my fixes. But I just double-checked my local build which is up-to-date and I still can't reproduce it.

Can you ask others to check as well?
Comment 21 m.a.riosv 2013-12-17 01:48:08 UTC
Created attachment 90853 [details]
Screenshot after refresh.

This is what I see after refresh with:
Win7x64Ultimate
Version: 4.2.0.0.beta2+ Build ID: d663228bd348c844f38914c9e2167ef01fadf3b3
         TinderBox: Win-x86@42, Branch:libreoffice-4-2, Time: 2013-12-14_23:15:05
I think it is fine.
Comment 22 Sebastian Goetze 2014-01-20 19:05:17 UTC
Created attachment 92477 [details]
Sample Spreadsheet that (still) exhibits the faulty behavior
Comment 23 Sebastian Goetze 2014-01-20 19:07:05 UTC
I was checking 4.1.5 RC1 and it still exhibited almost the same faulty behavior:
Just instead of the first two filter fields being forgotten, it's just the first one now.

I reduced my spreadsheet to the bare minimums and attach it to this comment, same as the 'corrected' filter (how it looked, when I saved the file).

Steps to reproduce:
* Open the file PivotProblem.ods
* refresh the Pivot Table in tab 'due'

=> Table collapses

Steps to resolve:
* Open the Pivot-Filter (e.g. Double-Click on 'Filter')
* Change the first 'Field' to "Eingang", like shown on the attached pic
* Hit 'OK' => table is back


Additional problem as described before (different bug?):
* Try deselect values in the column filter (e.g. 'debitor')
=> the only ones you can de-select are the ones not present (e.g. 'empty')

I hope this helps narrow it down...



I tried to also check with 4.2, but it always died on me:
Problem signature:
  Problem Event Name:	BEX
  Application Name:	soffice.bin
  Application Version:	4.2.1.0
  Application Timestamp:	52d1dc8d
  Fault Module Name:	MSVCR110.dll
  Fault Module Version:	11.0.51106.1
  Fault Module Timestamp:	5098858e
  Exception Offset:	000a326c
  Exception Code:	c0000409
  Exception Data:	00000007
  OS Version:	6.3.9600.2.0.0.768.101
  Locale ID:	1033
  Additional Information 1:	778a
  Additional Information 2:	778a9cc615938f1d841c2208f97969c9
  Additional Information 3:	54b4
  Additional Information 4:	54b4cb5ea792e74cdfe13da12a366cd2

Probably a different problem, though...
Comment 24 Kohei Yoshida 2014-01-20 19:25:57 UTC
File a new bug please.
Comment 25 Sebastian Goetze 2014-01-20 20:05:31 UTC
(In reply to comment #24)
> File a new bug please.

I opened bug 73845...