Bug Hunting Session
Bug 85614 - Accelerate 4.3.4 release to resolve sorting issue with 4.2.7 and 4.3.3
Summary: Accelerate 4.3.4 release to resolve sorting issue with 4.2.7 and 4.3.3
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
4.2.7.1 rc
Hardware: All All
: medium critical
Assignee: Not Assigned
URL:
Whiteboard:
Keywords: regression
Depends on: 81633 85405 85584
Blocks: mab4.2 85490
  Show dependency treegraph
 
Reported: 2014-10-29 18:37 UTC by Luke
Modified: 2014-12-04 16:22 UTC (History)
7 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Luke 2014-10-29 18:37:07 UTC
Bug 81309 - Sorting should automatically adjust references is an ill conceived enhancement. It is the cause of serious regressions and incompatibilities that are tracked in Bug 81633 and its many dupes. It was backported to the Stable 4.2 branch in commit 
http://cgit.freedesktop.org/libreoffice/core/commit/?id=d4aed409279d3b9b8b95d84418fb7f279367cc30&h=libreoffice-4-2

Enhancements should not be backported to stable releases and the option to disable it cannot be backported, since there is UI freeze in 4.2.  Therefore, we should revert this commit before 4.2.7 is released to the public.
Comment 1 Joel Madero 2014-10-29 18:42:31 UTC
Please don't add me to these - I have nothing else to add to this sorting issue. Kohei did great work, 4.4 has the option to do what was previously available. Nothing else from me. Removing myself.
Comment 2 Luke 2014-10-29 18:54:29 UTC
> I have nothing else to add to this sorting issue. 

Joel,
This is about preventing the enhancement from causing data loss with people upgrading the "stable" 4.2 branch. It has nothing to do with 4.2.

Also bug 45146 was given as the reason behind the enhancement. You were the one that confirmed and approved the enhancement and set the priority to severe.

https://bugs.freedesktop.org/show_activity.cgi?id=45146
Comment 3 Luke 2014-10-29 19:06:45 UTC
Sorry that should read: It has nothing to do with 4.4.
Comment 4 Michael Meeks 2014-10-29 20:19:11 UTC
So - 4.2.x is end-of-life now. There are of course plenty of bugs & issues with it still - but that will always be the case.

https://wiki.documentfoundation.org/ReleasePlan/4.2

So - modulo any engineering willing to do the work to release an un-scheduled 4.2.8 to fix just this issue, I'd be inclined to move to 4.3.x which has the fix you want - I back-ported it myself.
Comment 5 Luke 2014-10-30 00:38:16 UTC
Michael,

I don't follow. The release plan you linked to clearly shows a planned 4.2.7 release. The current 4.2.7.rc build suffers from this serious regression, while the 4.2.6 available for download does not.

So it was OK to backport a patch implementing an enhancement to the stable branch. But it's NOT OK to revert the patch because it's near EOL? Even though that patch is now damaging peoples spreadsheets( See Bug 81633 and it's dupes)

Or are you saying the release plan has changed?
Comment 6 Michael Meeks 2014-10-30 10:42:25 UTC
Really ? the sorting change was introduced in 4.2.6 ? can you point to the commit ? either way - from what you see in the schedule, 4.2.7 is already done.

Then again reverting that commit is not a huge deal I think; but it's basically up to the release schedule; the 29th is ----way---- late to file this request.
Comment 7 Jean-Baptiste Faure 2014-10-30 11:15:48 UTC
(In reply to Michael Meeks from comment #6)
> Really ? the sorting change was introduced in 4.2.6 ? can you point to the
> commit ? either way - from what you see in the schedule, 4.2.7 is already
> done.

No, not in 4.2.6 but in 4.2.7, see commit notification here :https://bugs.freedesktop.org/show_bug.cgi?id=81309#c10

The last version working as before is 4.2.6.
I already wrote that implementing such change in a bugfix was a bad idea:
https://bugs.freedesktop.org/show_bug.cgi?id=81309#c14
 
> Then again reverting that commit is not a huge deal I think; but it's
> basically up to the release schedule; the 29th is ----way---- late to file
> this request.

I tried to revert this commit on my local git repository and Git said it was not possible.

Best regards. JBF
Comment 8 Michael Meeks 2014-10-30 13:24:48 UTC
Well; I had no idea that this was introduced in 4.2.6 - that does seem late for this change - somehow that completely escaped me during the discussion: I have no clue why. Co-ordination was not ideal around this issue. Ultimately we shipped 4.2.7 today - and it takes days to produce the builds; so ...

Apparently using 4.3.x or a custom build is the solution for now.
Comment 9 Jean-Baptiste Faure 2014-10-30 13:55:35 UTC
(In reply to Michael Meeks from comment #8)
> Well; I had no idea that this was introduced in 4.2.6 - that does seem late
> for this change - somehow that completely escaped me during the discussion:
> I have no clue why. Co-ordination was not ideal around this issue.
> Ultimately we shipped 4.2.7 today - and it takes days to produce the builds;
> so ...

Perhaps my English is too bad to make me understandable, but I tried to warn the ESC on this point 3 weeks ago: 
http://lists.freedesktop.org/archives/libreoffice/2014-October/064033.html
> The problem is not in the change itself, it is in the fact that the
> change has been done in bugfix versions, 4.2.7 and 4.3.1 because ...

and
http://lists.freedesktop.org/archives/libreoffice/2014-October/063906.html

> Apparently using 4.3.x or a custom build is the solution for now.

Currently, if this sorting problem is a blocker for you, you need to stay on 4.2.6.3 or build 4.3.4 yourself (The fixes for bug 85215 have been backported to 4.3 but not to 4.3.3). Or you could become a QA team member and work with the daily-builds of the master.

Best regards. JBF
Comment 10 Michael Meeks 2014-10-30 17:12:24 UTC
Oh; hmm, did we not ship that in 4.3.3 - urk; so yes staying on 4.2.6 is a good idea. I believe Andras volunteered to back-port of the conditional & old default for this from the Collabora 4.2 branch to libreoffice-4-2 so people incapable of doing that cherry-pick for themselves can build it ;-)

Anyhow - thanks for raising that JBF - I'm still -deeply- confused by the apparent problems that remain with the conditional though; is there a tracker bug for open issues around this ?
Comment 11 Luke 2014-10-30 20:07:34 UTC
Jean-Baptiste Faure Wed Oct 8 02:28:20 PDT 2014 wrote
> "In other words, what should I answer on users mailing list when a user
says that his spreadsheets that worked fine since years, suddenly do not
work anymore?"

Since JBF's prescient post on Oct 8, we have had 
Bug 84847
Bug 85405
Bug 85479
Bug 85571

With quotes like "I noticed Calc had corrupted all of my spreadsheets so badly that I needed to restore them from a backup and switched back to an old copy of Excel"

Let's do the right things by our users and fix this regression.
Comment 12 Jean-Baptiste Faure 2014-10-30 20:18:31 UTC
Kohei pushed a backport of the fix for 4.3 for bug 81633: 

https://gerrit.libreoffice.org/gitweb?p=core.git;a=commitdiff;h=67f3ce3a9df2bc62db5602dd84975047c1137b92

It adds an hidden configuration option to 4.2 branch allowing to restore the old behavior of sorting. You will need to build this version yourself because 4.2.7 is already released and is intended to be the last release for the 4.2 branch.

Best regards. JBF
Comment 13 Luke 2014-10-30 21:15:54 UTC
Thanks JBF for following though with the ECS and Kohei for backporting it. I am in the process of building 4.2.7 with the fix now.

Once has undergone testing, how can we get it released? Before anyone else says it's too close to EOL, let me remind people of the rules:

https://wiki.documentfoundation.org/Development/Git_For_LibreOffice_Developers#Managing_git_branches

Backporting the original commit was a clear violation.
Comment 14 Michael Meeks 2014-10-30 21:31:17 UTC
> let me remind people of the rules:

this sort of language is clearly inflamatory and un-necessary. All bug fixes bear the risk of some sort of other knock-on issue.

In fact back-porting this --feature-- to allow this to be configurable is a serious violation of 'the rules' - but was approved by the ESC; I don't think there is any leg to stand on here wrt. complaining about rules =)
Comment 15 Luke 2014-10-30 23:10:17 UTC
Michael,
I don't follow your logic. The rules clearly state that the "only allowed features [are those] approved by the ESC". As this latest one was approved, it's allowed, while the original was not. Even if ESC did not approve, it resolves a serious regression that caused data loss. So again by the rules, a bug fix is allowed. And, how is reminding people of the rules is inflammatory? 

Instead of focusing on whether this is a fix or feature, we should be working to ensure that it goes live as soon as possible. What steps need to be done? I'll spend a few hours testing it tonight.
Comment 16 Michael Meeks 2014-10-31 13:59:08 UTC
> I don't follow your logic. The rules clearly state that the "only allowed
> features [are those] approved by the ESC".

I havn't read it - it seems a shaky fragment of a wiki page to build an argument on; but last I looked: tripple reviewed features by a plurality of entities are allowed at any time without explicit approval.

But that's all moot - this was a bug fix; with a chunk of work done to fix a bug that was filed.

> So again by the rules, a bug fix is allowed. And, how is reminding people
> of the rules is inflammatory? 

Your attitude & approach here is highly counter-productive. This was a bug fix, done in someone's spare time, against a backdrop of aggressive feedback - there being so much of it on every side, that it is hard to tell who is right vs. wrong, what is urgent vs. not and so on.

> Instead of focusing on whether this is a fix or feature, we should be
> working to ensure that it goes live as soon as possible.

So - the bug is resolved WONTFIX for 4.2.x - the real problem seems to be that 4.3.x apparently also ships (apparently) in the unexpected configuration - right ? we could of course ship a new build of that live branch more quickly  *but* there are still vague, unclear claims that the fixed version of 4.3.x has (unspecified) problems with it still. IIWY I'd focus on that issue, and crisping up exactly what (if anything) remains as an issue there - that is useful work for 4.3 / 4.4.
Comment 17 Joey Reid 2014-10-31 21:22:40 UTC
@Michael Meeks

Why are you being an obstructionist on this issue? You are not even a member of the ESC and yet you claim to have the authority to close this serious bug as WONTFIX.

>  a backdrop of aggressive feedback - there being so much of it on every side

Have you even looked at Bug 81633 the source of this Bug 45146? You are wrongly confusing the DOZENS of legitimately upset people whose spreadsheets were damaged with ONE very confused user who admitted he was not skilled with spreadsheets. Despite a developer telling us that using absolute references was the correct way to resolve his sorting issue, this ill-thought idea was allowed to wreak havoc on our user base.

We screwed up here. Instead of fighting people who are trying to clean up this mess, why aren’t you helping?

@Michael Meeks

Why are you being an obstructionist on this issue? You are not even a member of the ESC and yet you claim to have the authority to close this serious bug as WONTFIX.

>  a backdrop of aggressive feedback - there being so much of it on every side

Have you even looked at Bug 81633 the source of this Bug 45146? You are wrongly confusing the DOZENS of legitimately upset people whose spreadsheets were damaged with ONE very confused user who admitted he was not skilled with spreadsheets. Despite a developer telling us that using absolute references was the correct way to resolve his sorting issue, this ill-thought idea was allowed to wreak havoc on our user base.

We screwed up here. Instead of fighting people who are trying to clean up this mess, why aren’t you helping?
Comment 18 Joey Reid 2014-10-31 23:37:38 UTC
Sorry for the double posting. I was in the process of copy/pasting my post to review in Writer when I accidentally and prematurely sent it. And I should have asked does single member have this authority?

When will the Windows version of this fix be made available for testing? It looks like the 4.2 daily only run once a week.
Comment 19 Michael Meeks 2014-11-01 01:34:56 UTC
> @Michael Meeks
> Why are you being an obstructionist on this issue ? You are not even
> a member of the ESC and yet you claim to have the authority to close
> this serious bug as WONTFIX.

I'm not even a member of the ESC ? Interesting - http://lmgtfy.com/?q=michael+meeks+esc

Wrt. WONTFIX; LibreOffice 4.2.7 is already released; putting new patches into it is therefore not possible -> back to WONTFIX. Now - if there was a new bug, titled "Accelerate 4.3.4 release" or somesuch - that might be worth consideration.
Comment 20 Luke 2014-11-03 01:49:55 UTC
I only recently became involved in this issue because I noticed a stream of Ask LO questions about 4.3 breaking their spreadsheets. As a result, I asked a Stanford MBA who works on spreadsheets for a living to explain it to me. He said that there is no need for this feature and that absolute references should be used when users create sortable tables that use lookup values. When I told him it was the new default, he replied that breaking backwards compatibility whimsically will ensure Excel’s continued dominance in the business world. 

THE FACTS:
1) We added a feature that breaks existing Calc and Excel spreadsheets
2) We made this the default behavior
3) We backported this new default behavior to the stable 4.2 branch 
4) Despite JBF's warnings to the ESC 3 weeks ago and this bug report, 4.2.7 was released anyways.

Can we please stop with the excuses and start to take some action to clean up this embarrassing situation before more damage is done?

I spent the weekend testing 4.2.8, and it doesn't fix anything as it defaults to the new behavior and introduces even more sorting bugs.

I've offered to roll forward all of the 100 or so non sorting commits in 4.2.6-4.2.7 myself. Since no one has accepted this offer, Pushing 4.3.4 out the door is probably the best TDF is capable of. Michael, I've changed the title per your request.
Comment 21 Michael Meeks 2014-11-03 12:14:42 UTC
Luke. When I asked for a new bug - you re-titled the existing one. You also continued to use extraordinarily inflamatory: "THE FACTS:" shouting assertions of your view point which is (clearly) not shared by many others.

This sort of behavior leads to extraordinarily irritating developers. That in turn leads to de-prioritising your request; do you actually want an accelerated release of 4.3.next ?

I assume you do. I considered actually filing a new bug - but really our release process is not tracked in bugzilla; so lets close this guy, and I'll add it to the ESC agenda. Cloph - any chance you can look at making some pre 4.3.4 builds speculatively - preferably with just this revert in them.

Luke - I still have -no- answer however, on whether this bug is truly fixed; and doing endless (extremely expensive) rounds of releasing without an answer there would be extremely lame.

> I spent the weekend testing 4.2.8, and it doesn't fix anything as it
> defaults to the new behavior and introduces even more sorting bugs.

Um - this data point is pretty useless to me; what is 4.2.8 ? what git hash were you testing ? did it include Andras' back-port etc. ?

We -badly- need to get some hard data on this.
Comment 22 Luke 2014-11-03 19:39:07 UTC
Michael,
The goal of this bug report has always been to fix sorting behavior caused by http://cgit.freedesktop.org/libreoffice/core/commit/?id=d4aed409279d3b9b8b95d84418fb7f279367cc30&h=libreoffice-4-2 

before it reached users of the stable release. Now that it has, it's now a matter of getting it fixed as quickly as possible. In the facts, if I said anything is factually incorrect, I apologize. These mistakes are causing data loss. Devs, QA, and end users are all irritated by this issue. Let's work on fixing it.

> Luke - I still have -no- answer however, on whether this bug is truly fixed; 
> and doing endless (extremely expensive) rounds of releasing without an answer 
> there would be extremely lame.

Fixed in what? You want us testing 4.2.8 or 4.3.4?

> Um - this data point is pretty useless to me; what is 4.2.8 ? what git hash
> were you testing ? did it include Andras' back-port etc. ?

I did a $ git checkout libreoffice-4-2 
which has a Build ID: 67f3ce3a9df2bc62db5602dd84975047c1137b92
This includes Andras' back-port. The default behavior is still the new sorting behavior minus the sorting bug fixes that are in 4.3.4.

Kohei commited a patch to 4.2.8 a few hours ago. If you'd like, I'll test that out tonight.
Comment 23 Michael Meeks 2014-11-03 20:39:31 UTC
> These mistakes are causing data loss. Devs, QA, and end users are all irritated
> by this issue. Let's work on fixing it.

That's a good place to be; thanks for being constructive =)

>> Luke - I still have -no- answer however, on whether this bug is truly fixed; 
>> and doing endless (extremely expensive) rounds of releasing without an answer 
>> there would be extremely lame.
>
> Fixed in what? You want us testing 4.2.8 or 4.3.4?

Well - we need to test 4.3.<latest> from the latest snapshots for that branch - ultimately, that's what we'd be releasing I think - so the most hammering you can give that is appreciated. Then again - we're really aiming not for something perfect - but something as substantially better as we can get it in a short time I think =)

> Kohei commited a patch to 4.2.8 a few hours ago. If you'd like, I'll
> test that out tonight.

I reviewed & cherry-picked it a few hours ago indeed; it had been waiting a while - but I think its best to focus on 4.3.x for now.
Comment 24 Joel Madero 2014-11-06 15:08:24 UTC
Please stop reporting these - we're NOT going to accelerate release. Luke...I could be wrong but I think you personally keep pushing this. ESC has talked about this as length - we're not accelerating a release for a single issues. Closing as WONTFIX - again ,don't open this, it's been discussed at length and it's not going to happen. Continuing to push for it isn't going to change that.
Comment 25 Joel Madero 2014-11-06 16:11:12 UTC
...apparently I might be wrong so at least moving back to NEEDINFO and letting Michael move it forward however the consensus sees fit. Apologies for the noise.
Comment 26 Michael Meeks 2014-11-06 16:26:30 UTC
Thanks Joel - I too have mixed feelings on this; the plan is to do an RC1 of 4.3.next early - and to ask QA to do some heavy testing on it: if that flies then  we may release that a 4.3.next out of cycle - but it's all tangled around the 4.4 release schedule - this comes at an -incredibly- bad time owing to the looming feature freeze.
Comment 27 Joel Madero 2014-11-06 16:39:29 UTC
That being said - I hope that every person who has brought this issue up and pushed it so hard takes the time to download RC1 and test it like crazy (not for the sorting issue, but for issues generally)
Comment 28 benico 2014-11-13 09:01:57 UTC
Hi Guys.  If apache office had a better conditional formatting interface I would have switched.  But anycase because of this issue I've downgraded all the Windows versions at work to 4.3.0 - and personally I'm still on 4.1.2.3 (which does not throw away the call out boxes in my graphs).  
Thanks for working to solve this ;-)
Comment 29 Jean-Baptiste Faure 2014-11-15 15:29:49 UTC
LibreOffice 4.3.4 has been released one week before it was announced in the release plan. So closing as Resolved Fixed.

Best regards. JBF
Comment 30 Michael Meeks 2014-11-15 16:27:48 UTC
> LibreOffice 4.3.4 has been released one week before it was announced
> in the release plan. So closing as Resolved Fixed.

If you carefully examine the release schedule from before it was adjusted to take account of the more rapid release, checkout for example:

https://wiki.documentfoundation.org/index.php?title=ReleasePlan/4.3&oldid=88804#4.3.4_release

cf.
"Release 4.3.4	Week 51 , Dec 15, 2014 - Dec 21, 2014"

You can see that we shipped dramatically earlier than we would have done; around a month earlier. That we also shipped more quickly than even the adjusted release schedule is (I hope) appreciated and is a credit to Cloph.
Comment 31 Eike Rathke 2014-12-04 16:22:48 UTC
Moved the "see also" list to "Depends on" to be able to see the dependency tree view.