Bug Hunting Session
Bug 68274 - provide better update mechanism -- Mozilla ARchive (mar) based incrementals
Summary: provide better update mechanism -- Mozilla ARchive (mar) based incrementals
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Installation (show other bugs)
Version:
(earliest affected)
Inherited From OOo
Hardware: All All
: medium enhancement
Assignee: Not Assigned
URL: https://wiki.documentfoundation.org/D...
Whiteboard:
Keywords:
: 39028 83406 86985 92835 99951 (view as bug list)
Depends on:
Blocks: Automatic-Updater
  Show dependency treegraph
 
Reported: 2013-08-19 13:10 UTC by retired
Modified: 2019-06-04 13:47 UTC (History)
34 users (show)

See Also:
Crash report or crash signature:


Attachments
screenshot of standard sparkle options (18.85 KB, image/png)
2013-08-19 13:10 UTC, retired
Details

Note You need to log in before you can comment on or make changes to this bug.
Description retired 2013-08-19 13:10:46 UTC
Created attachment 84246 [details]
screenshot of standard sparkle options

Current behavior: existing update check is confusing and does not allow direct download let alone easy install.

Expected behavior: LibreOffice should use Sparkle on OS X with three appcasts files. The channels could be analogue to Firefox Stable / beta / nightly.

If you want to see sparkle in action it is integrated into tons of software on OS X and free to use:
https://github.com/andymatuschak/Sparkle

E.g. in Transmission.

Allow users how often to check for updates and which release channel they want to be on.

              
Operating System: Mac OS X
Version: 3.3 all versions
Comment 1 Jorendc 2013-08-19 13:14:13 UTC
Looks a valid enhancement request to me.
Comment 2 Emir Sarı (away) 2013-08-22 05:19:11 UTC
OH, I want this so badly!
Comment 3 retired 2013-10-15 09:04:00 UTC
Since users running outdated versions of LO and this would be a great solution to the problem (at least for OS X) I raise importance to "high".
Comment 4 William Entriken 2014-07-15 18:22:11 UTC
There is additional discussion on the Sparkle project.  https://github.com/sparkle-project/Sparkle/issues/380 

Some Sparkle developers and hackers want to help make this happen.

Are there some experts here that can help explain build details for Mac?
Comment 5 retired 2014-07-16 05:57:47 UTC
kendy made a very smart suggestion: why not look into using the Mozilla MPL licensed Software Update mechanism?

https://wiki.mozilla.org/Software_Update

Major benefit would be, that it is (hopefully) cross-platform and thus would improve the update UX on all OS.
Comment 6 Adolfo Jayme 2014-07-17 12:38:22 UTC
Also, note that there is an implementation of Sparkle for Windows: http://winsparkle.org/
Comment 7 Sierk Bornemann 2014-07-21 07:21:21 UTC
(In reply to comment #5)
> kendy made a very smart suggestion: why not look into using the Mozilla MPL
> licensed Software Update mechanism?
> 
> https://wiki.mozilla.org/Software_Update
> 
> Major benefit would be, that it is (hopefully) cross-platform and thus would
> improve the update UX on all OS.

Sparkle on OSX just WORKS. Mozillas Software Update _still_ has problems on OSX to update Firefox as a non-admin user.
Comment 8 Adolfo Jayme 2014-09-04 00:07:54 UTC
*** Bug 83406 has been marked as a duplicate of this bug. ***
Comment 9 Adolfo Jayme 2014-12-04 16:41:28 UTC
*** Bug 86985 has been marked as a duplicate of this bug. ***
Comment 10 brij 2015-03-02 20:11:28 UTC
Hi All,

I would like to start working on this bug. I went through the associated bugs and comments. According to them (stating the obvious), we need to re-use mozilla's update system's code and provide auto-update.

Kindly point me to the direction where I can start.

Thanks a lot!
Comment 11 Jan Holesovsky 2015-03-05 16:20:23 UTC
brij: If the answer is that the Mozilla's approach is the best, first we need to collect the code pointers to their implementation; so can you please find out the code that:

1) asks for updates in Firefox, and downloads the new stuff?

2) the code in Firefox that applies the update?

3) the code that builds a suitable update pack, and

4) the code for the server part that distributes the info that there is a new
  update

When you have this, we need to plug that into the LibreOffice infrastructure accordingly.  For the 1) and 2), that would be have to be plugged into extensions/source/update/* , for 3), that would be something to plug into our release process, and 4) would be for the infra team.

Can you please add the code pointers here?  If you struggle with that, please let me know, I'll help searching :-)  Thanks in advance!
Comment 12 Nathan Yee 2015-03-12 05:02:31 UTC
I am interested in working on this bug in advance of GSoC. I've found the code for the updater via Mozilla IRC, but I did not have time to look it through.

https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/updater
Comment 13 brij 2015-03-12 12:23:58 UTC
Nathan: You might want to start by looking into nsIUpdateService.idl . This is the UI file and gives you an idea of the flow.
Comment 14 cen.is.imba 2015-03-12 23:59:11 UTC
I haven't looked into the OS X part yet but according to Mozilla wiki, on Windows it pretty much boils down to:

1. Download update in the background
2. Start a maintenance service which avoids UAC prompt (I think this works because you install the service as privileged, something along those lines)
3. Close Firefox, patch it with service, bring it back, close service (through some hackery)


As I see it, all that really needs to be done is to change downloads to be done in the background and move the patching code into a service. I'm not sure any special framework or major changes are needed, just a separation of the patching process. Any thoughts on that?
Comment 15 Michael Bauer 2015-03-13 00:04:56 UTC
One thing that needs double checking is whether the process correctly picks up the installed LO locale(s) and maintains that. It has a nasty habit of switching back to English.
Also the current procedure is that you download the default installer with ALL locales. This is not efficient, esp on bad bandwidth, so (though it could be a developmental feature for 1.1) is to only pull the patch for the locales required.
Comment 16 Laurent Godard 2015-03-13 08:21:58 UTC
enhancement proposal

checking new LibreOffice version and installed Extensions seem to be coupled

Dealing with libreoffice update, these 2 actions should be separated and an other enhancement would be specifying the checking/update policy per extension

but as first step let only separate libreoffice update notification and Extensions as a whole
Comment 17 Dennis Roczek 2015-03-13 10:03:46 UTC
enhancement proposal (well for version 1.2 or so ^^) for windows:
checking for offline help and update to newest version. ;-)
Comment 18 Jan Holesovsky 2015-03-13 10:21:26 UTC
Please let's not pollute this bug with enhancement requests; if you want to file any, create separate bugs for that, and make this one depend on them :-)

Thank you!
Comment 19 Michael Bauer 2015-03-13 10:53:28 UTC
This is not an enhancement. It is a prerequisite. How do you think the millions of non-English LO will feel if we implement an automatic update which resets them to English by accident or design? Making sure the new update does NOT mess up the locale is a showstopper. 

Thank you.
Comment 20 Michael Bauer 2015-03-13 11:18:15 UTC
Pls ignore my last comment, I misread. :$
Comment 22 Nathan Yee 2015-05-26 03:14:04 UTC
I have started looking for code pointers relevant to this bug. Here is what I have currently found:

The files that we are interested in are primarily nsUpdateService.js, nsIUpdateService.idl, and possibly nsUpdateServiceStub.idl.

How updates are created:
- select best update from a list of updates:
    IDL: nsIUpdate selectUpdate([array,
                                 size_is(updateCount)] in nsIUpdate updates,
                                 in unsigned long updateCount)
    JS:
      /**
       * Determine the update from the specified updates that should be offered.
       * If both valid major and minor updates are available the minor update will
       * be offered.
       * @param   updates
       *          An array of available nsIUpdate items
       * @return  The nsIUpdate to offer.
       */
      selectUpdate: function AUS_selectUpdate(updates) {...}
- download the update:
    IDL: AString downloadUpdate(in nsIUpdate update, in boolean background)
    JS:
      /**
       * Download and stage the given update.
       * @param   update
       *          A nsIUpdate object to download a patch for. Cannot be null.
       */
    downloadUpdate: function Downloader_downloadUpdate(update) {...}
    - lines 4232-4252 in nsUpdateService.js are significant

- apply the update:
    IDL: void applyOsUpdate(in nsIUpdate update)
For more info on the update service check nsIUpdateService.idl (lines 350-456).

The functions appear to rely on an nsIUpdate object. It's interface is in nsIUpdateService.idl as 'nsIUpdate'.

Infra needed:
- general information about updates: https://wiki.mozilla.org/Software_Update
- information about Windows Silent Update Service: https://wiki.mozilla.org/Windows_Service_Silent_Update
- XML format for determining updates: https://wiki.mozilla.org/Software_Update:updates.xml_Format
We will need to look further into this later.
  
Misc:
- code performing check:
    In the Checker prototype, getUpdateURL(force) gets the URL to the XML update details file on a server, checkForUpdates(listener, force) checks for updates.
- code triggering update install is detailed in the "How updates are created" section
Comment 23 Nathan Yee 2015-05-27 19:30:49 UTC
Current information regarding this issue can be found here: https://wiki.documentfoundation.org/Development/Online_Update
Comment 24 Commit Notification 2015-07-22 05:20:43 UTC
Nathan Yee committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=b59955bf2124b558147033782bf067a3723e4730

online update tdf#68274: fix --enable-online-update=mar on Windows

It will be available in 5.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 25 Shahar Or 2016-02-10 13:23:25 UTC
*** Bug 39028 has been marked as a duplicate of this bug. ***
Comment 26 V Stuart Foote 2016-05-19 20:01:14 UTC
*** Bug 99951 has been marked as a duplicate of this bug. ***
Comment 27 V Stuart Foote 2016-05-19 20:23:46 UTC
clearing the Whiteboard target entry, that was Nathan Yee's preliminary work setting up Mozilla's Mozilla ARchive, or "mar", incremental binary updating--still lots to be done.

Clearly has moved beyond an OS X only issue. And not clear why this would not simply be closed duplicate of bug 54242 -- or better close that to this.

=-ref-=
status here
https://wiki.documentfoundation.org/Development/Online_Update
https://wiki.mozilla.org/Software_Update

https://gerrit.libreoffice.org/#/c/17053/
Comment 28 steve -_- 2016-05-19 21:19:00 UTC
Stuart: this bug was OS X only, since the OP (me) suggested to use Sparkle to deal with updates. The sparkle framework is OS X only.

If TDF / LO core devs decide to instead work on the mozilla update mechanism that is fine and then of course it is OS agnostic (iirc).

Sparkle: https://github.com/sparkle-project/Sparkle
Comment 29 steve -_- 2016-05-19 21:19:46 UTC
ups. sorry, OP not me, but I am sure I filed a dupe a long time ago ;)
Comment 30 V Stuart Foote 2016-05-19 22:27:27 UTC
@Steve, looks like it went cross platform from comment 5 onward...
Comment 31 Sierk Bornemann 2016-05-19 22:35:16 UTC
(In reply to steve -_- from comment #28)
> The sparkle framework is OS X only.

Sparkle OS X: Sparkle
https://sparkle-project.org/
https://github.com/sparkle-project/Sparkle

Sparkle Windows: WinSparkle
https://winsparkle.org/
https://github.com/vslavik/winsparkle
Comment 32 V Stuart Foote 2016-06-28 19:34:31 UTC
*** Bug 92835 has been marked as a duplicate of this bug. ***
Comment 33 V Stuart Foote 2016-06-28 19:36:08 UTC
*** Bug 83406 has been marked as a duplicate of this bug. ***
Comment 34 Mike Kaganski 2016-10-04 23:43:02 UTC
Does the work on MAR-based LO updater for Windows account for the case when LO for Windows is initially installed using MSI (the prevalent majority case, mandatory for enterprise deployment)?

I ask because the updater should not break mandatory MSI functionality, such as:

1. Add/remove components of installed software. This could be broken if we simply replace some files in installation directory:
1.1. If we simply replace a file 1-to-1, then removing a feature with that file and then adding it again will restore old file from MSI kept in Windows\Installer directory, which is undesirable at least, and can prevent from working because of incompatibilities of different versions of different files trying to work together. Also, installer may detect different version of a file, and work improperly;
1.2. If an update wants to add new files or remove existing, this is another sort of problems for installer trying to add/remove features.

2. Uninstall the software: if the updater had added new files to installation, then uninstaller must not have those files (that it knows nothing about) left in Program Files after uninstall (or it will be bad UX).

3. Repair/reinstall: I suppose it's obvious what problems may be encountered here, least of those is unexpected reverting to an older version.
Comment 35 Nathan Yee 2016-11-17 02:44:25 UTC
(In reply to Mike Kaganski from comment #34)
> Does the work on MAR-based LO updater for Windows account for the case when
> LO for Windows is initially installed using MSI (the prevalent majority
> case, mandatory for enterprise deployment)?
> 
> I ask because the updater should not break mandatory MSI functionality, such
> as:
> 
> 1. Add/remove components of installed software. This could be broken if we
> simply replace some files in installation directory:
> 1.1. If we simply replace a file 1-to-1, then removing a feature with that
> file and then adding it again will restore old file from MSI kept in
> Windows\Installer directory, which is undesirable at least, and can prevent
> from working because of incompatibilities of different versions of different
> files trying to work together. Also, installer may detect different version
> of a file, and work improperly;
> 1.2. If an update wants to add new files or remove existing, this is another
> sort of problems for installer trying to add/remove features.
> 
> 2. Uninstall the software: if the updater had added new files to
> installation, then uninstaller must not have those files (that it knows
> nothing about) left in Program Files after uninstall (or it will be bad UX).
> 
> 3. Repair/reinstall: I suppose it's obvious what problems may be encountered
> here, least of those is unexpected reverting to an older version.

I do not know if the work I did on the updater accounts for these changes. It has been a while since I have worked on the updater, and in hindsight I believe that it would have been a better idea to use an existing software updating framework instead of porting Mozilla's updater. From what I remember seeing, their implementation was tightly coupled to Firefox and thus I think it would have been much cleaner and safer to use a different approach for an LO update mechanism.

I would personally recommend scrapping the existing work on the updater and looking into an alternative framework such as Sparkle.
Comment 36 Jan Holesovsky 2016-11-18 10:42:50 UTC
Unfortunately Sparkle was not cross-platform :-(
Comment 37 Sierk Bornemann 2016-11-18 12:20:37 UTC
(In reply to Jan Holesovsky from comment #36)
> Unfortunately Sparkle was not cross-platform :-(

Sparkle for macOS: Sparkle
https://sparkle-project.org/
https://github.com/sparkle-project/Sparkle

Sparkle for Windows: WinSparkle
https://winsparkle.org/
https://github.com/vslavik/winsparkle

Linux:
Updates via distribution's package management anyway


One user of Sparkle (at least on the macOS platform) for its cross platform software:

* Java Plugin/JRE from Oracle

Other cross platform software projects, which also are using Sparkle for updating itself (at least on the macOS platform):

* Transmission
* VLC Media Player
[…]

How did all these projects achieve that in doing so and in doing so well and in doing so cross platform?

What have they done to achieve that, what LibreOffice can't do and achieve either?
Comment 38 Buovjaga 2016-11-18 13:25:35 UTC
(In reply to Sierk Bornemann from comment #37)
> (In reply to Jan Holesovsky from comment #36)
> > Unfortunately Sparkle was not cross-platform :-(
> 
> Sparkle for macOS: Sparkle
> https://sparkle-project.org/
> https://github.com/sparkle-project/Sparkle
> 
> Sparkle for Windows: WinSparkle
> https://winsparkle.org/
> https://github.com/vslavik/winsparkle

Two different products, not a single cross-platform one.

Markus M. will work on getting the current updater in shape. Let's not get carried away. There's plenty of LibreOffice work we can direct our energy to while waiting.
Comment 39 Buovjaga 2016-11-18 13:43:49 UTC
(In reply to Buovjaga from comment #38)
> Markus M. will work on getting the current updater in shape. Let's not get
> carried away. There's plenty of LibreOffice work we can direct our energy to
> while waiting.

Quick update: Markus just pushed the updater to a feature branch: https://cgit.freedesktop.org/libreoffice/core/log/?h=feature/mar-updater
Comment 40 Adolfo Jayme 2016-11-28 14:21:23 UTC
(In reply to Nathan Yee from comment #35)
> I would personally recommend scrapping the existing work on the updater and
> looking into an alternative framework such as Sparkle.

That’s what I’ve been saying since before this project started. It’s sad that nobody seems to listen.
Comment 41 steve -_- 2016-11-28 14:42:33 UTC
I strongly support sparkle usage. The counter argument atm from devs is, that sparkle is not cross platform. And then again, snap-builds (whatever that is) on Linux is dealt with and that's not cross from what I can tell.

Obviously work is happening since moggi pushed the mar updater for windows to a feature branch. That branch also has recent action. So work is being done at this time.

Downside for macOS: moggi is not interested to work on any updater stuff regarding macOS. That's his preferences and I respect that. But I wonder, how updates on macOS then can be moved forward into some constructive direction.

Adolfo are you in touch with QA / Dev department? We all agree this is an important problem in LO I guess. Figuring out options to move this forward would be great.
Comment 42 Markus Mohrhard 2017-01-02 19:27:02 UTC
(In reply to Adolfo Jayme from comment #40)
> (In reply to Nathan Yee from comment #35)
> > I would personally recommend scrapping the existing work on the updater and
> > looking into an alternative framework such as Sparkle.
> 
> That’s what I’ve been saying since before this project started. It’s sad
> that nobody seems to listen.

The mar based updater code is actually quite good. I'm not sure why people recommend to use something else without having had a look into the code.

The only problems with the mar based updater are infra problems. Otherwise the code solves many problems that big and complex projects like LibreOffice or Firefox have. Additionally the Mozilla security team did a quite good analysis of the problem space and raised many security issues that are fixed. Does anyone here volunteer to do the same security audit for any other system?

I'm happy to hand over the work on an automated updater if someone wants to deal with all these boring low level platform details.
Comment 43 Markus Mohrhard 2017-01-03 01:27:27 UTC
(In reply to steve -_- from comment #41)
> I strongly support sparkle usage. The counter argument atm from devs is,
> that sparkle is not cross platform. And then again, snap-builds (whatever
> that is) on Linux is dealt with and that's not cross from what I can tell.


How are they related? There is a huge difference between some release format (they are already all platform dependent) and introducing 3 platform dependent update mechanisms. This means 3 times the code and 3 times the attack surface.

> 
> Obviously work is happening since moggi pushed the mar updater for windows
> to a feature branch. That branch also has recent action. So work is being
> done at this time.
> 
> Downside for macOS: moggi is not interested to work on any updater stuff
> regarding macOS. That's his preferences and I respect that. But I wonder,
> how updates on macOS then can be moved forward into some constructive
> direction.

Yes, I will not do the work necessary to enable the feature on OSX. However the code is there and you just need to find a developer to spend a bit of time enabling the feature. It is not that it would not work, just that I will not spend my free time debugging platform problems and dealing with OSX. Mozilla's updater code works on OSX and all the platform bits are already in our repo.


> 
> Adolfo are you in touch with QA / Dev department? We all agree this is an
> important problem in LO I guess. Figuring out options to move this forward
> would be great.

Did anyone here who says that the mar based updater is a bad idea and sparkle is such a good idea have ever looked into the code. Especially all the security stuff related to installing automated updates. Do you really want to audit 3 different frameworks? Do you really suppose that you will find developers willing to maintain 3 different update frameworks. I suppose I have better chances convincing all other developers to just drop OSX support.
Comment 44 Markus Mohrhard 2017-01-03 02:11:52 UTC
And to make sure it is not only a rant but also shows some of the things that any alternative would need to provide:

* cross-platform, at least Windows and Linux, OSX is a plus
* simple signing of the update files
* silent updater for limited user accounts on windows (https://bugzilla.mozilla.org/show_bug.cgi?id=711475)
* verification of updater, updater service, and update file signature (e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=709173, https://bugzilla.mozilla.org/show_bug.cgi?id=704285, https://bugzilla.mozilla.org/show_bug.cgi?id=708854)
* a way to have the silent update service optional
* version checks for the update files (on the client side, needs to happen in the updater)(https://bugzilla.mozilla.org/show_bug.cgi?id=708688)
* update service should drop all unnecessary permissions (might execute untrusted stuff)
* a decent security review (e.g. at least the points raised by the Mozilla security team need to be handled) (especially the silent update service is a high risk piece of code)
* handling of staged updates on windows (work around for the nasty file is still in use problem on windows)


Now most of the stuff is already implemented in our updater branch. I even managed to get the update service building and have most of the security stuff adapted to our code. To show that an alternative is better you need to cross at least these points from the list and show that it is less work to integrate it correctly than it is to finish the current code.
Comment 45 Ben Hearsum 2017-04-11 15:25:59 UTC
Hi there, I'm one of the Mozilla engineers who works a lot on our update system (mostly the server side). If there's any questions are confusion about the way our updates work feel free to get in touch (email, irc - whatever) - I'm happy to help if you're having issues.

I also wanted to point you at https://wiki.mozilla.org/ReleaseEngineering/Funsize - which is the system we use to generate partial MARs these days.
Comment 46 Markus Mohrhard 2017-07-09 11:44:03 UTC
(In reply to Ben Hearsum from comment #45)
> Hi there, I'm one of the Mozilla engineers who works a lot on our update
> system (mostly the server side). If there's any questions are confusion
> about the way our updates work feel free to get in touch (email, irc -
> whatever) - I'm happy to help if you're having issues.
> 
> I also wanted to point you at
> https://wiki.mozilla.org/ReleaseEngineering/Funsize - which is the system we
> use to generate partial MARs these days.

Hey Ben,

thanks for the offer. I have by now implemented at least some parts of the mar based updater for Linux daily builds. We have some unique challenges that are different to Mozilla that we need to solve (e.g. we distribute MSI on windows and will need to generate MSP that we wrap in the MAR files).

Funsize looks like something I need to evaluate. We are still hoping to be able get basic automatic updates based on the Mozilla updater code working in 6.0
Comment 47 Xisco Faulí 2019-06-04 13:47:55 UTC
*** Bug 125625 has been marked as a duplicate of this bug. ***