Bug 117731 - Batch libreoffice --convert-to offers no way to wait for document completion
Summary: Batch libreoffice --convert-to offers no way to wait for document completion
Status: RESOLVED WONTFIX
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: LibreOffice (show other bugs)
Version:
(earliest affected)
7.0.3.1 release
Hardware: x86-64 (AMD64) Linux (All)
: medium normal
Assignee: Not Assigned
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-05-21 20:25 UTC by Richard Elkins
Modified: 2020-12-15 16:03 UTC (History)
4 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Richard Elkins 2018-05-21 20:25:17 UTC
Description:
Command-line: libreoffice --headless --convert-to odt:writer8 myfile.txt
The returned status code ($?) is zero.  So far, so good.
But, when I try to use the ODT file (E.g. copy it), it doesn't exist yet.
Yes, I tried `sync; sync; sync` but it did not help.
If I add a `sleep 1` immediately after checking the status code, then this seems to allow enough time for some libreoffice subprocess (?) to finish.

Before version 6, I did not need the sleep step.  Maybe, this is some sort of optimization?  If so, please provide an option to indicate that libreoffice should hold up the process until the desired output is available.

I am using libreoffice 1:6.0.3-0ubuntu1 on Xubuntu 18.04.

`libreoffice --help` shows: 
LibreOffice 6.0.3.2 00m0(Build:2)


Steps to Reproduce:
Linux batch script, starting with an existing text file called "myfile.txt":

	libreoffice --headless --convert-to odt:writer8 myfile.txt
	RC=$?
	if [ $RC -ne 0 ]; then 
		echo '*** libreoffice conversion failed for myfile.txt'
		exit 86
	fi
        cp myfile.odt somewhere-else.odt



Actual Results:  
cp: cannot stat 'myfile.odt': No such file or directory



Expected Results:
Copy completes as normal because myfile.odt is available.


Reproducible: Always


User Profile Reset: No



Additional Info:




User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0
Comment 1 Jean-Baptiste Faure 2018-06-16 11:54:31 UTC
Not reproducible for me under Ubuntu 16.04 x86-64 with LibreOffice 6.0.4 from Ubuntu PPA. The script completes as expected.

To be sure, does it works for you if you try only the command

libreoffice --headless --convert-to odt:writer8 myfile.txt

in a terminal ?

Do you have some non standard settings for you file system?

Status set to NEEDINFO, please set it back to UNCONFIRMED once requested
informations are provided.


Best regards. JBF
Comment 2 Richard Elkins 2018-06-16 14:07:41 UTC
Like you, when I ran LibreOffice under Xubuntu 16.04 and 17.10, there was no timing issue.  This only has appeared for me in 18.04.

In a terminal window (I've done this before),
`libreoffice --headless --convert-to odt:writer8 myfile.txt; ls *.odt` result:
ls: cannot access '*.odt': No such file or directory

My LibreOffice packages installed:
ii  libreoffice                           1:6.0.3-0ubuntu1         amd64        office productivity suite (metapackage)
ii  libreoffice-avmedia-backend-gstreamer 1:6.0.3-0ubuntu1         amd64        GStreamer backend for LibreOffice
ii  libreoffice-base                      1:6.0.3-0ubuntu1         amd64        office productivity suite -- database
ii  libreoffice-base-core                 1:6.0.3-0ubuntu1         amd64        office productivity suite -- shared library
ii  libreoffice-base-drivers              1:6.0.3-0ubuntu1         amd64        Database connectivity drivers for LibreOffice
ii  libreoffice-calc                      1:6.0.3-0ubuntu1         amd64        office productivity suite -- spreadsheet
ii  libreoffice-common                    1:6.0.3-0ubuntu1         all          office productivity suite -- arch-independent files
ii  libreoffice-core                      1:6.0.3-0ubuntu1         amd64        office productivity suite -- arch-dependent files
ii  libreoffice-draw                      1:6.0.3-0ubuntu1         amd64        office productivity suite -- drawing
ii  libreoffice-gnome                     1:6.0.3-0ubuntu1         amd64        office productivity suite -- GNOME integration
ii  libreoffice-gtk                       1:6.0.3-0ubuntu1         all          transitional package for LibreOffice gtk2 backend
ii  libreoffice-gtk2                      1:6.0.3-0ubuntu1         amd64        office productivity suite -- GTK+ 2 integration
ii  libreoffice-gtk3                      1:6.0.3-0ubuntu1         amd64        office productivity suite -- GTK+ 3 integration
ii  libreoffice-help-en-us                1:6.0.3-0ubuntu1         all          office productivity suite -- English_american help
ii  libreoffice-impress                   1:6.0.3-0ubuntu1         amd64        office productivity suite -- presentation
ii  libreoffice-java-common               1:6.0.3-0ubuntu1         all          office productivity suite -- arch-independent Java support files
ii  libreoffice-librelogo                 1:6.0.3-0ubuntu1         all          Logo-like progamming language for LibreOffice
ii  libreoffice-math                      1:6.0.3-0ubuntu1         amd64        office productivity suite -- equation editor
ii  libreoffice-nlpsolver                 0.9+LibO6.0.3-0ubuntu1   all          "Solver for Nonlinear Programming" extension for LibreOffice
ii  libreoffice-ogltrans                  1:6.0.3-0ubuntu1         amd64        LibreOffice Impress extension for slide transitions using OpenGL
ii  libreoffice-report-builder            1:6.0.3-0ubuntu1         all          LibreOffice component for building database reports
ii  libreoffice-report-builder-bin        1:6.0.3-0ubuntu1         amd64        LibreOffice component for building database reports -- libraries
ii  libreoffice-script-provider-bsh       1:6.0.3-0ubuntu1         all          BeanShell script support provider for LibreOffice scripting framework
ii  libreoffice-script-provider-js        1:6.0.3-0ubuntu1         all          JavaScript script support provider for LibreOffice scripting framework
ii  libreoffice-script-provider-python    1:6.0.3-0ubuntu1         all          Python script support provider for LibreOffice scripting framework
ii  libreoffice-sdbc-hsqldb               1:6.0.3-0ubuntu1         amd64        HSQLDB SDBC driver for LibreOffice
ii  libreoffice-sdbc-postgresql           1:6.0.3-0ubuntu1         amd64        PostgreSQL SDBC driver for LibreOffice
ii  libreoffice-style-elementary          1:6.0.3-0ubuntu1         all          office productivity suite -- Elementary symbol style
ii  libreoffice-style-galaxy              1:6.0.3-0ubuntu1         all          office productivity suite -- Galaxy (Default) symbol style
ii  libreoffice-style-tango               1:6.0.3-0ubuntu1         all          office productivity suite -- Tango symbol style
ii  libreoffice-wiki-publisher            1.2.0+LibO6.0.3-0ubuntu1 all          LibreOffice extension for working with MediaWiki articles
ii  libreoffice-writer                    1:6.0.3-0ubuntu1         amd64        office productivity suite -- word processor

Just now, I reproduced this anomaly.  My script is as follows:

### Create somehow a file called myfile.txt
rm myfile.odt somewhere-else.odt
libreoffice --headless --convert-to odt:writer8 myfile.txt
RC=$?
if [ $RC -ne 0 ]; then 
	echo '*** libreoffice conversion failed for myfile.txt'
	exit 86
fi
#sleep 3
cp myfile.odt somewhere-else.odt

As is, it consistently produces the reported anomaly.
When I uncomment the sleep step, all is well.

This might be an interface issue with newer version dependencies of LibreOffice.  Its hard to tell.  How would I investigate without knowing a lot of LibreOffice internals?  

I am open to suggestion and further investigation.  I am a multi-level developer in case you want me to try something unusual.

Suggestion: have someone try my test script under any flavor of Ubuntu 18.04.  If you cannot reproduce this report, then just close it as unreproducable.

Please note: This report should be not a high priority as my 3-second delay consistently works for me i.e. I am not stuck.
Comment 3 Richard Elkins 2018-06-16 14:10:49 UTC
`libreoffice --version`:
LibreOffice 6.0.3.2 00m0(Build:2)
Comment 4 Richard Elkins 2018-06-16 14:30:54 UTC
You might be tempted to think that Ubuntu or my hardware is running a little slow.  If you insert a `sync; sync; sync` right after the libreoffice step which flushes buffers, that does not help consistently.  Something got spawned by or because of libreoffice.  It finishes long after libreoffice exits.  Bad idea, in my opinion.

When libreoffice exits, that ODT file should be immediately available.
Comment 5 Jean-Baptiste Faure 2018-06-16 18:06:02 UTC
You should report this behavior against Ubuntu 18.04 on https://bugs.launchpad.net/

Best regards. JBF
Comment 6 QA Administrators 2019-01-11 15:21:58 UTC Comment hidden (obsolete)
Comment 7 Richard Elkins 2019-01-11 18:17:20 UTC
Opened on Launchpad as #1777285 against libreoffice (Ubuntu).
Comment 8 Richard Elkins 2019-06-23 17:26:06 UTC
Moved to bugs.launchpad.net and confirmed:
https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1777285
Comment 9 Richard Elkins 2019-06-24 20:55:51 UTC
The source code indicates that soffice.bin is the one starting a background process that does not finish before soffice.bin exits.

See: https://github.com/LibreOffice/core/blob/master/shell/source/unix/exec/shellexec.cxx
Go to line 218:

    OString cmd =
#ifdef LINUX
        // avoid blocking (call it in background)
        "( " + aBuffer.makeStringAndClear() + " ) &";
#else
        aBuffer.makeStringAndClear();
#endif
    FILE *pLaunch = popen(cmd.getStr(), "w");
    if ( pLaunch != nullptr )
    {
        if ( 0 == pclose( pLaunch ) )
            return;

It would be interesting to understand why execution is in background *only* for Linux. In my opinion, it is undesirable for command line execution in any O/S.

Other opinions?  It is possible for me to be perfectly content with the artificial `sleep N` in my shell script.
Comment 10 Xisco Faulí 2019-06-26 08:43:24 UTC
Could you please try to reproduce it with the latest version of LibreOffice
from https://www.libreoffice.org/download/libreoffice-fresh/ ?
I have set the bug's status to 'NEEDINFO'. Please change it back to
'UNCONFIRMED' if the bug is still present in the latest version.
Comment 11 Richard Elkins 2019-06-26 22:17:39 UTC
Yes, I have tried this with the latest download.  This is an off-the-shelf Xubuntu 18.04.latest LTS desktop that I use primarily for Libreoffice, browsing, and Python development.  I tend to stay with Ubuntu's packages including Libreoffice.

Please see comment #9.  

The current source code has special #ifdef LINUX logic to fork the transaction into background and then immediately exit to the command line.  It does not wait for the ODT file to be ready for use.  Hence, any Linux/Unix bash command that references the ODT file will definitely fail with file not found.  Workaround: sleep for 2/3 seconds before trying to access the ODT file.

For Mac and Windows, the transaction takes place in foreground so that when soffice.bin exits to the O/S, the ODT file is guaranteed to be available for use.  This is the desirable effect for all O/Ses.

Why is there more concern with "blocking" in the Unix/Linux environments as opposed to Mac (Unix bash environment) and Windows?

IMO, this is a bug for the Linux environment.  There is no reason for special #ifdef LINUX logic.

Has anyone else tried the above script yourself using a Linux desktop with the current or recent Libreoffice?
Comment 12 Richard Elkins 2019-06-26 22:32:28 UTC
Seriously, I am not trying to make work for anyone.  We are all busy and I do not want to be a pest.  

If the #ifdef LINUX for Linux and Unix in shellexec.cxx has a purpose, I'd like to know.  I can live with my `sleep` logic.
Comment 13 Richard Elkins 2019-06-26 22:38:20 UTC
The command (OString cmd) is executed in background through a pipe no matter what.  The difference is that in Mac and Windows, the foreground process waits (blocked) while in Linux and Unix, the foreground process continues to the exit.

Pardon my misuse of terminology.
Comment 14 Xisco Faulí 2020-01-20 17:31:39 UTC
(In reply to Richard Elkins from comment #9)
> The source code indicates that soffice.bin is the one starting a background
> process that does not finish before soffice.bin exits.
> 
> See:
> https://github.com/LibreOffice/core/blob/master/shell/source/unix/exec/
> shellexec.cxx
> Go to line 218:
> 
>     OString cmd =
> #ifdef LINUX
>         // avoid blocking (call it in background)
>         "( " + aBuffer.makeStringAndClear() + " ) &";
> #else
>         aBuffer.makeStringAndClear();
> #endif
>     FILE *pLaunch = popen(cmd.getStr(), "w");
>     if ( pLaunch != nullptr )
>     {
>         if ( 0 == pclose( pLaunch ) )
>             return;
> 
> It would be interesting to understand why execution is in background *only*
> for Linux. In my opinion, it is undesirable for command line execution in
> any O/S.
> 
> Other opinions?  It is possible for me to be perfectly content with the
> artificial `sleep N` in my shell script.

it was added in https://cgit.freedesktop.org/libreoffice/core/commit/?id=f5fe1aa1 to fix bug 32427

@Richard, can you confirm removing the mentioned ifdef condition would fix the issue? In that case we could submit a patch removing it. Most likely bug 32427 is obsolete nowadays...
Comment 15 Richard Elkins 2020-01-20 23:40:07 UTC
@Xisco Faulí - Are you asking me to download and build LibreOffice for that one source change?  I wouldn't know where to begin.  I am a developer but up to my eyeballs in Astrophysics projects at the moment.  

Is there anyone on the development or testing team who already has the source downloaded and knows how to build LibreOffice?  I am confident that such a person could easily reproduce this anomaly on Linux, Android, or Unix with the script I provided.  Then, make the change I suggested.  Re-run the script and see that the anomaly disappears.  Also, after a change like this, wouldn't one need to run your standard regression test suite?
Comment 16 QA Administrators 2020-01-21 03:33:18 UTC Comment hidden (obsolete)
Comment 17 Buovjaga 2020-05-09 13:14:36 UTC
(In reply to Richard Elkins from comment #0)
> Steps to Reproduce:
> Linux batch script, starting with an existing text file called "myfile.txt":
> 
> 	libreoffice --headless --convert-to odt:writer8 myfile.txt
> 	RC=$?
> 	if [ $RC -ne 0 ]; then 
> 		echo '*** libreoffice conversion failed for myfile.txt'
> 		exit 86
> 	fi
>         cp myfile.odt somewhere-else.odt
> 
> Actual Results:  
> cp: cannot stat 'myfile.odt': No such file or directory
> 
> Expected Results:
> Copy completes as normal because myfile.odt is available.

I get the expected result. I guess you should bite the bullet, clone the source and try modifying it.

https://wiki.documentfoundation.org/Development/BuildingOnLinux
https://wiki.documentfoundation.org/Development/Linux_Build_Dependencies

Arch Linux 64-bit
Version: 7.0.0.0.alpha1+
Build ID: c73418d8d1258ea0a8c77c07672fd182e2b28b26
CPU threads: 8; OS: Linux 5.6; UI render: default; VCL: kf5; 
Locale: fi-FI (fi_FI.UTF-8); UI: en-US
Calc: threaded
Built on 9 May 2020
Comment 18 Xisco Faulí 2020-06-17 14:34:43 UTC
(In reply to Buovjaga from comment #17)
> (In reply to Richard Elkins from comment #0)
> > Steps to Reproduce:
> > Linux batch script, starting with an existing text file called "myfile.txt":
> > 
> > 	libreoffice --headless --convert-to odt:writer8 myfile.txt
> > 	RC=$?
> > 	if [ $RC -ne 0 ]; then 
> > 		echo '*** libreoffice conversion failed for myfile.txt'
> > 		exit 86
> > 	fi
> >         cp myfile.odt somewhere-else.odt
> > 
> > Actual Results:  
> > cp: cannot stat 'myfile.odt': No such file or directory
> > 
> > Expected Results:
> > Copy completes as normal because myfile.odt is available.
> 
> I get the expected result. I guess you should bite the bullet, clone the
> source and try modifying it.
> 
> https://wiki.documentfoundation.org/Development/BuildingOnLinux
> https://wiki.documentfoundation.org/Development/Linux_Build_Dependencies
> 
> Arch Linux 64-bit
> Version: 7.0.0.0.alpha1+
> Build ID: c73418d8d1258ea0a8c77c07672fd182e2b28b26
> CPU threads: 8; OS: Linux 5.6; UI render: default; VCL: kf5; 
> Locale: fi-FI (fi_FI.UTF-8); UI: en-US
> Calc: threaded
> Built on 9 May 2020

@Richard Elkins, have you had time to try it ?
Comment 19 QA Administrators 2020-12-15 03:45:55 UTC Comment hidden (obsolete)
Comment 20 Richard Elkins 2020-12-15 16:03:32 UTC
Since there seems to be no interest nor resource to fix the source code where I indicated in comment 9, just forget it.  My work-around works fine.

I marked this report as RESOLVED and WONTFIX.