Bug 96720 - FILESAVE: .odt report produces different ODF Validator Result than a copy saved from Writer (probably resulting in slow file load)
Summary: FILESAVE: .odt report produces different ODF Validator Result than a copy sav...
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Base (show other bugs)
Version:
(earliest affected)
5.0.4.2 release
Hardware: All All
: medium major
Assignee: Not Assigned
URL:
Whiteboard:
Keywords: haveBacktrace, perf
Depends on:
Blocks: Database-Reports
  Show dependency tree / graph
 
Reported: 2015-12-25 11:08 UTC by Sparrowbe
Modified: 2024-11-07 07:53 UTC (History)
8 users (show)

See Also:
Crash report or crash signature:

Attachments
database and output report that cause the problem (59.34 KB, application/x-zip)
2015-12-25 11:08 UTC, Sparrowbe 		Details
callgrind_annotate --tree (6.93 MB, text/plain)
2016-01-03 01:22 UTC, Terrence Enger 		Details
kcachegrind screenshot (204.01 KB, image/png)
2016-01-03 22:13 UTC, Julien Nabet 		Details
Perf flamegraph (1.81 MB, image/svg+xml)
2019-04-19 07:41 UTC, Buovjaga 		Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Sparrowbe 2015-12-25 11:08:44 UTC 
Created attachment 121540 [details]
database and output report that cause the problem

Test Scenario:
- a report is generated from the database attached to this bug
- the output file is automatically opened in Writer, which takes unreasonably long time, freezing the program for minutes
- in Writer, File -> Save a Copy is chosen and the file is saved under different name, with default settings
- both .odf files - Base-generated and Copy - are parsed with the validator at http://odf-validator.rhcloud.com/

Result:
- Base-generated file causes "/META-INF/manifest.xml[2,88]: Error: element "manifest:manifest" is missing "version" attribute" error
- saved copy file passes the version attribute check but fails on "/styles.xml[2,9525]: Error: unexpected attribute "draw:fill""
- every attempt to open the Base-generated .odf file in Writer leads to heavy CPU load (on single core only) and repetitive GUI freezes which may last for minutes, or even block the program forever (the latter happens mostly when the file is closed and reopened)
- the copy file opens relatively quickly in Writer, doesn't cause GUI freeze.
Comment 1 Buovjaga 2015-12-27 18:31:29 UTC 
Reproduced the validation errors, but NOT the slowness.

I do observe a difference in the behavior of the loading progress bar: .odt coming from Base makes it do several quick completions while the save-as-copy .odt gives one slower progress bar completion.

I used the report wizard and added several of the fields.
When the report was opened in Writer, I did Save as copy and noted the temp folder that was offered.
I copied the Base generated report from the temp folder to safety.

I also got this error with the Base generated .odt:
styles.xml[3,20]: Error: unexpected character literal

Win 7 Pro 64-bit Version: 5.2.0.0.alpha0+
Build ID: a4764cfa80270f973da5861d0ddc28298bf16f4d
CPU Threads: 4; OS Version: Windows 6.1; UI Render: default; 
TinderBox: Win-x86@62-merge-TDF, Branch:MASTER, Time: 2015-12-24_22:45:12
Locale: fi-FI (fi_FI)
Comment 2 Robert Großkopf 2015-12-28 10:28:10 UTC 
(In reply to Beluga from comment #1)
> Reproduced the validation errors, but NOT the slowness.
> 
> I do observe a difference in the behavior of the loading progress bar: .odt
> coming from Base makes it do several quick completions while the
> save-as-copy .odt gives one slower progress bar completion.

Same here: Only a little bit slower (8,0 seconds to 7,15 seconds) but a lot of flickering in the progress bar while loading the Base *.odt. Same while executing the report in Base.

My system: OpenSUSE 42.1 Leap 64bit rpm Linux
Comment 3 Sparrowbe 2015-12-28 11:38:50 UTC 
Beluga and Robert, thank you for your feedback. But have you observed the "Page 1 of x" counter in the status bar? 
On all my machines (Windows 7/8.1 with Version: 5.0.4.2 Locale: pl-PL (pl_PL), either 32 and 64bit) the file generated by Base opens quickly, but is stuck on "Page 1 of 231" for a long time and any GUI actions like vertical scrolling or opening menus are greatly delayed, with "(Not Responding)" periodically added to the window title during this period.
On the other hand, the "Copy" document has all 640 pages loaded at once and is responsive immediately.

What is more, loading the "640_records_from_Base.odt" for the second time causes my Writer to eventually freeze permanently (>15 minutes), with no activity visible in the Task Manager.
Comment 4 Robert Großkopf 2015-12-28 11:53:09 UTC 
(In reply to Sparrowbe from comment #3)
> Beluga and Robert, thank you for your feedback. But have you observed the
> "Page 1 of x" counter in the status bar? 
> On all my machines (Windows 7/8.1 with Version: 5.0.4.2 Locale: pl-PL
> (pl_PL), either 32 and 64bit) the file generated by Base opens quickly, but
> is stuck on "Page 1 of 231" for a long time and any GUI actions like
> vertical scrolling or opening menus are greatly delayed, with "(Not
> Responding)" periodically added to the window title during this period.
> On the other hand, the "Copy" document has all 640 pages loaded at once and
> is responsive immediately.

You are right. Didn't notice 1 of 231 and scrolling-problems to the end of the document. The saved and reopened file shows 1 of 640 instead - and no scrolling-problem at all.
Comment 5 Lionel Elie Mamane 2015-12-28 14:27:47 UTC 
Yes, the report builder creates an odt file on its own, without going through writer, and then opens that file in writer. As such, minor differences in structure, within the choices allowed by the ODF standard are normal.

The missing version in the manifest tag, apparently a standard violation, would be a genuine bug and should be fixed.

The slowness is most probably a writer problem and needs to be reported / fixed on the writer side. (In the meantime, if report builder can do something specific to work around it, I'm open to suggestions.)
Comment 6 Alex Thurgood 2015-12-30 16:33:40 UTC 
Confirming slow loading, display, and scrolling to near freeze on OSX 10.11.2 Intel Core Duo with the Report Builder generated ODF and LibreOffice 5.0.3.2
Comment 7 Alex Thurgood 2015-12-30 16:38:29 UTC 
(In reply to Alex Thurgood from comment #6)
> Confirming slow loading, display, and scrolling to near freeze on OSX
> 10.11.2 Intel Core Duo with the Report Builder generated ODF and LibreOffice
> 5.0.3.2

The scrolling issue is really painful, LO staggers along, causing regular spinning beachball moments.

Maybe Miklos would know more about this ? Adding to CC.

Miklos : any thoughts on this behaviour ?
Comment 8 Terrence Enger 2016-01-03 01:22:43 UTC 
Created attachment 121686 [details]
callgrind_annotate --tree

I created a report file by ...
(*) delete all but first 40 records in the .odb
(*) run the report and copy the generated .odt
Then I opened the saved .odt under callgrind.

The callgrind dumpfile is 64MB, so probably not worth attaching.

I shall set whiteboard haveCallgrind.
Comment 9 Julien Nabet 2016-01-03 08:12:57 UTC 
Terrence: I'm not expert at Kcachegrind but can't open your file with it.
Would it be possible you put the 64MO file available somewhere?
Comment 10 Terrence Enger 2016-01-03 14:23:39 UTC 
Comment on attachment 121686 [details]
callgrind_annotate --tree

Julian,

The attachment is plain text, output from callgrind_annotate.  I
should have said that.

So far, I have not used a file hosting site, and I do not know long it
would take me to upload 64MB.

However I can do other analyses or data collections.  I have wasted
too much time trying to collect callgrind data while LibreOffice was
doing particular long-running things (Writer window with progress bar,
Writer window after progress bar but still with grey document area,
with page 1 painted but before the count of pages starts to increase,
page count increasing).  But perhaps from the attachment you can pick
out a function or functions which it would be good to instrument
individually.

HTH,
Terry.
Comment 11 Julien Nabet 2016-01-03 22:13:42 UTC 
Created attachment 121704 [details]
kcachegrind screenshot

I retrieved cachegrind trace but even compressed with bzip2 (which was better than gzip for this file), it's still 30MB.
I attached a screenshot of the result.
Comment 12 QA Administrators 2017-03-06 14:00:02 UTC Comment hidden (obsolete)
Comment 13 Terrence Enger 2017-03-22 03:17:11 UTC 
With the daily Linux dbgutil bibisect repository running on
debian-stretch, I see the slow initial open remains.  CPU times ...

  - Open the .odb            :   :02
  - Writer window appears    :  2:52
  - top of document rendered :  6:10 (status bar says Page 1 of 231)
  - I cancelled              : 46:10 (still 100% CPU)
Comment 14 QA Administrators 2018-07-06 02:46:31 UTC Comment hidden (obsolete, spam)
Comment 15 Buovjaga 2019-04-19 07:41:14 UTC 
Created attachment 150872 [details]
Perf flamegraph

I tried with 640_records_from_Base.odt and it really looks like it is unable to finish loading it. I cancelled after a few minutes.

Arch Linux 64-bit
Version: 6.3.0.0.alpha0+
Build ID: 1fee3f1da6291bfbcd75455512726215d41b3e83
CPU threads: 8; OS: Linux 5.0; UI render: default; VCL: gtk3; 
Locale: fi-FI (fi_FI.UTF-8); UI-Language: en-US
Calc: threaded
Built on 18 April 2019
Comment 16 QA Administrators 2021-04-19 03:39:57 UTC Comment hidden (obsolete)
Comment 17 Jorge Teixeira 2022-11-07 21:37:15 UTC 
I can confirm the validation errors (per https://odfvalidator.org/) with LO 7.4.2.3 by following the steps on the reproducer, but only mild/sadly-typical slowness in Writer.


Version: 7.4.2.3 (x64) / LibreOffice Community
Build ID: 382eef1f22670f7f4118c8c2dd222ec7ad009daf
CPU threads: 8; OS: Windows 10.0 Build 19044; UI render: Skia/Raster; VCL: win
Locale: pt-PT (pt_PT); UI: en-US
Calc: threaded
Comment 18 QA Administrators 2024-11-07 03:15:12 UTC Comment hidden (obsolete)
Comment 19 Robert Großkopf 2024-11-07 07:53:28 UTC 
Still the same with
Version: 24.8.3.1 (X86_64) / LibreOffice Community
Build ID: 65412f067af443213e726c93f137ccc85c9a1e06
CPU threads: 6; OS: Linux 6.4; UI render: default; VCL: kf5 (cairo+xcb)
Locale: de-DE (de_DE.UTF-8); UI: de-DE
Calc: threaded