Please refer to the file at https://www.elsevier.com/__data/assets/excel_doc/0015/91122/title_list.xlsx that has already been used as a benchmark in a large number of reports about LibO fileopen/filesave speed. This is a huge xlsx file with all the scientific publications indexed in the Scopus database. Recently, the opening time of such file has been addressed. In fact, recent LibO is quite efficient at opening it (6-7 s on my machine). Thanks to the developers for that! Unfortunately, a weird side effect of this action, is that LibO is now *much* more efficient at dealing with large spreadsheet when those are in the xlsx format than it its own one. To see it, after having opened the file in attachment, save it as ods. Saving still takes a lot of time, but I believe there is a separate bug for that. This is not the issue here. Now, close LibO and restart it, trying to open the file again... This takes a long time (49-50 s on my machine). The bad aspect is not just the long absolute opening time of the ods file. It is the weird fact that opening the native format takes 600% more time than opening the foreign one. At least the file in the native file format is smaller (about 45% smaller).
Created attachment 121986 [details] Same file in ODS format Hello, Thank you for reporting the bug. I can confirm that the bug is present in LO 5.1.0.1 on Ubuntu 15.04 Opening the ODS file takes x10 times longer than the XLSX file that is the origin of the ODS one.
"Native format" is a marketing / advocacy term, not a technical one.
Always thought it was it was technical. Aren't /native/ formats those format in which LibO is validated to have correct roundtrip (and for which the internal data structures are guaranteed to have a non lossy correspondence with the file ones?) Isn't this the reason why LibO shows a warning dialog when saving in other formats?
The specification around XLSX makes it much easier to tune the import performance than the ODF specification. We employ a number of steps in the XLSX import filter that are not really possible in ODF without changing the file format.
Thanks, this is an interesting clarification. One more question. Most of the time spent in loading the file in the ODS format, goes /before/ the progress bar starts working. Any clue about this? Is there anything that can be done about this specific aspect?
** Please read this message in its entirety before responding ** To make sure we're focusing on the bugs that affect our users today, LibreOffice QA is asking bug reporters and confirmers to retest open, confirmed bugs which have not been touched for over a year. There have been thousands of bug fixes and commits since anyone checked on this bug report. During that time, it's possible that the bug has been fixed, or the details of the problem have changed. We'd really appreciate your help in getting confirmation that the bug is still present. If you have time, please do the following: Test to see if the bug is still present on a currently supported version of LibreOffice (5.2.5 or 5.3.0 https://www.libreoffice.org/download/ If the bug is present, please leave a comment that includes the version of LibreOffice and your operating system, and any changes you see in the bug behavior If the bug is NOT present, please set the bug's Status field to RESOLVED-WORKSFORME and leave a short comment that includes your version of LibreOffice and Operating System Please DO NOT Update the version field Reply via email (please reply directly on the bug tracker) Set the bug's Status field to RESOLVED - FIXED (this status has a particular meaning that is not appropriate in this case) If you want to do more to help you can test to see if your issue is a REGRESSION. To do so: 1. Download and install oldest version of LibreOffice (usually 3.3 unless your bug pertains to a feature added after 3.3) http://downloadarchive.documentfoundation.org/libreoffice/old/ 2. Test your bug 3. Leave a comment with your results. 4a. If the bug was present with 3.3 - set version to "inherited from OOo"; 4b. If the bug was not present in 3.3 - add "regression" to keyword Feel free to come ask questions or to say hello in our QA chat: http://webchat.freenode.net/?channels=libreoffice-qa Thank you for helping us make LibreOffice even better for everyone! Warm Regards, QA Team MassPing-UntouchedBug-20170306
Bug still present as of 5.3.1 RC 1. On an haswell laptop, the xlsx file opens in a matter of seconds, the ods file opens in a time that is likely twenty times larger, making the ods format not an practical option. In my opinion the most serious issue and the worse part of the matter is that the LibO file causes LibO to appear as frozen for a long time before showing a progress bar at all. On opening a file, LibO should first of all show visual elements capable of giving the user some feedback about what is going on and only later start its file loading task. Conversely, at least in Linux, when trying to open the ods file, you end up looking at an empty window that, not getting drawn, shows through the other applications that were previously on the desktop for almost a minute. E.g., if you had firefox open, you still see firefox through a frozen transparent LibO window, which prevents you from interacting with firefox unless you realize that you need to minimize it. Hence, LibO also ends up messing the Desktop experience. Similarly, you get an entry that stays unnamed for about a minute on the task bar.
** Please read this message in its entirety before responding ** To make sure we're focusing on the bugs that affect our users today, LibreOffice QA is asking bug reporters and confirmers to retest open, confirmed bugs which have not been touched for over a year. There have been thousands of bug fixes and commits since anyone checked on this bug report. During that time, it's possible that the bug has been fixed, or the details of the problem have changed. We'd really appreciate your help in getting confirmation that the bug is still present. If you have time, please do the following: Test to see if the bug is still present with the latest version of LibreOffice from https://www.libreoffice.org/download/ If the bug is present, please leave a comment that includes the information from Help - About LibreOffice. If the bug is NOT present, please set the bug's Status field to RESOLVED-WORKSFORME and leave a comment that includes the information from Help - About LibreOffice. Please DO NOT Update the version field Reply via email (please reply directly on the bug tracker) Set the bug's Status field to RESOLVED - FIXED (this status has a particular meaning that is not appropriate in this case) If you want to do more to help you can test to see if your issue is a REGRESSION. To do so: 1. Download and install oldest version of LibreOffice (usually 3.3 unless your bug pertains to a feature added after 3.3) from http://downloadarchive.documentfoundation.org/libreoffice/old/ 2. Test your bug 3. Leave a comment with your results. 4a. If the bug was present with 3.3 - set version to 'inherited from OOo'; 4b. If the bug was not present in 3.3 - add 'regression' to keyword Feel free to come ask questions or to say hello in our QA chat: https://kiwiirc.com/nextclient/irc.freenode.net/#libreoffice-qa Thank you for helping us make LibreOffice even better for everyone! Warm Regards, QA Team MassPing-UntouchedBug
Still an issue as of 6.0.2.1. Opening the large spreadsheet in excel format takes much longer than in the xlsx format. Again, issue is made much worse by the way in which progress bars appear. "Loading" progress bar starts appearing only halfway through the opeining, meaning that the user for ~ half a minute only sees an apparently hung screen.
There isn't file on link https://www.elsevier.com/__data/assets/excel_doc/0015/91122/title_list.xlsx Someone has it on his locale machine? Please attach it into bug directly ODS file : opening it took 2m30sec in Version: 6.3.0.0.alpha0+ Build ID: c57dc7d41bd62f933cffab6131edb7252606382d CPU threads: 4; OS: Windows 6.1; UI render: default; VCL: win; Locale: ru-RU (ru_RU); UI-Language: en-US Calc: threaded
*** Bug 108285 has been marked as a duplicate of this bug. ***
Changing to High -> Major * Tedious slowness.. * Native format being substantial slower compared XLSX format; quite a bad impression.. (I'm actually using the XLSX to avoid the waiting time for ODS) * An optimization seems to be within reach.. looking at the XLSX performance
Recent Flamegraph: See attachment 161648 [details] for bug 108285
I'm sure there will always be a XLSX file that open faster in Libo than the same file as ODS, but what's the point of having such a bug? it's too general and it will never be solved. Instead, let's use this ticket to focus on the performance issue seen while opening the attached document it takes real 1m59,400s user 2m5,970s sys 0m9,378s in Version: 7.1.0.0.alpha0+ Build ID: 8aee8c0cf1bdda1866594e75b0f9bd4b9a69c724 CPU threads: 4; OS: Linux 4.19; UI render: default; VCL: gtk3 Locale: en-US (en_US.UTF-8); UI: en-US Calc: threaded
@Julien, would you mind getting a flamegraph from the attached document ?
Created attachment 161667 [details] Example file ODS
Created attachment 161668 [details] Example file XLSX
(In reply to Xisco Faulí from comment #14) > I'm sure there will always be a XLSX file that open faster in Libo than the > same file as ODS, but what's the point of having such a bug? it's too > general and it will never be solved. Instead, let's use this ticket to focus > on the performance issue seen while opening the attached document > > it takes > > real 1m59,400s > user 2m5,970s > sys 0m9,378s > > in > > Version: 7.1.0.0.alpha0+ > Build ID: 8aee8c0cf1bdda1866594e75b0f9bd4b9a69c724 > CPU threads: 4; OS: Linux 4.19; UI render: default; VCL: gtk3 > Locale: en-US (en_US.UTF-8); UI: en-US > Calc: threaded Sounds like GTK3.. 30 seconds Version: 7.1.0.0.alpha0+ (x64) Build ID: 191288d6a7fb52b31038a21c4e71ee57ffa3bacd CPU threads: 4; OS: Windows 6.3 Build 9600; UI render: Skia/Raster; VCL: win Locale: nl-NL (nl_NL); UI: en-US Calc: CL
(In reply to Xisco Faulí from comment #14) > I'm sure there will always be a XLSX file that open faster in Libo than the > same file as ODS, but what's the point of having such a bug? it's too > general and it will never be solved Please explain.. to general.. the ODS takes 20 seconds.. the XLSX file within 7 seconds.. There is comment 4 - explaining the issue, but some improvement wouldn't hurt (if possible).. or should the XLSX format be default for 1024plus-Column?
(In reply to Xisco Faulí from comment #15) > @Julien, would you mind getting a flamegraph from the attached document ? You can use https://bugs.documentfoundation.org/attachment.cgi?id=161648 More generally, let's wait more feedback from Flamegraphs before providing more.
(In reply to Julien Nabet from comment #20) > (In reply to Xisco Faulí from comment #15) > > @Julien, would you mind getting a flamegraph from the attached document ? > > You can use https://bugs.documentfoundation.org/attachment.cgi?id=161648 I fail to see the relation between this ticket and bug 161648 > More generally, let's wait more feedback from Flamegraphs before providing > more. fair enough
(In reply to Telesto from comment #18) > (In reply to Xisco Faulí from comment #14) > > I'm sure there will always be a XLSX file that open faster in Libo than the > > same file as ODS, but what's the point of having such a bug? it's too > > general and it will never be solved. Instead, let's use this ticket to focus > > on the performance issue seen while opening the attached document > > > > it takes > > > > real 1m59,400s > > user 2m5,970s > > sys 0m9,378s > > > > in > > > > Version: 7.1.0.0.alpha0+ > > Build ID: 8aee8c0cf1bdda1866594e75b0f9bd4b9a69c724 > > CPU threads: 4; OS: Linux 4.19; UI render: default; VCL: gtk3 > > Locale: en-US (en_US.UTF-8); UI: en-US > > Calc: threaded > > Sounds like GTK3.. 30 seconds > Version: 7.1.0.0.alpha0+ (x64) > Build ID: 191288d6a7fb52b31038a21c4e71ee57ffa3bacd > CPU threads: 4; OS: Windows 6.3 Build 9600; UI render: Skia/Raster; VCL: win > Locale: nl-NL (nl_NL); UI: en-US > Calc: CL Same time with x11 Version: 7.1.0.0.alpha0+ Build ID: 8aee8c0cf1bdda1866594e75b0f9bd4b9a69c724 CPU threads: 4; OS: Linux 4.19; UI render: default; VCL: x11 Locale: en-US (en_US.UTF-8); UI: en-US Calc: threaded and Versão: 6.4.3.2 (x64) ID da versão: 747b5d0ebf89f41c860ec2a39efd7cb15b54f2d8 Processos do CPU: 1; SO: Windows 6.1 Service Pack 1 Build 7601; Gestão da interface: padrão; VCL: win; Configuração regional: es-ES (es_ES); Idioma da interface: pt-PT Calc: threaded Again, if a file is opening faster with one filter than with another one, it's a problem in the filter itself. Even better, I would use CSV as default for 1024plus-Column files. sarcasm off
Files attached are completely unrelated
Same results when multi-threaded calculation is disabled
Noel wrote a patch for this: https://gerrit.libreoffice.org/c/core/+/96516 Before: real 0m15,074s user 0m15,847s sys 0m2,081s After: real 0m12,268s user 0m14,442s sys 0m0,749s Arch Linux 64-bit Version: 7.1.0.0.alpha0+ Build ID: ad0351b84926075297fb74abbe9b31a0455782af CPU threads: 8; OS: Linux 5.7; UI render: default; VCL: kf5 Locale: fi-FI (fi_FI.UTF-8); UI: en-US Calc: threaded Built on 17 June 2020
(In reply to Buovjaga from comment #25) > Noel wrote a patch for this: https://gerrit.libreoffice.org/c/core/+/96516 > ... Yes but for the moment a QA test fails on Jenkins.
For me it takes real 1m38,793s user 1m43,851s sys 0m9,720s without the patch and real 1m22,985s user 1m34,677s sys 0m3,728s with the patch
Created attachment 162210 [details] framegraph ( 1st try )
Created attachment 162230 [details] framegraph ( 2st try )
Noel Grandin committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/aabcf90da9a90240bddc140485f210dcab66724c tdf#97177 speedup loading of large ODS file It will be available in 7.1.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Pushed a small improvement, sorry, I don't have any more ideas
after Noel's commit it takes real 1m10,605s user 1m31,436s sys 0m3,892s and before real 1m25,321s user 1m41,265s sys 0m8,422s so some improvement has been done. For the record, I double my ram last week and that might explain why it takes less time now than in previous measurements.
Created attachment 162304 [details] perf flamegraph Here's a Flamegraph retrieved on pc Debian x86-64 with master sources updated today (after https://cgit.freedesktop.org/libreoffice/core/commit/?id=aabcf90da9a90240bddc140485f210dcab66724c: "tdf#97177 speedup loading of large ODS file drop ScSimpleRangeList and just use ScRangeList, which saves us a conversion step. Then teach ScRangeList to do a simple merge, since we are loading in row order, and can just check the last few entries. Then fix a case of optimisation doing the wrong thing in ScAttrArray::SetPatternAreaImpl where std::vector::reserve repeatedly resizes the data array and breaks the normal doubling-resizing inside vector. On my machine the time goes from 5.4s to 4.8s")
Noel Grandin committed a patch related to this issue. It has been pushed to "libreoffice-7-0": https://git.libreoffice.org/core/commit/d1f613399d9c188ff4d5f41a57fbea6b02b536d9 tdf#97177 speedup loading of large ODS file It will be available in 7.0.0.1. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
This should have already been fixed. In addition, see bug 133835 and bug 136838 for some nice improvements in autofilter these days.