Description: XLSX: Long time for file opens and using 100% of one core of CPU after opening Steps to Reproduce: 1. Open file from attach 2. It opens a long time 3. After opening it takes 100% of one core of CPU I think there are problems with long formulas in D2:G38 range Actual Results: Long time for file opens and using 100% of one core of CPU after opening Expected Results: A file opens fast and doesn't use 100% of one core of CPU after opening Reproducible: Always User Profile Reset: No Additional Info: Версия: 7.0.0.0.alpha0+ (x64) ID сборки: 47c598260a8f07cf3a1e4cab061df3f2d261932c Потоков ЦП: 4; ОС:Windows 10.0 Build 17763; Отрисовка ИП: GL; VCL: win; Локаль: ru-RU (ru_RU); Язык интерфейса: ru-RU Calc: threaded
Created attachment 157561 [details] XLSX file
Julien, hello. Could you test it problem and (if you can confirm it) make perfgraph : for opening file process and for time after opening (I'm not sure, but possible it can be different problems)
Created attachment 157567 [details] perf flamegraph On pc Debian x86-64 with master sources updated today, I got kindda of hanging at the beginning of loading.
After 20/30 secs waiting, I stopped the test (Ryzen 2600, 32Gb) I'll give another try once this first pb will be fixed.
Noel, possibly you will be interested to look at it
Created attachment 157586 [details] Modified sample file without INDIRECT and ranges formula from 10000 to 50 Modifying the sample file without INDIRECT function and ranges in formula from 10000 to 50, the issue still persist. After open I'm not able to do anything on it.
it takes real 1m19,083s user 1m15,514s sys 0m2,075s in Version: 7.2.0.0.alpha0+ / LibreOffice Community Build ID: 78eaf6489a6542378ffab7eef39ec0a2c5f1a10a CPU threads: 4; OS: Linux 5.7; UI render: default; VCL: gtk3 Locale: en-US (en_US.UTF-8); UI: en-US Calc: threaded
Import time went down to 1 minute 20 seconds after https://cgit.freedesktop.org/libreoffice/core/commit/?id=edf13fe1247e7ef411a9ff5435385573fad01f56 author Noel Grandin <noel.grandin@collabora.co.uk> 2020-03-03 09:23:30 +0200 committer Noel Grandin <noel.grandin@collabora.co.uk> 2020-03-10 09:26:19 +0100 commit edf13fe1247e7ef411a9ff5435385573fad01f56 (patch) tree a0f03fa1b87b8cb5e14b5113754d53322c1c78b8 parent 6db846c3c0bc1c44da1f3c7a8dea385930acc3b1 (diff) tdf#93831 xlsx file full of pictures of numbers slow to open @Roman, is the import time acceptable for you now ?
(In reply to Xisco Faulí from comment #8) > > @Roman, is the import time acceptable for you now ? it's better than before the patch, but Excel opens it for 5 sec only. In LO it looks as LO hangs instead opening. Funny, LO and Excel take 100% of one CPU core when you try scroll the sheet.
Created attachment 171057 [details] Flamegraph Here's a new Flamegraph of the file opening on pc Debian x86-64 with master sources updated today (without enable-dbgutil) by using gen rendering (to avoid accessibility stuff with gtk3).
Created attachment 171058 [details] Flamegraph during scrolling
Noel Grandin committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/79ef2cef85f35aeb11b620fdc82acd595a7cb4fb tdf#130326 related, speed up scrolling It will be available in 7.2.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Noel Grandin committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/526f0fce45fb014b09057403ae37c125e5a18655 tdf#130326 speed up opening XLSX It will be available in 7.2.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Noel Grandin committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/566e43d2129dd6c70bb718296ce66353e7ff824b tdf#130326 speed up XLSX load It will be available in 7.2.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Noel Grandin committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/44711d9eb53eb6247ebdb9293a3eb5e643f78059 tdf#130326 related, speed up drawing It will be available in 7.2.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Created attachment 171084 [details] perf flamegraph during opening Here's a new Flamegraph with master sources updated today (526f0fce45fb014b09057403ae37c125e5a18655)
Why does this document appear to contain a ton of drawings? Where are they?
Navigator shows 19562 text boxes.
Noel Grandin committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/f9fae84cbd9663d6a394aa8e4e4b927c11384815 tdf#130326 speed up XLSX load in ScFlatSegmentsImpl It will be available in 7.2.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Created attachment 171110 [details] perf flamegraph during opening Here's a new Flamegraph with master sources updated today (ac87ae4615fc96c247ca98a321c69400411dab06)
Sorry, I forgot to change env var, last one is with gtk3 rendering not with gen rendering :-(
It takes 57 sec for file opening and still takes about 50% of one CPU core while scrolling in build from 12 April 2021 Version: 7.2.0.0.alpha0+ (x64) / LibreOffice Community Build ID: 7a0e0a84a02f505200331c19b28d45e898cd5a12 CPU threads: 4; OS: Windows 10.0 Build 18363; UI render: Skia/Raster; VCL: win Locale: ru-RU (ru_RU); UI: ru-RU Calc: threaded
Noel Grandin committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/40f9888ddc5def405d5a7724cb70f26eca527f01 tdf#130326 flatten SfxItemPropertyMap It will be available in 7.2.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
it takes real 0m34,414s user 0m32,675s sys 0m1,670s in Version: 7.2.0.0.alpha0+ / LibreOffice Community Build ID: 4eac7a11e5d39ca6c783f65f1ca2df009b9a37e4 CPU threads: 4; OS: Linux 5.7; UI render: default; VCL: gtk3 Locale: en-US (en_US.UTF-8); UI: en-US Calc: threaded Nice improvement!!! @Roman, you branch from comment 22 is missing two commits improving this issue...
Noel Grandin committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/f6d1da91ccda960d1ee738d688a2cb374e576aef tdf#130326 no need to fill ScMatrix with zeros here It will be available in 7.2.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Noel Grandin committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/64152322b99556b2e80a58cdf04a586e6941ba10 tdf#130326 speed up row height calculated It will be available in 7.2.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
(In reply to Xisco Faulí from comment #24) > it takes > > real 0m34,414s > user 0m32,675s > sys 0m1,670s > > in > > Version: 7.2.0.0.alpha0+ / LibreOffice Community > Build ID: 4eac7a11e5d39ca6c783f65f1ca2df009b9a37e4 > CPU threads: 4; OS: Linux 5.7; UI render: default; VCL: gtk3 > Locale: en-US (en_US.UTF-8); UI: en-US > Calc: threaded > > Nice improvement!!! > > @Roman, you branch from comment 22 is missing two commits improving this > issue... OK, 43 sec, in Version: 7.2.0.0.alpha0+ (x64) / LibreOffice Community Build ID: 77419c6f3aba1fd5b1660795923c22a39bdb1bad CPU threads: 4; OS: Windows 6.1 Service Pack 1 Build 7601; UI render: Skia/Raster; VCL: win Locale: ru-RU (ru_RU); UI: ru-RU Calc: CL I think CPU is matter here (Intel Core 2 Quad Q9450, 4 CPU core with 2.66GHz)
So I had this improved down to around 17s, but since the ScNavigatorDlg/ScContentTree welding, it is back to around 24s on my machine, Caolan, perhaps you have some ideas about reducing how often the navigator updates? (The navigator also rebuilds the tree every time the cursor changes cells, which makes clicking around the lower parts of this spreadsheet - around row 1000 - very very slow)
Created attachment 171350 [details] perf flamegraph during opening Here's an updated Flamegraph with master sources updated today (1b1a9c6c12ebe4cac19e34ff5e4818998bbb2537) + gen rendering.
Noel Grandin committed a patch related to this issue. It has been pushed to "libreoffice-7-1": https://git.libreoffice.org/core/commit/91385bbc7cccfdf59f60a24eaf81894772134af0 tdf#130326 related, speed up drawing It will be available in 7.1.4. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Noel Grandin committed a patch related to this issue. It has been pushed to "libreoffice-7-1": https://git.libreoffice.org/core/commit/7d0e88ce674a7e809352842bcce3cd67f548a524 Revert "tdf#130326 related, speed up drawing" It will be available in 7.1.4. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
(In reply to Noel Grandin from comment #28) > So I had this improved down to around 17s, but since the > ScNavigatorDlg/ScContentTree welding, it is back to around 24s on my machine, > > Caolan, perhaps you have some ideas about reducing how often the navigator > updates? Maybe instead of immediately updating just schedule an idle to do the update. So at least there isn't multiple updates during load and instead just one at the end ?
(In reply to m.a.riosv from comment #18) > Navigator shows 19562 text boxes. Yes, 19562 text boxes 0 cm wide. It's very difficult to open their contextual menu. It's a use case where we see that the possibility to edit these objects from the Navigator is missing. If you succeed to open the contextual menu you can change the size and the background of the object to make it visible. This file make me thinking to files produced by Crystal Report in RTF format where each cell is made with 4 small line objects. There is several bug reports related to such files. Perhaps LO should remove an 2D object if its area is null. Best regards. JBF
Noel Grandin committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/1333600682019891efdb4acf51c4c1a9b7d3532d tdf#130326 clamp number of items in calc content tree It will be available in 7.2.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
I have done what I can here, I'm not going to work on this any more
(In reply to Jean-Baptiste Faure from comment #33) > (In reply to m.a.riosv from comment #18) > > Navigator shows 19562 text boxes. > > .... > Perhaps LO should remove an 2D object if its area is null. > > Best regards. JBF Of course NO. If LO opens 2D objects with wrong sizes it means we just should fix it but not delete those objects.
(In reply to Roman Kuznetsov from comment #36) > ... > Of course NO. If LO opens 2D objects with wrong sizes it means we just > should fix it but not delete those objects. Do you mean Excel shows indeed 19562 text boxes with the initial xlsx file? If yes, could you provide a screenshot? If they don't appear in the screenshot, I think Jean-Baptiste is right and we should delete these nonsense text boxes.
Caolán McNamara committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/ff8addee9c2c43c6ec064395e909665932e10899 Related: tdf#130326: GetDrawNames doesn't do anything when !bisInNavigatoeDlg It will be available in 7.2.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Those constant refreshes on each click in a cell are since: commit b41332475783c31136673fb44cf4c411bb0148f8 Author: Steve Yin <steve_y@apache.org> Date: Mon Dec 2 15:54:29 2013 +0000 Integrate branch of IAccessible2 in sc/source/ui/navipi/navipi.cxx with + case FID_KILLEDITVIEW: + aLbEntries.ObjectFresh( SC_CONTENT_OLEOBJECT ); + aLbEntries.ObjectFresh( SC_CONTENT_DRAWING ); + aLbEntries.ObjectFresh( SC_CONTENT_GRAPHIC ); + break; even though we already have a handler of case SfxHintId::ScDrawChanged: m_xLbEntries->Refresh( ScContentId::GRAPHIC ); m_xLbEntries->Refresh( ScContentId::OLEOBJECT ); m_xLbEntries->Refresh( ScContentId::DRAWING ); break; and those "Refresh" as opposed to "ObjectFresh" try not to clear and refill the navigator unless content has changed. and that ObjectFresh seems to really center on the use of "sKeyString" which is only set by ScContentTree::KeyInputHdl (in that same commit) if space is used to select an entry with the comment "Make KEY_SPACE has same function as DoubleClick". I suspect ObjectFresh is a cut and paste job from Refresh that just wants to keep the same item selected (and expanded) in the navigator that was selected by the user before any content change that might cause the navigator to refill
Caolán McNamara committed a patch related to this issue. It has been pushed to "libreoffice-7-1": https://git.libreoffice.org/core/commit/59ec383c75837b017cd593adb2a85fba9b2970c5 Related: tdf#130326: GetDrawNames doesn't do anything when !bisInNavigatoeDlg It will be available in 7.1.4. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
May be I missed something? It takes 1 min 20 sec in today's daily build Version: 7.2.0.0.alpha0+ (x64) / LibreOffice Community Build ID: 15a9bee9ef26ce13ed1e26319306a88b6d886158 CPU threads: 4; OS: Windows 10.0 Build 18363; UI render: default; VCL: win Locale: ru-RU (ru_RU); UI: ru-RU Calc: threaded
Caolán McNamara committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/1a891f2fadae01aca95157b09d6ea8e223bee1ea Related: tdf#130326 allow bulk_insert_for_each to insert under a node It will be available in 7.2.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Caolán McNamara committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/c0cbd13c945c5f47057b413efc88b2e3bb72d025 Related: tdf#130326 drop ScContentTree::ObjectFresh It will be available in 7.2.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
(In reply to Julien Nabet from comment #37) > (In reply to Roman Kuznetsov from comment #36) > > ... > > Of course NO. If LO opens 2D objects with wrong sizes it means we just > > should fix it but not delete those objects. > > Do you mean Excel shows indeed 19562 text boxes with the initial xlsx file? > If yes, could you provide a screenshot? > If they don't appear in the screenshot, I think Jean-Baptiste is right and > we should delete these nonsense text boxes. I mean it doesn't matter Excel shows or not those objects. There are they there and we must import it as is. Another question if we import it wrong. Then we should fix it
(In reply to Roman Kuznetsov from comment #44) > (... > > I mean it doesn't matter Excel shows or not those objects. There are they > there and we must import it as is. > Another question if we import it wrong. Then we should fix it In comment 9, you talked about Excel which took 9sec to open the file so yes it matters if Excel shows them or not. If it doesn't show them perhaps it doesn't take these 19562 text boxes into account and it's indeed easier to open more quickly.
Caolán McNamara committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/99668e79c13fee09f8d776dbcd28e7667d7f68fe Related: tdf#130326 skip calling Refresh if its already just called It will be available in 7.2.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Caolán McNamara committed a patch related to this issue. It has been pushed to "libreoffice-7-1": https://git.libreoffice.org/core/commit/f275e019199baa931f1e4681e1bf29d68aae7a32 Related: tdf#130326 drop ScContentTree::ObjectFresh It will be available in 7.1.4. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Caolán McNamara committed a patch related to this issue. It has been pushed to "libreoffice-7-1": https://git.libreoffice.org/core/commit/5f5cf0d66d14ff246a4fce681459eb605d6c5f39 Related: tdf#130326 skip calling Refresh if its already just called It will be available in 7.1.4. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Created attachment 171520 [details] Flamegraph I don't know if there's still something we can improve here but here's an updated Flamegraph concerning file opening with master sources updated today + gen rendering.
34 sec for opening in Version: 7.2.0.0.alpha0+ / LibreOffice Community Build ID: a52590d76b89dc75be2aa87f4287624c89f1e82f CPU threads: 4; OS: Mac OS X 10.16; UI render: default; VCL: osx Locale: ru-RU (ru_RU.UTF-8); UI: en-US Calc: threaded and it's still has a delay when I try scrolling the sheet
(In reply to Roman Kuznetsov from comment #50) > 34 sec for opening in > ... > and it's still has a delay when I try scrolling the sheet There's still https://gerrit.libreoffice.org/c/core/+/114960 to wait for. After this if Noel hasn't submitted a new patch, I'll retrieve another Flamegraph hoping the new graph will provide other leads.
(In reply to Julien Nabet from comment #51) > (In reply to Roman Kuznetsov from comment #50) > > 34 sec for opening in > > ... > > and it's still has a delay when I try scrolling the sheet > > There's still https://gerrit.libreoffice.org/c/core/+/114960 to wait for. > After this if Noel hasn't submitted a new patch, I'll retrieve another > Flamegraph hoping the new graph will provide other leads. Argh, forget my last comment, this concerned tdf#79049.
(In reply to Roman Kuznetsov from comment #27) > OK, 43 sec, in > > Version: 7.2.0.0.alpha0+ (x64) / LibreOffice Community > Build ID: 77419c6f3aba1fd5b1660795923c22a39bdb1bad > CPU threads: 4; OS: Windows 6.1 Service Pack 1 Build 7601; UI render: > Skia/Raster; VCL: win > Locale: ru-RU (ru_RU); UI: ru-RU > Calc: CL > > I think CPU is matter here (Intel Core 2 Quad Q9450, 4 CPU core with 2.66GHz) Still the same time with the same CPU in Version: 7.4.0.0.alpha0+ (x64) / LibreOffice Community Build ID: ccb78b98e0618cce365562fe326d018892b8104a CPU threads: 4; OS: Windows 6.1 Service Pack 1 Build 7601; UI render: Skia/Raster; VCL: win Locale: ru-RU (ru_RU); UI: en-US Calc: CL Jumbo