Description: The lastest master hangs while loading the attached spreadsheet. The hang is not CPU-bound, so might be a deadlock. REGRESSION -- earlier builds (including 6.0.0.0.alpha) did not hang Steps to Reproduce: 1. Download the attached "hang.ods" 2. /path/to/program/soffice hang.ods Do not enable macros (if prompted) Actual Results: LO window becomes unresponsive. "top" show a java process running with low CPu (1.7%). Expected Results: No hang. Reproducible: Always User Profile Reset: No Additional Info: Version: 6.1.0.0.alpha0+ Build ID: aa0b08980aba7bc82ab75151129b0c643cde7dfa CPU threads: 8; OS: Linux 4.10; UI render: default; VCL: gtk2; TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2017-12-04_23:24:32 Locale: en-US (en_US.UTF-8); Calc: group threaded User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0
Created attachment 138222 [details] hang.ods
Trying to reproduce it, I found sometimes it hungs and sometimes not and I found a reliable way of reproducing the hang: 1. Launch LibreOffice in safe-mode: instdir/program/soffice --safe-mode 2. Open the document -> It doesn't hang 3. Go to sheet COST -> it hangs Regression introduced by: author Tor Lillqvist <tml@collabora.com> 2017-08-29 22:11:06 +0300 committer Dennis Francis <dennis.francis@collabora.co.uk> 2017-11-21 16:09:40 +0530 commit facc02e40557d382f8509f42f2340a692678d0e3 (patch) tree 8aaf61c9f650aeabbabc50b07d2275114cd82e83 parent 91d0e65018300c259bba7b565bbf82f2e380784d (diff) Make formula group weight take number of cells referenced into account And not just the number of rows in the group. This means that even relatively short formula groups that calculate over large ranges of cells will be eligible for parallelized calculation. The weight of a formula group is the number of rows in the group times the weight of the formula. The weight of a formula is for now the number of cells referenced by all cell ranges in the formula, divided by 10. For instance, the weight of SUM(A1:B100) would be 20. If no cell ranges are used, the formula's weight is one. Bisected with: bibisect-linux64-6.0 Adding Cc: to Tor Lillqvist and Dennis Francis
It hangs when threading is enabled, it "only" is slow when threading is disabled. Also, any input on sheet COST leads to a waiting time. The conditional formatting is executed every time after input for each cell where it is applied and that is visible (plus an extra row which unfortunately hits here because the window freeze is just above the first row with a conditional format). The slowness is gone if the commit mentioned above is "undone" by just returning 1 in ScFormulaCell::GetWeight() as a test, thus not triggering a group calculation as it is smaller than mnOpenCLMinimumFormulaGroupSize. In this case, adding messages to the FormulaLogger (e.g, aScope.addGroupSizeThresholdMessage() that even calls GetWeight() again) for each cell a group was not executed is suboptimal.. The slowness is also related to the fact that each group is only 2 cells (rows 8 and 9) in each column and doing all group related setup and gathering values is much more overhead than just calculating the cells. The worst part then is the SoftwareInterpreter nonsense because it has no clue about the semantics of ranges and implicit intersections (which this document heavily uses) and the implications of arrays, and blindly replaces every range encountered with an array, forcing calculations to array mode where they should not, e.g. matrix divisions of entire columns just to pick one result element and throw a million away. Certainly the weight gathering should be reworked to not hit the threshold for just a very few cells. Even more necessarily the SoftwareInterpreterFunc::operator() needs to be taught when to replace a range with an array and when not.
(In reply to Eike Rathke from comment #3) > The worst part then is the SoftwareInterpreter nonsense because it has no > clue about the semantics of ranges and implicit intersections (which this > document heavily uses) and the implications of arrays, and blindly replaces > every range encountered with an array, forcing calculations to array mode > where they should not, e.g. matrix divisions of entire columns just to pick > one result element and throw a million away. > > Even more necessarily the SoftwareInterpreterFunc::operator() needs to be > taught when to replace a range with an array and when not. I was wrong on this. Actually it is ScGroupTokenConverter::convert() (called from ScFormulaCell::InterpretFormulaGroup()) that blindly forces the array. SoftwareInterpreterFunc::operator() only takes the generated formula::svDoubleVectorRef to create the matrix tokens.
Eike Rathke committed a patch related to this issue. It has been pushed to "master": http://cgit.freedesktop.org/libreoffice/core/commit/?id=da81ec6862490a87bc240edb81c0a9b5fd04a1b3 Do not recalculate the same weight over and over again, tdf#114251 related It will be available in 6.1.0. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
That ^^^ commit does not solve anything, it's just something I spotted while debugging.
Eike Rathke committed a patch related to this issue. It has been pushed to "master": http://cgit.freedesktop.org/libreoffice/core/commit/?id=ecac94f3a6872538c19827603b52f1ed7e671a4c Type svIndex does not necessarily mean range name, tdf#114251 related It will be available in 6.1.0. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Eike Rathke committed a patch related to this issue. It has been pushed to "master": http://cgit.freedesktop.org/libreoffice/core/commit/?id=3193e78aba75c5b38d2683d9fd8593820048e8b9 Related: tdf#114251 disable ocName named expressions for vectorization It will be available in 6.1.0. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
@Dennis, Tor: loading with threading enabled and switching to sheet COST gave me sc/source/core/data/formulacell.cxx:2582: void ScFormulaCell::MaybeInterpret(): Assertion `!pDocument->mbThreadedGroupCalcInProgress' failed. For cell at position {nRow = 8, nCol = 17, nTab = 7} via ScInterpreter::GetDouble() with svDoubleRef generating an implicit intersection with range {aStart = {nRow = 0, nCol = 10, nTab = 7}, aEnd = {nRow = 1048537, nCol = 10, nTab = 7}} => {nRow = 8, nCol = 10, nTab = 7}
Eike Rathke committed a patch related to this issue. It has been pushed to "master": http://cgit.freedesktop.org/libreoffice/core/commit/?id=67444cbe5dae6e24db776ab712017ad063319276 Related: tdf#114251 disable vectorization of svDoubleRef It will be available in 6.1.0. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Eike Rathke committed a patch related to this issue. It has been pushed to "master": http://cgit.freedesktop.org/libreoffice/core/commit/?id=99210a149c859fcd683870b280adaeeffd1250e4 Unit test for implicit intersection with shared formulas, tdf#114251 related It will be available in 6.1.0. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Eike Rathke committed a patch related to this issue. It has been pushed to "libreoffice-6-0": http://cgit.freedesktop.org/libreoffice/core/commit/?id=14ce530d467a21d2d7fdae5272c5cc4b83104bdc&h=libreoffice-6-0 Do not recalculate the same weight over and over again, tdf#114251 related It will be available in 6.0.0.1. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Eike Rathke committed a patch related to this issue. It has been pushed to "libreoffice-6-0": http://cgit.freedesktop.org/libreoffice/core/commit/?id=22cfc4385ed2e272013fc300f593ab9fae61f3b2&h=libreoffice-6-0 Related: tdf#114251 disable ocName named expressions for vectorization It will be available in 6.0.0.1. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Eike Rathke committed a patch related to this issue. It has been pushed to "libreoffice-6-0": http://cgit.freedesktop.org/libreoffice/core/commit/?id=21575d3bd1caa615687f44982ad2092e9c9290f1&h=libreoffice-6-0 Related: tdf#114251 disable vectorization of svDoubleRef It will be available in 6.0.0.1. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Eike Rathke committed a patch related to this issue. It has been pushed to "libreoffice-6-0": http://cgit.freedesktop.org/libreoffice/core/commit/?id=ee7c13e134bfa620535714721085c1c0856a84df&h=libreoffice-6-0 Type svIndex does not necessarily mean range name, tdf#114251 related It will be available in 6.0.0.1. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Eike Rathke committed a patch related to this issue. It has been pushed to "master": http://cgit.freedesktop.org/libreoffice/core/commit/?id=b366adcf5aca8bb22ccd0dd270ab08e9c8976b5b Invert logic, include only known good, tdf#114251 related It will be available in 6.1.0. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Eike Rathke committed a patch related to this issue. It has been pushed to "libreoffice-6-0": http://cgit.freedesktop.org/libreoffice/core/commit/?id=16d75204418455c0d9dff13607432dae33a7566e&h=libreoffice-6-0 Invert logic, include only known good, tdf#114251 related It will be available in 6.0.0.2. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Created attachment 138673 [details] hang2.ods - see comment #19
Unfortunately something is still wrong. I downloaded master, built 2017-12-25_23:59:18 (I don't know for certain that it has the patches, but several days have elapsed). The original "hang.ods" no longer hangs when loading, but a newer version of that spreadsheet (attached) does STEPS TO REPRODUCE: 1. Download the attached "hang2.ods" (not hang.ods) 2. /path/to/soffice hang2.ods 3. If prompted about macros, click "Disable Macros" (Hangs with empty white window. Does not respond to clicking the window-manager close button) 4. ^C to kill the hung process 5. /path/to/soffice hang2.ods 6. When prompted to send crash report, click "Don't Send". 7. If prompted, click "Disable Macros" pure virtual method called pure virtual method called terminate called recursively terminate called without an active exception
Can open hang2.ods with no problems, but is see a lot of value errors Version: 6.1.0.0.alpha0+ Build ID: 2bf1cc7372088ec31ac5f0fb60de57feda59d3b7 CPU threads: 8; OS: Linux 4.9; UI render: default; VCL: kde4; Locale: nl-BE (en_US.UTF-8); Calc: group threaded (homebuild) But have a crash with, crash report has been sent Version: 6.1.0.0.alpha0+ Build ID: 86a939d5fd93173949447337738fa0625803ec92 CPU threads: 8; OS: Linux 4.9; UI render: default; VCL: kde4; TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2017-12-25_09:06:24 Locale: nl-BE (en_US.UTF-8); Calc: group threaded
The #VALUE! errors are expected if macros are disabled, because most of the formulas directly or indirectly call a user-defined function written in Basic. If you can load the spreadsheet with macros disabled, please try with macros ENABLED, to test formula re-evaluation and conditional formats. For a complete test (requires linux/unix): 1. Download the attachment "tickers.csv" and place it in $HOME/Jts/tickers.csv 2, Run soffice hang2.ods ENABLE macros when prompted (if not prompted, go to tools->options->Security->Macro Security and select "medium", click OK, then File->Reload) 3. Click the yellow "Reload tickers.csv" button on any sheet This WFM on LO 5.5 or earlier, but hangs when loading on the most recent master build: Version: 6.1.0.0.alpha0+ Build ID: a9b202a6b7000e7af34f2a639ca207122a3968bf CPU threads: 12; OS: Linux 4.13; UI render: default; VCL: gtk2; TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2017-12-26_23:09:36 Locale: en-US (en_US.UTF-8); Calc: group threaded
Created attachment 138700 [details] tickers.csv - see comment #23
Oops, wrong instructions in comment #21 -- it won't find tickers.csv as-is. This works: 1. Download tickers.csv and put it anywhere you like 2. open hang2.ods and Enable macros. 3. View the _PROTO sheet (click the second button on the bottom) 4. Modify cell AE4 to contain the path to tickers.csv (use Control-Z to undo LO's helpful removal of forward slashes :-) 5. Click the yellow "Reload tickers.csv" button If all goes well you should see a new "HD" sheet with valid cell values. I'm sorry for posting wrong instructions
Confirm hang with macros enabled Version: 6.1.0.0.alpha0+ Build ID: 2bf1cc7372088ec31ac5f0fb60de57feda59d3b7 CPU threads: 8; OS: Linux 4.9; UI render: default; VCL: kde4; Locale: nl-BE (en_US.UTF-8); Calc: group threade Can load document with macros enabled in Version: 5.4.5.0.0+ Build ID: a61f45499856aad9910d82af1312a163504c15c2 CPU threads: 8; OS: Linux 4.9; UI render: default; VCL: kde4; Locale: nl-BE (en_US.UTF-8); Calc: group but sheet hd result with a lot of #value errors
Not sure if this is relevant (and the right place), but both files are also crashing when clicking on a button - for example - (%ITM, TV%) at the sheet VRSK, with Macro's disabled and without the separate CSV Version: 6.1.0.0.alpha0+ Build ID: a9b202a6b7000e7af34f2a639ca207122a3968bf CPU threads: 4; OS: Windows 6.3; UI render: default; TinderBox: Win-x86@42, Branch:master, Time: 2017-12-26_23:10:23 Locale: nl-NL (nl_NL); Calc: CL
Created attachment 138718 [details] hang3.ods - please use this one (removed sheets with old broken formulas)
There were some bad formulas on some sheets in the prior demos, so I uploade a new one (hang3.ods). Please use that one to avoid being distracted. Running the 12/26 non-dbg master, hang3.ods segfaults while opening. See gdbtrace-nodbg-hang3-master2017-12-26.log Running 12/28 dbg master build, hang3.ods gets assertion failures and subsequent SIGABRTs See gdbtrace-dbg-hang3-master2017-12-28.log The exact build info is in the gdb log files.
Created attachment 138719 [details] gdbtrace-nodbg-hang3-master2017-12-26.log (segfault; no symbols)
Created attachment 138720 [details] --backtrace log of assert fails opening hang3.ods on 12/28 WITH debug syms
Created attachment 138721 [details] gdbtrace-dbg-hang3-master2017-12-28.log (assert fails)
Eike Rathke committed a patch related to this issue. It has been pushed to "libreoffice-5-4": http://cgit.freedesktop.org/libreoffice/core/commit/?id=174822c33c8d35d514fbb713e6a22d20559bb034&h=libreoffice-5-4 Related: tdf#114251 disable ocName named expressions for vectorization It will be available in 5.4.5. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
> It will be available in 5.4.5. Please be careful. 5.4.x is the only version which currently works well with this spreadsheet, so I'd hate to see the problems currently plaguing master creep into that branch!
Eike Rathke committed a patch related to this issue. It has been pushed to "libreoffice-5-4": http://cgit.freedesktop.org/libreoffice/core/commit/?id=0443eb8bf5ed79d13ec1aa49ac83ed2b2e8eea54&h=libreoffice-5-4 Type svIndex does not necessarily mean range name, tdf#114251 related It will be available in 5.4.5. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Eike Rathke committed a patch related to this issue. It has been pushed to "libreoffice-5-4": http://cgit.freedesktop.org/libreoffice/core/commit/?id=811b40327c15e197b0967393e4071f48a60af42a&h=libreoffice-5-4 Related: tdf#114251 disable vectorization of svDoubleRef It will be available in 5.4.5. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
master builds still hang while loading "hang3.ods" (attached). Version: 6.1.0.0.alpha0+ Build ID: e29565272c57c6cd09cb015c2e66bfe8a1f4d7d2 CPU threads: 12; OS: Linux 4.13; UI render: default; VCL: gtk2; TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2018-01-29_23:29:16 Locale: en-US (en_US.UTF-8); Calc: group threaded I also re-tested with the latest 5.4 build (5.4.5.0.0+ 2018-01-30_10:34:41), since some patches were submitted to 5.4 per comment #31,32,33, 34. Everything seems normal, i.e. no hang or other problems. FWIW I never observed this bug on a 5.4 build.
My commits didn't claim to fix anything, they only touched related areas, hence the "Related" in the commit message, which also lead to overeager vectorization in earlier releases without causing this "hang" (which it is not, it is just unbearable slow, eventually it will finish). The real cause is triggering the threaded group interpreter here, see my comment 3 and comment 4. Something for Dennis to fix.
Can't repro in master now. There were a lot of fixes regarding cpu threading recently, may this is fixed now. Can anyone confirm ?
Confirming that this seems to have been fixed in latest master. Version: 5.5.0.0.alpha0+ Build ID: 514fd7073cbed62a01c8cecfd72cd65e810d679d CPU threads: 12; OS: Linux 4.15; UI render: default; VCL: gtk2; TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2017-05-22_03:04:13 Locale: en-US (en_US.UTF-8); Calc: group
(In reply to Jim Avera from comment #38) > Confirming that this seems to have been fixed in latest master. OOPS! Ignore my previous comment (it connected to a running process which was executing an older v5.5 LO which works).
Master loads the spreadsheet successfully, but still hangs when using macros to change the data (it might just be super-slow, don't know; but it sits there using 100% CPU). STEPS TO REPRODUCE: 1. Download tickers.csv and hang3.ods (attached) 2. Run the test described in comment 23 but use hang3.ods Namely: load hang3.ods and ENABLE macros On sheet "_PROTO", edit cell AE4 to contain the path to tickers.csv (use control-Z to undo unhelpful auto-corrections) Click the yellow "Reload" button RESULTS: 100% CPU "hang" This spreadsheet works fine using LO 5.5
Dear Jim Avera, To make sure we're focusing on the bugs that affect our users today, LibreOffice QA is asking bug reporters and confirmers to retest open, confirmed bugs which have not been touched for over a year. There have been thousands of bug fixes and commits since anyone checked on this bug report. During that time, it's possible that the bug has been fixed, or the details of the problem have changed. We'd really appreciate your help in getting confirmation that the bug is still present. If you have time, please do the following: Test to see if the bug is still present with the latest version of LibreOffice from https://www.libreoffice.org/download/ If the bug is present, please leave a comment that includes the information from Help - About LibreOffice. If the bug is NOT present, please set the bug's Status field to RESOLVED-WORKSFORME and leave a comment that includes the information from Help - About LibreOffice. Please DO NOT Update the version field Reply via email (please reply directly on the bug tracker) Set the bug's Status field to RESOLVED - FIXED (this status has a particular meaning that is not appropriate in this case) If you want to do more to help you can test to see if your issue is a REGRESSION. To do so: 1. Download and install oldest version of LibreOffice (usually 3.3 unless your bug pertains to a feature added after 3.3) from http://downloadarchive.documentfoundation.org/libreoffice/old/ 2. Test your bug 3. Leave a comment with your results. 4a. If the bug was present with 3.3 - set version to 'inherited from OOo'; 4b. If the bug was not present in 3.3 - add 'regression' to keyword Feel free to come ask questions or to say hello in our QA chat: https://kiwiirc.com/nextclient/irc.freenode.net/#libreoffice-qa Thank you for helping us make LibreOffice even better for everyone! Warm Regards, QA Team MassPing-UntouchedBug
Thanks Eike, the hang is gone, and all seems to be working now! Version: 6.4.0.0.alpha0+ Build ID: f75c2b04785aa05cff3bcd52689feb7400a14e8e CPU threads: 12; OS: Linux 4.18; UI render: default; VCL: gtk3; TinderBox: Linux-rpm_deb-x86_64@86-TDF, Branch:master, Time: 2019-06-15_11:49:26 Locale: en-US (en_US.UTF-8); UI-Language: en-US Calc: threaded