Bug 65046 - Threaded calculation of formula groups (multi-threading)
Summary: Threaded calculation of formula groups (multi-threading)
Status: CLOSED WORKSFORME
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: LibreOffice (show other bugs)
Version:
(earliest affected)
Inherited From OOo
Hardware: Other All
: high enhancement
Assignee: Not Assigned
URL:
Whiteboard: BSA
Keywords:
Depends on:
Blocks: Calc-Enhancements Calc-Threaded
  Show dependency treegraph
 
Reported: 2013-05-27 17:11 UTC by Rinat
Modified: 2021-11-02 07:22 UTC (History)
20 users (show)

See Also:
Crash report or crash signature:


Attachments
The document on what I experience the slow down. (5.29 MB, application/vnd.oasis.opendocument.graphics)
2016-02-18 11:38 UTC, Michael Desnoyelles
Details
new test file (2.47 MB, application/vnd.oasis.opendocument.spreadsheet)
2021-11-02 06:18 UTC, Rinat
Details
new test file2 (964.19 KB, application/vnd.oasis.opendocument.spreadsheet)
2021-11-02 06:53 UTC, Rinat
Details
new test file3 (2.51 MB, application/vnd.oasis.opendocument.spreadsheet)
2021-11-02 07:04 UTC, Rinat
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Rinat 2013-05-27 17:11:41 UTC
Problem description: Libreoffice uses only one CPU out of many when it counts ods documents, decompreessing them or in any documents which includes big table.

Steps to reproduce:
1. Create Spreadsheet
2. Fill many cells with data or formulas
3. save it, close office, and open file again

Current behavior:
it work very slowly, uses one CPU core at 100% percent, and doesn't use rest CPU cores. This behavior is in linux and windows systems.

Expected behavior:
Libreoffice will use all avalable CPU cores for counting and decompressing and that will make it more efficient.
              
Operating System: Ubuntu
Version: 4.0.3.3 release
Comment 1 Tom 2013-05-28 06:05:13 UTC
PC - at home
Windows 7 x64 Home premium
Celeron E3300 2.5 GHz 2 cores
3GB ram

PC- at work (there is over 150, I can sit and work on any of it but the one I use mostly)
Windows 7 x64 Professional
Core i3 2.6 GHz 4 cores
4gb Ram

Laptop
Ubuntu 12.04 x64 and Windows 7 x64 Home Premium
AMD APU E-450 1.6 GHz 2 cores
4Gb ram
(Interesting that on laptop in Ubuntu LibreOffice, also using one core, works faster even though the laptop is slower than my PCs)
Comment 2 Robinson Tryon (qubit) 2014-02-04 20:21:57 UTC
(In reply to comment #0)
> Problem description: Libreoffice uses only one CPU out of many when it
> counts ods documents, decompreessing them or in any documents which includes
> big table.
> 
> Steps to reproduce:
> 1. Create Spreadsheet
> 2. Fill many cells with data or formulas
> 3. save it, close office, and open file again
> 
> Current behavior:
> it work very slowly, uses one CPU core at 100% percent, and doesn't use rest
> CPU cores. This behavior is in linux and windows systems.
> 
> Expected behavior:
> Libreoffice will use all avalable CPU cores for counting and decompressing
> and that will make it more efficient.

Sounds like an enhancement request.
Severity -> Enhancement
Comment 3 Rpnpif 2014-07-11 07:25:07 UTC
All documents of some pages with images in writer are displayed very slowly when browsing.
Better display accelerating is welcome and multithreading would help greatly.

Firefox is a good example of rapid displaying code for long pages with images. Perhaps is it reusable or it give ideas ?
Comment 4 Michael Desnoyelles 2016-02-18 11:38:32 UTC
Created attachment 122763 [details]
The document on what I experience the slow down.
Comment 5 Michael Desnoyelles 2016-02-18 15:53:32 UTC
Intel Core Quad CPU Q6600 @2.40GHz
8GB Ram
Win 10 pro x64

Trouble with Draw that slow down cause only one core is working at 100%. De document is freezing for a minor changement.

I use many different type of fonts and some 3D spheres
Comment 6 René Genz 2016-08-14 19:28:58 UTC
Calc experiences slow downs with charts

To recreate:
- open Calc and create small data set: rows: 0 to 2 in 0.001 steps (approximately 2000 rows); 5 columns, all start at 0, but have different increments
- select all data points from data set
- create chart: Chart Type=XY (Scatter) -- Points and Lines -- [Finish]
- GUI of LibreOffice considerably slows down after creation of graph

If chart is out of view, GUI is very fast again.


In LibreOffice 4.4.4.3 (distributed with Fedora 22) moving view to different rows is:
- slow, if complete chart is in view
- slow, if chart is partly visible
- fast, if chart is not in view


At least with version 5.1.x drawing speed has improved for me.


In LibreOffice 5.1.5.2 (distributed with Fedora 24) and in LibreOffice 5.2.0.4 (tested with Flatpak on Fedora 24) moving view to different rows is:
- fast, if complete chart is in view
- slow, if chart is partly visible
- fast, if chart is not in view


It seems the whole chart is redrawn, if part of it is visible. Then again if more is visible.
This results in slow work with those charts.
Maybe the chart can be used from memory all time and only be redrawn, if its data has been changed?


On the same machine (Intel Core 2 Duo E8500 @ 3.16GHz; 4 GB RAM; GeForce 9500 GS) Microsoft Office 2010 32 bit on Windows 7 32 bit is fast in all 3 cases.

LibreOffice on Windows on the same machine shows the behaviour described above.
Comment 7 Alexander Alzate 2016-10-21 20:18:39 UTC
Resource usage in LO is still a problem and I think it should be taken as an important task.

I promote the usage of LO for all document generation in the enterprise I work, but performance is a recurrent problem.

By implementing a serious multithreading system into LO and a better resource usage policy the LO reception in corporation environment would be better IMHO.
Comment 8 dfraipont 2017-08-31 16:46:32 UTC
Same here.
We really need it.
Comment 9 Xisco Faulí 2017-08-31 22:25:52 UTC
*** Bug 112151 has been marked as a duplicate of this bug. ***
Comment 10 dfraipont 2017-08-31 22:37:28 UTC
Hello,

Is it possible to add Multi-Threading?
At least 1 thread per document.
So when opening, editing or running macro in large document, this don't block all other opened?
Or add an argument to run multi process.
Comment 11 Tom B 2017-10-01 15:41:07 UTC
One place I would very much like to see multithreading is PDF export. When exporting a 100 slide Impress presentation to PDF it takes about 5 minutes. It would be a lot quicker if it scaled with multi-core/thread CPUs.
Comment 12 Xisco Faulí 2017-10-31 18:38:17 UTC
*** Bug 98184 has been marked as a duplicate of this bug. ***
Comment 13 Buovjaga 2017-11-02 14:50:43 UTC
*** Bug 112799 has been marked as a duplicate of this bug. ***
Comment 14 Kevin Suo 2017-11-30 15:03:00 UTC
*** Bug 114176 has been marked as a duplicate of this bug. ***
Comment 15 Agustín 2018-02-27 12:32:19 UTC
I have the same problem, 

- Problem description: Libreoffice uses only one CPU out of many when convert, concat, insert fields in odt documents, I use Unoruntime

- Installation:
Libreoffice 4.4

Java version "1.7.0_91" OpenJDK Runtime Environment (IcedTea 2.6.2) (suse-21.2-x86_64) OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode)

SUSE Linux Enterprise Server 12 SP1  (x86_64)
 
- Current behavior: (same as the first comment)
it work very slowly, uses one CPU core at 100% percent, and doesn't use rest CPU cores. This behavior is in linux and windows systems.

- Expected behavior: (same as the first comment)
Libreoffice will use all avalable CPU cores for counting and decompressing and that will make it more efficient.
Comment 16 Kevin Suo 2018-02-27 13:56:30 UTC
(In reply to Agustín from comment #15)
This bug is already confirmed, so do not change the status back to UNCONFIRNED.

Also, Muilti-Threading is already a work-in-process and is available on master branch for Calc. It was disabled on 6.0 branch at this moment because of bugs, but I expect this feature be enabled soon in releases.
Comment 17 Aron Budea 2018-02-28 02:02:11 UTC
(In reply to Kevin Suo from comment #16)
> Also, Muilti-Threading is already a work-in-process and is available on
> master branch for Calc. It was disabled on 6.0 branch at this moment because
> of bugs, but I expect this feature be enabled soon in releases.
Note that this is temporarily disabled by default on master with the following commit:
https://cgit.freedesktop.org/libreoffice/core/commit/?id=4696d3728f0aba1087001bc543fc0867dd0ebdda
Comment 18 Thomas Lendo 2019-02-24 21:19:24 UTC
Changing bug summary from 'libreoffice multithreading' to 'Threaded calculation of formula groups (multi-threading)'.
Comment 19 melutovich 2019-08-04 19:05:33 UTC
The summary was changed from 'libreoffice multithreading' to
to 'Threaded calculation of formula groups (multi-threading)'

However some of the bugs marked as a duplicate of this one fit better to the original summary.
Comment 20 Xisco Faulí 2019-11-11 12:13:03 UTC
*** Bug 128396 has been marked as a duplicate of this bug. ***
Comment 21 Xisco Faulí 2019-12-03 10:21:37 UTC
Changing priority to 'high' since the number of duplicates is higher than 5 or the number of people in CC higher than 20
Comment 22 b. 2020-10-04 05:51:52 UTC
as of now and afaik there are two threaded options available, 'openCL' for LO in general, and 'calc - calculate - threaded calculation' for calc, 

thus 'WFM'? 

there is a fantastic speedup in opening the sample file with v. 7.1 compared to 6.1, even 'unparallelized', 

thus 'WFM'? 

be careful, both parallelizing options contributed to at least two 'hard to spot' bugs, tdf#132451, tdf#131340, tdf#129753, tdf#124577, tdf#103532 
it's unclear if there are more ... 

'un-duping' 98184 as it's a different request to have 'open file' 'non-blocking' (be able to do (interactive) work on other files during the wait time) than to speed up things - and open file - by parallelizing computations, 

besides: acc. to the sample file the bug is about draw, doe's draw have 'formula groups'? 

besides: graphic work -> inkscape and .svg, 

besides: export from draw to .svg is ... poor ...
Comment 23 raal 2021-01-01 22:41:02 UTC
*** Bug 139355 has been marked as a duplicate of this bug. ***
Comment 24 coocoolatte 2021-07-02 16:28:21 UTC
I think the level of importance of this bug should be changed from "enhancement" to "major". Here is my rationale:
- The bug has not been addressed in 8 years; worse: it has remained in status "NEW" for 8 years!
- Nowadays, all the consumer-grade CPU vendors have split their CPUs' power in several slower cores rather than single faster ones.

This makes Calc barely usable in a professional environment with heavy calculation loads. I do my best to push everyone towards open source software, with some success, but in the case of Calc, it's becoming ridiculous.

In an environment where everybody has 8 core CPUs, my colleagues can launch heavy tasks (like 300,000 rows, each with a lookup in a 25,000 table) and have the result in about 20 minutes in Excel, while I have to wait 3 hours for the same thing (with LibreOffice 7 in Debian and the same hardware). I actually came to learn R just to avoid re-creating a Windows partition dedicated to Excel use on my computer.

Therefore, I believe that to be an Excel challenger taken seriously in the multicore-friendly XXIst century, Calc not being able to make use of those multicore CPUs is a major bug.

Note: Activating "Settings > Calc > Calculation > multi-thread" has no effect and enabling OpenCL is not persistent after LibreOffice required restart.
Comment 25 Aron Budea 2021-07-03 07:00:39 UTC
Let's consider this done, the bug report isn't actionable, threaded calculation is already implemented (META is bug 114159), if someone has specific performance issues either compared to other applications, or related to cores not being sufficiently utilized (at least in case of Calc formula), consider opening a new bug report.
Comment 26 pharmankur 2021-07-03 08:23:26 UTC
I Agree with @fredgib@free.fr on -->

Note: Activating "Settings > Calc > Calculation > multi-thread" has no effect and enabling OpenCL is not persistent after LibreOffice required restart.

-----------

Calc speed is not improved visibly since last 8 Yrs (since I started using it). Here I am not discrediting any contributions done under the hood which may be many! 

But net result for the user is not pleasing at all. In my system monitor (Linux Mint) I always get one core stuck at 100% even if so called multithreading is enabled ( I am on Libreoffice Version 7.1.4.2 )

For many of my Calculation intensive sheets and still prefer to copy-paste data on Google sheet, Get calculation data from there and paste the result back to Libreoffice calc. ... This takes pretty less time than applying same formulae in calc itself !

Other strategy I am following is :-
> First disable "Autocalculate" from Data --> calculate --> Autocalculate
> Do all data inputs in your sheets
> Then once all data is in the place, do Data --> calculate --> Recalculate HARD
 OR press Shift+Ctrl+F9
> Now wait for considerable time ( from "Plant a coffee seed --- to ---> Have coffee !" )
> Finally it gets over .

------------------------------------------

So I will NOT say the basic issue is resolved and will not go with @Aron Budea to "Consider" it Done.
Comment 27 Aron Budea 2021-07-03 08:48:42 UTC
That still isn't actionable, please do feel free to open new bug report(s) with specifics. (reproduction steps, expectations etc.)
Comment 28 Timur 2021-07-21 12:26:26 UTC
It would be useful if fredgib could attach his anonymized 300,000 rows sample.
Comment 29 coocoolatte 2021-10-30 13:52:09 UTC
(In reply to Aron Budea from comment #27)
> That still isn't actionable, please do feel free to open new bug report(s)
> with specifics. (reproduction steps, expectations etc.)

Sorry I am not very familiar with bug reporting, Bugzilla and LibreOffice development process...

What I have witnessed is exactly what is described in the original post of this thread, with all the versions since v6.0, with 6 or 8 intel cores and Debian Testing, therefore I don't see the added value of copy-pasting it in a new thread.

Moreover, maybe because English is not by mother tongue, I don't quite understand what you mean by "it is not actionable".

Could you please be more specific in a "newbie-friendly" sense?
Comment 30 Kevin Suo 2021-10-30 14:53:21 UTC
@fredgib:
So what exactly is not working at this stage? You mentioned that "What I have witnessed is exactly what is described in the original post of this thread". However, it was many years ago and things have changed a lot, see bug 114159 and https://bugs.documentfoundation.org/showdependencytree.cgi?id=139443. Citing the description of what has been reported many years ago does not help. You should re-test with a most recent release.

The point is, one should clearly describe what exactly is not working today with the multi-threading calculation in Calc, and if it does not work then one should describe which function/formula is not working, it provide clear and simple reproducible steps. Only in this way the devs can jump in and help to improve. In my opinion it does not matter if new bug reports should be opened for this purpose.
Comment 31 coocoolatte 2021-11-01 15:25:20 UTC
(In reply to Kevin Suo from comment #30)
> @fredgib:
> So what exactly is not working at this stage? You mentioned that "What I
> have witnessed is exactly what is described in the original post of this
> thread". However, it was many years ago and things have changed a lot, see
> bug 114159

I saw this, and like many users, I have however witnessed only one core used during processing by LibreOffice.

> and
> https://bugs.documentfoundation.org/showdependencytree.cgi?id=139443.

This is exactly what I experience too. But it was marked as a duplicate of THIS bug, so I thought THIS bug should be marked as "major", precisely because it had been marked as "enhancement" for 7 or 8 years without being addressed in a way that forced LibreOffice to actually use all the cores.

> Citing
> the description of what has been reported many years ago does not help. You
> should re-test with a most recent release.

As I mentioned implicitly by "with all the versions since v6.0", I tested it with all the versions since 6.0, to the present version 7.2.2, with exactly the same result.

> 
> The point is, one should clearly describe what exactly is not working today
> with the multi-threading calculation in Calc,

I think we are several to have done this, in https://bugs.documentfoundation.org/showdependencytree.cgi?id=139443, in this thread or in support forums (one example among several instances in several languages: https://ask.libreoffice.org/t/how-can-i-make-libreoffice-multi-threaded/18749)

> and if it does not work then
> one should describe which function/formula is not working, it provide clear
> and simple reproducible steps. 

I thought any resource-intensive task would show the problem. In case I was wrong and, as you suggest, it might depend on the nature of the task, here is what I daily struggle with:
1. create a 30,000-row and two-column file with:
  - first column being unique integers
  - second column being whatever text
2. create another similar file, this time with 300,000 rows.
3. In the second file, create a third column with a formula only having a VLOOKUP to fetch the second column's value in the first file, against the first column's integer.
4. monitor your CPU resource consumption (and have a cocktail): only one core is fully used, while the other are idle.

> Only in this way the devs can jump in and
> help to improve. In my opinion it does not matter if new bug reports should
> be opened for this purpose.

I agree. Whatever works to have some skilled people solve that issue...
Comment 32 Aron Budea 2021-11-02 01:09:17 UTC
(In reply to fredgib from comment #31)
> I thought any resource-intensive task would show the problem. In case I was
> wrong and, as you suggest, it might depend on the nature of the task, here
> is what I daily struggle with:
> 1. create a 30,000-row and two-column file with:
>   - first column being unique integers
>   - second column being whatever text
> 2. create another similar file, this time with 300,000 rows.
> 3. In the second file, create a third column with a formula only having a
> VLOOKUP to fetch the second column's value in the first file, against the
> first column's integer.
> 4. monitor your CPU resource consumption (and have a cocktail): only one
> core is fully used, while the other are idle.
Please create a new report, and include sample files created based on the description you gave. Set the new report to "block" the VLOOKUP meta bug: bug 109329.
Comment 33 Rinat 2021-11-02 06:18:55 UTC
Created attachment 176067 [details]
new test file
Comment 34 Kevin Suo 2021-11-02 06:35:57 UTC
(In reply to Rinat from comment #33)
I see your test file. Now please explain how to reproduce the bug? Please use 1,2,3...., and explain the current behaviour and your expected behavior.

E.g.:

Steps to Reproduce:

1. Open the file;
2. Set formula "=vlookup(?)" in cell ?
3. Fill down to the end of the column?
4. ???

Current Behaviour:
In step 3, the formula filling down operation takes ? minutes to finish and uses only 1 core of CPU.
???

Expected Behaviour:
Formula filling down operation in step 3 should finish in ? minutes and should use all my CPU cores.
???

Then copy-paste the libreoffice version information as shown in "Help -> About LibreOffice". Also state the operating system information (e.g. Ubuntu 18.04 LTS x64 etc), your GPU information. Be aware that this should be the system you have tested the bug.
Comment 35 Rinat 2021-11-02 06:53:22 UTC
Created attachment 176068 [details]
new test file2
Comment 36 Rinat 2021-11-02 07:04:20 UTC
Created attachment 176069 [details]
new test file3
Comment 37 Rinat 2021-11-02 07:22:34 UTC
I've made several test today.
I think the issue that I've reported 8 years ago is reduced. I also don't remember the spread sheet I used for reproduction. And a way to reproduce is changed from my experience too. 

New way to reproduce:
1. Create Spreadsheet
2. Fill many cells with data or formulas. BUT SET a limits for reproduction size!!
columns A-AZ, 20000 rows, up to 20 symbols per cell. Don't use too many cascaded formulas.
3. save it, close office, and open file again.

Currently it consumes up to 2 cores which is good.
Also new Libreoffice uses GPU, which was not available 8 years old version.

The opening speed for me is decent now. Issue I've reported is resolved.