On opening a 16M CSV file in calc I wondered why it was taking so long and why the PC's fans started blowing like crazy. On reviewing the output from top I could see that just one of the eight available cores was running at 100%, presumably why the fans kicked in, with LO. All the other cores were basically idle. Is there some way of making this more efficient? Is LO single threaded?
Do you have enable Menu/Tools/Options/LibreOffice calc/calculate - CPU threading ssettings.
Yes, it's on. Is calc the only part that offers multi CPU support?
(In reply to Owen Savill from comment #2) > Yes, it's on. Is calc the only part that offers multi CPU support? Calc uses multithreading only for calulating of many the same formulas. For parsing of CSV it doesn't work. In general LO is single threaded
Hi Michael, Do we use the fastparser for parsing CSV files ?
Calc has a single threaded CSV parser - that is correct. Given the general simplicity of the CSV file format, it seems likely that profiling that with kcachegrind and optimizing it further would probably yield more of a win than trying to thread it - but of course, its possible that a parse/insert separation would be helpful. It is also possible that some column iterators for data insertion might help optimize the mdds side. Can you paste a few rows of your CSV file ? how homogeneous is it ? are there large gaps in lots of the columns - or is it uniform ? Anyhow - an enhancement here - someone might like to hack on that: sc/source/filter/orcus/orcusfiltersimpl.cxx bool ScOrcusFiltersImpl::importCSV(ScDocument& rDoc, SfxMedium& rMedium) const Looks like a good place to poke. I believe much of the CSV parsing is inside the standalone orcus library which should be easy to hack on - but whether that's where the slowness is its hard to say without profiling =) HTH.
Owen Savill: Is this still an issue? If yes, would you please provide your comments to Michael Meeks's requests in comment 5?
Hi Noel, I thought this might interest you =) importing huge CSVs is probably something you've looked at its a big spreadsheet use-case; did you do any work on that since 2019 ? =)
Calc uses its own CSV import filter, not the one from orcus. Removing it from the "orcus bugs" meta issue.
> Is this still an issue? If yes, would you please provide your comments to Michael Meeks's Hi, Sorry about not responding earlier. I've just tried a 37Mb CSV file and it opens very quickly. Did you do something to the code?
> I've just tried a 37Mb CSV file and it opens very quickly Good, then let's mark this as RESOLVED WORKSFORME since we do not know which commit had fixed this.