On opening a 16M CSV file in calc I wondered why it was taking so long and why the PC's fans started blowing like crazy.
On reviewing the output from top I could see that just one of the eight available cores was running at 100%, presumably why the fans kicked in, with LO. All the other cores were basically idle.
Is there some way of making this more efficient? Is LO single threaded?
Do you have enable Menu/Tools/Options/LibreOffice calc/calculate - CPU threading ssettings.
Yes, it's on. Is calc the only part that offers multi CPU support?
(In reply to Owen Savill from comment #2)
> Yes, it's on. Is calc the only part that offers multi CPU support?
Calc uses multithreading only for calulating of many the same formulas. For parsing of CSV it doesn't work. In general LO is single threaded
Do we use the fastparser for parsing CSV files ?
Calc has a single threaded CSV parser - that is correct. Given the general simplicity of the CSV file format, it seems likely that profiling that with kcachegrind and optimizing it further would probably yield more of a win than trying to thread it - but of course, its possible that a parse/insert separation would be helpful.
It is also possible that some column iterators for data insertion might help optimize the mdds side.
Can you paste a few rows of your CSV file ? how homogeneous is it ? are there large gaps in lots of the columns - or is it uniform ?
Anyhow - an enhancement here - someone might like to hack on that:
bool ScOrcusFiltersImpl::importCSV(ScDocument& rDoc, SfxMedium& rMedium) const
Looks like a good place to poke. I believe much of the CSV parsing is inside the standalone orcus library which should be easy to hack on - but whether that's where the slowness is its hard to say without profiling =)