Bug Hunting Session
Bug 43911 - FILEOPEN problem: error message mentions rows instead of columns for csv files with more than 1024 columns
Summary: FILEOPEN problem: error message mentions rows instead of columns for csv file...
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
3.4.4 release
Hardware: All All
: medium normal
Assignee: Eike Rathke
URL:
Whiteboard: target:3.6.0
Keywords:
: 40415 48516 (view as bug list)
Depends on:
Blocks: 39868
  Show dependency treegraph
 
Reported: 2011-12-17 08:30 UTC by Claudio Fontana
Modified: 2015-08-24 09:55 UTC (History)
6 users (show)

See Also:
Crash report or crash signature:


Attachments
100 column test file (105.57 KB, text/csv)
2011-12-28 22:02 UTC, cemyuceer
Details
2000 column test file (2.35 MB, text/csv)
2011-12-28 22:03 UTC, cemyuceer
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Claudio Fontana 2011-12-17 08:30:58 UTC
I saw in the LibreOffice 3.3 release notes that the problem with the spreadsheet rows limit was fixed, so I went on and removed OpenOffice 3.1 and replaced it with LibreOffice 3.4.

In general it seems to be better, but still when I try to import data from a CSV file, only the first 1000 rows are imported.
Comment 1 Markus Mohrhard 2011-12-26 06:02:28 UTC
Can't reproduce that problem with master/beta2.

Could you test with a 3.5 beta2 build? If the problem is still there it would be great if you could add a test document.
Comment 2 cemyuceer 2011-12-28 19:57:48 UTC
Although the error message mentions "max number of rows", I think the bug is related with "max number of columns" during CVS import. 
I have a CVS file that has 1600+ columns and 10 data rows. 
LibreOffice gives the above mentioned "maximum number of rows exceeded" error message for this file. 
Hence the bug should be related with "number of columns"
Comment 3 cemyuceer 2011-12-28 22:02:17 UTC
Created attachment 54932 [details]
100 column test file

Here is a 100 column CSV test file, which can be opened by LibreOffice
Comment 4 cemyuceer 2011-12-28 22:03:23 UTC
Created attachment 54933 [details]
2000 column test file

And, here is a 2000 column CSV test file, which LibreOffice can not open
Comment 5 Claudio Fontana 2011-12-29 01:04:15 UTC
(In reply to comment #2)
> Although the error message mentions "max number of rows", I think the bug is
> related with "max number of columns" during CVS import. 
> I have a CVS file that has 1600+ columns and 10 data rows. 
> LibreOffice gives the above mentioned "maximum number of rows exceeded" error
> message for this file. 
> Hence the bug should be related with "number of columns"

Yes, that is probably what happened to me as well.
The data I was importing had thousands of columns.
Comment 6 Korrawit Pruegsanusak 2012-01-05 04:39:22 UTC
Markus, is this a fix for this bug? or it's just a part of it?

http://cgit.freedesktop.org/libreoffice/core/commit/?id=51caf9b45403b92deadf934347f2d6f9ffecdf08

IOW, could we mark this bug as RESOLVED/FIXED?
Comment 7 Markus Mohrhard 2012-01-05 04:44:06 UTC
It is not a fix for this bug. It was just a fix for a related problem I discovered during a quick look at the code. There is no easy fix for this problem without more investigation.

Normally the commit message should read: related fdo#43911 but that did not fit into the commit message
Comment 8 cemyuceer 2012-01-05 05:04:14 UTC
Hi Markus,

Have you been able to test it, with the "2000 column test file" I have uploaded?

I think the quick fix of this problem is to increase the "Max Column Limit during CSV import". It seems that the limit is 1024 now. More practical limit would be 8192 or even better 65536.

Thank you,
Cem
Comment 9 Markus Mohrhard 2012-01-05 14:21:03 UTC
> I think the quick fix of this problem is to increase the "Max Column Limit
> during CSV import". It seems that the limit is 1024 now. More practical limit
> would be 8192 or even better 65536.
> 

No. We won't increase the column limit soon. First we would need to rework all internal cell datastructures like I did for sheets in 3.5. We still use fixed size containers for columns. That means that increasing the column limit will increase the the memory needed for every sheet extremelly. The column limit of 1024 is a design decision by us.

If you are really in need for LibO with a bigger column limit you can easily build one yourself and I tell you the line you need to change to increase the limit.

This bug report is about the wrong error message and not about the the column limit of 1024 in 3.4+, 256 till 3.3.
Comment 10 Claudio Fontana 2012-01-05 14:50:28 UTC
(In reply to comment #9)
> > I think the quick fix of this problem is to increase the "Max Column Limit
> > during CSV import". It seems that the limit is 1024 now. More practical limit
> > would be 8192 or even better 65536.
> > 
> 
> No. We won't increase the column limit soon. First we would need to rework all
> internal cell datastructures like I did for sheets in 3.5. We still use fixed
> size containers for columns. That means that increasing the column limit will
> increase the the memory needed for every sheet extremelly. The column limit of
> 1024 is a design decision by us.
> 
> If you are really in need for LibO with a bigger column limit you can easily
> build one yourself and I tell you the line you need to change to increase the
> limit.
> 
> This bug report is about the wrong error message and not about the the column
> limit of 1024 in 3.4+, 256 till 3.3.

I understand and ack your technical comments.
However, this bug report (which I opened) is about the data not being fully imported. At the beginning I thought it was a limit on the rows, but apparently it is a limit on the columns (possibly a wrong error message).
Fact of the matter is, the main issue for me was that I had to open and edit the file using Microsoft Office, because libreoffice could not open the file correctly for further editing.
Comment 11 cemyuceer 2012-01-05 15:07:48 UTC
> > > ...  It seems that the limit is 1024 now. More practical limit
> > > would be 8192 or even better 65536.
> > > 
> > 
> > No. We won't increase the column limit soon. ...  The column limit of
> > 1024 is a design decision by us.
> > 
> > This bug report is about the wrong error message and not about the the column
> > limit of 1024 in 3.4+, 256 till 3.3.
> 
> I understand and ack your technical comments.
> However, this bug report (which I opened) is about the data not being fully
> imported. At the beginning I thought it was a limit on the rows, but apparently
> it is a limit on the columns (possibly a wrong error message).
> Fact of the matter is, the main issue for me was that I had to open and edit
> the file using Microsoft Office, because libreoffice could not open the file
> correctly for further editing.

Hi Markus,
I also have found my way to this "bug report" only because I have a 1600+ column CSV file that I am trying to import into LibO.
So, our hopes is to convince developers to increase the "Max Column" limit, or that they switch to a dynamic size column container approach in order to use necessary memory.
Is it the final decision that the distributed official LibO version will have a "Max Column" limit of 1024 in the near future releases? And people like me will have to modify the "Max Column" limit and recompile/recreate a special LibO version for our needs?
Thank you,
Cem
Comment 12 Markus Mohrhard 2012-01-05 15:30:04 UTC
> > I understand and ack your technical comments.
> > However, this bug report (which I opened) is about the data not being fully
> > imported. At the beginning I thought it was a limit on the rows, but apparently
> > it is a limit on the columns (possibly a wrong error message).
> > Fact of the matter is, the main issue for me was that I had to open and edit
> > the file using Microsoft Office, because libreoffice could not open the file
> > correctly for further editing.
> 
> Hi Markus,
> I also have found my way to this "bug report" only because I have a 1600+
> column CSV file that I am trying to import into LibO.
> So, our hopes is to convince developers to increase the "Max Column" limit, or
> that they switch to a dynamic size column container approach in order to use
> necessary memory.

This is not an easy switch. This is more like half a heart transplant and therefore is nothing that will be done just from today to tomorrow. I needed around one month for the change for the sheet change and I think that changing the column container might take much more time.

> Is it the final decision that the distributed official LibO version will have a
> "Max Column" limit of 1024 in the near future releases? And people like me will
> have to modify the "Max Column" limit and recompile/recreate a special LibO
> version for our needs?

It is not a final decision but I doubt that we will increase the column limit if we did not change the column container to a dynamic container. In my dev builds a empty sheet already needs around 150 KB. For the change we need someone whohas enough time to work on it.

Feel free to step in here and help to improve the situation. You'll get as much help as possible by us and we will assist as much as we can.
Comment 13 cemyuceer 2012-01-05 15:49:22 UTC
> It is not a final decision but I doubt that we will increase the column limit
> if we did not change the column container to a dynamic container. In my dev
> builds a empty sheet already needs around 150 KB. For the change we need
> someone whohas enough time to work on it.
> 
> Feel free to step in here and help to improve the situation. You'll get as much
> help as possible by us and we will assist as much as we can.

I can spend like a day per week for contribution to the project.
Can you please, email me steps I should take?? And people I should be in contact with??
Thank you,
Cem
Comment 14 Markus Mohrhard 2012-01-05 16:30:50 UTC
(In reply to comment #13)
> > It is not a final decision but I doubt that we will increase the column limit
> > if we did not change the column container to a dynamic container. In my dev
> > builds a empty sheet already needs around 150 KB. For the change we need
> > someone whohas enough time to work on it.
> > 
> > Feel free to step in here and help to improve the situation. You'll get as much
> > help as possible by us and we will assist as much as we can.
> 
> I can spend like a day per week for contribution to the project.
> Can you please, email me steps I should take?? And people I should be in
> contact with??

http://wiki.documentfoundation.org/Development should contain all information needed to finish a first build. You should get help at #libreoffice dev or the dev mailing list if you need. It might be a good idea to start with an easy hack and then continue with something more difficult.
Comment 15 Markus Mohrhard 2012-04-10 13:04:25 UTC
*** Bug 48516 has been marked as a duplicate of this bug. ***
Comment 18 Kevin Hunter 2012-06-03 04:56:58 UTC
*** Bug 40415 has been marked as a duplicate of this bug. ***
Comment 19 Jean-Baptiste Faure 2012-06-03 12:46:45 UTC
Thank you for this fix. It resolves (in LO 3.6 only) a long standing OOo bug : https://issues.apache.org/ooo/show_bug.cgi?id=75199

Best regards. JBF