Bug Hunting Session
Bug 90653 - Opening .csv : Problem with rows
Summary: Opening .csv : Problem with rows
Status: RESOLVED INVALID
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
4.2.7.2 release
Hardware: Other All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard:
Keywords: haveBacktrace
Depends on:
Blocks: CSV-Import
  Show dependency treegraph
 
Reported: 2015-04-16 13:48 UTC by Mamoth
Modified: 2019-09-18 08:49 UTC (History)
3 users (show)

See Also:
Crash report or crash signature:


Attachments
The .csv file to open (3.15 MB, text/csv)
2015-04-16 13:48 UTC, Mamoth
Details
bt with debug symbols (10.34 KB, text/plain)
2019-08-10 11:38 UTC, Julien Nabet
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mamoth 2015-04-16 13:48:50 UTC
Created attachment 114824 [details]
The .csv file to open

Hello.

I am trying to import a .csv file (see attachment). The file has 16069 rows and is not especially large (3.3 Mb).
But when I open it with LO, it says "The data could not be loaded completely because the maximum number of characters per cell was exceeded."

First problem : Calc should not have any problem loading a file with this number of rows.

Second problem : 
After looking twice, I noticed that the last line of the Calc sheet does match the last line of the .csv file ( i.e. " La Taille de la Vente;37350;LA CELLE-GUENAND;INDRE-ET-LOIRE;CENTRE;491439.0;2217742.0;3811Z;Collecte des déchets non dangereux;;") but the row number does not match the one of the .csv file :
 - in the Calc sheet, the last row is number 15405 
 - in the .csv file, the last row is number 16069

So I suppose that some rows have been skipped during import.
Comment 1 Julien Nabet 2015-04-16 19:22:02 UTC
On pc Debian x86-64 with master sources updated today, I could reproduce this.

I noticed this on console:
warn:legacy.osl:23655:1:sc/source/ui/docshell/impex.cxx:602: lcl_appendLineData: data overflow
Comment 2 tommy27 2016-04-16 07:27:24 UTC Comment hidden (obsolete)
Comment 3 Mamoth 2016-04-18 07:36:32 UTC Comment hidden (obsolete)
Comment 4 Julien Nabet 2016-04-18 16:55:19 UTC Comment hidden (obsolete)
Comment 5 Mamoth 2016-04-18 18:59:04 UTC Comment hidden (obsolete)
Comment 6 libreoffice-bugs 2016-05-28 11:29:28 UTC
Problem still exists in 5.1.3.2
Comment 7 firfin 2017-07-17 13:20:29 UTC
Still present in LO Version: 5.1.6.2
Build ID: 1:5.1.6~rc2-0ubuntu1~xenial2 on Ubuntu 16.04

There are also similar bugs on the OOo bug tracker about this (see https://bz.apache.org/ooo/show_bug.cgi?id=123340 for instance) , so this might be inherited ( I am not runnig virtualbox anymore, so I can't easily check with the 3.3. version without wrecking my current install.)
Comment 8 Tristan Miller 2017-08-29 13:36:26 UTC
Using LibreOffice Calc 5.4.0.3, I can open Attachment 114824 [details].  However, that version of Calc still chokes on delimited text files that are a bit larger.
Comment 9 Andreas Säger 2017-10-30 13:27:25 UTC
The csv is invalid because it contains text values with line breaks that are indistinguishable from line breaks between 2 records. Any csv parser will fail with this.
Comment 10 Tristan Miller 2017-11-03 18:05:08 UTC
(In reply to Andreas Säger from comment #9)
> The csv is invalid because it contains text values with line breaks that are
> indistinguishable from line breaks between 2 records. Any csv parser will
> fail with this.

Could you elaborate on this?  True, the line breaks may be semantically wrong, but I don't see how they make the file syntactically incorrect or harder/impossible to parse.
Comment 11 QA Administrators 2018-11-04 03:58:24 UTC Comment hidden (obsolete)
Comment 12 Jonny Grant 2019-08-10 10:52:17 UTC
Hi
This bug is still present in Version: 6.0.7.3

"The data could not be loaded completely because the maximum number of characters per cell was exceeded."

The strange popup doesn't even say what line it is on. It should just continue after what ever issue it finds.


I can put a $100 bug-bounty on a fix roll out. can't believe this PR was filed 4 years ago, and still broken.
Comment 13 Julien Nabet 2019-08-10 11:38:45 UTC
Created attachment 153283 [details]
bt with debug symbols

On pc Debian x86-64 with master sources updated today, I could reproduce this.

I attached a bt just after:
warn:legacy.osl:21334:21334:sc/source/ui/docshell/impex.cxx:628: lcl_appendLineData: data overflow
Comment 14 Timur 2019-09-12 13:57:39 UTC
(In reply to Mamoth from comment #0)
>  - in the Calc sheet, the last row is number 15405 
>  - in the .csv file, the last row is number 16069
Last row number in LO depends on CSV open dialog options.
And in MSO Excel, which opens CSV without warning, the last row number is 16008.

On the other side, if open and saved with LO or MSO, csv remains at 16069. 

This file is not correct. I guess that's what Andreas referred to. 
Reporter should have given minimal example so that we know what a problem in LO is, not that we look for a problem in his file.
Comment 15 Tristan Miller 2019-09-13 07:50:18 UTC
Timur, you have closed this bug on the basis that the second problem reported by Mamoth is invalid.  But you are ignoring the fact that the first problem he reported (the unhelpful error message) is valid.  Even if there is a problem with the file, LibreOffice needs to inform the user in a way that allows them to understand and locate the problem.  I suggest this bug be reopened to deal with this first problem.  Incidentally, it is still reproducible for me with the following setup:

Version: 6.3.1.1
Build ID: 30(Build:1)
CPU threads: 12; OS: Linux 5.2; UI render: default; VCL: gtk3; 
Locale: en-GB (en_CA.UTF-8); UI-Language: en-US
Calc: threaded
Comment 16 Julien Nabet 2019-09-17 20:06:41 UTC
(In reply to Tristan Miller from comment #15)
> Timur, you have closed this bug on the basis that the second problem
> reported by Mamoth is invalid.  But you are ignoring the fact that the first
> problem he reported (the unhelpful error message) is valid.  Even if there
> is a problem with the file, LibreOffice needs to inform the user in a way
> that allows them to understand and locate the problem.  I suggest this bug
> be reopened to deal with this first problem.  Incidentally, it is still
> reproducible for me with the following setup:
> ...

LO gives some hint, a cell contains too much characters. So it means something prevented from the good split on the data.
LO does just csv file parsing to split at the right location depending on the entered parameters, it can't do a semantic analyze which would allow it to guess where a value should stop.
Comment 17 Tristan Miller 2019-09-18 08:25:11 UTC
(In reply to Julien Nabet from comment #16)
> LO gives some hint, a cell contains too much characters. So it means
> something prevented from the good split on the data.
> LO does just csv file parsing to split at the right location depending on
> the entered parameters, it can't do a semantic analyze which would allow it
> to guess where a value should stop.

The least it could do is indicate at which cell or which row the parsing failed.
Comment 18 Julien Nabet 2019-09-18 08:49:55 UTC
Tristan:the simplest thing would be to stop when there's a pb

Eike: do you think we may add some throw exception or at least an assert in https://opengrok.libreoffice.org/xref/core/sc/source/ui/docshell/impex.cxx?r=f805e0b9#636 in else part?

626  static bool lcl_appendLineData( OUString& rField, const sal_Unicode* p1, const sal_Unicode* p2 )
627  {
628      OSL_ENSURE( rField.getLength() + (p2 - p1) <= SAL_MAX_UINT16, "lcl_appendLineData: data overflow");
629      if (rField.getLength() + (p2 - p1) <= SAL_MAX_UINT16)
630      {
631          rField += OUString( p1, sal::static_int_cast<sal_Int32>( p2 - p1 ) );
632          return true;
633      }
634      else
635      {
636          rField += OUString( p1, SAL_MAX_UINT16 - rField.getLength() );
637          return false;
638      }
639  }

Indeed, putting more than SAL_MAX_UINT16 characters in a cell is more than suspect no ?