Bug 104865 - FILTER : Calc Macro hangs when CopyOutputData = True
Summary: FILTER : Calc Macro hangs when CopyOutputData = True
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
5.2.3.3 release
Hardware: x86-64 (AMD64) Linux (All)
: medium major
Assignee: Not Assigned
URL:
Whiteboard: target:7.1.0 target:7.0.0.1 target:6.4.6
Keywords: haveBacktrace
Depends on:
Blocks: Macro
  Show dependency treegraph
 
Reported: 2016-12-22 15:32 UTC by paul.noel1
Modified: 2020-07-07 09:12 UTC (History)
8 users (show)

See Also:
Crash report or crash signature:


Attachments
Calc macro and file.ods (557 bytes, text/plain)
2016-12-23 09:58 UTC, paul.noel1
Details
File .ods (26.70 KB, application/vnd.oasis.opendocument.spreadsheet)
2016-12-23 10:22 UTC, paul.noel1
Details
bt with symbols (2.64 KB, text/plain)
2016-12-26 17:58 UTC, Julien Nabet
Details
bt with debug symbols (14.40 KB, text/plain)
2020-06-25 14:18 UTC, Julien Nabet
Details

Note You need to log in before you can comment on or make changes to this bug.
Description paul.noel1 2016-12-22 15:32:09 UTC
Hello,

When using 3 filters with the same operator "NOT_EQUAL" the macro hangs if the property "CopyOutputData" is set to True.
This occurs following the command : "sheet.filter()"

This is not the case with the operator "EQUAL"

Regards,
P.NOEL
Comment 1 Julien Nabet 2016-12-23 09:14:55 UTC
Would it be possible you attach a file with the example so we can try to reproduce this quickly? (by using this link: https://bugs.documentfoundation.org/attachment.cgi?bugid=104865&action=enter)

(have in mind that any attachment is automatically made public so remove private /confidential part from it).
Comment 2 paul.noel1 2016-12-23 09:58:45 UTC
Created attachment 129892 [details]
Calc macro and file.ods

The Calc file has 7 columns and my goal is to display only empty cells column G but being unable to use operator EMPTY I tried to achieve that by using operator NOT=EQUAL.

Macro hangs when operator is NOT_EQUAL and oFilterDesc.CopyOutputData is True
noe : French version 5.2.3.3 

Regards,
P. NOEL
Comment 3 Julien Nabet 2016-12-23 10:15:00 UTC
The file attached is a "bas" file which contains only the macro.
Could you also attach the ods file?
Comment 4 paul.noel1 2016-12-23 10:22:44 UTC
Created attachment 129893 [details]
File .ods

See attached file
Comment 5 Julien Nabet 2016-12-26 17:55:37 UTC
On pc Debian x86-64 with master sources updated today, I could reproduce the hang.
However, I could reproduce this either CopyOutputData is TRUE or False or operator is "NOT_EQUAL" or "EQUAL"
Comment 6 Julien Nabet 2016-12-26 17:58:44 UTC
Created attachment 129951 [details]
bt with symbols

I used "Ctrl-C" during the hang to retrieve this bt
Comment 7 Julien Nabet 2016-12-26 18:00:44 UTC
Eike: thought you might be interested in this one.

I noticed that nRealRow2 was equal to 1048575
(see http://opengrok.libreoffice.org/xref/core/sc/source/core/data/table3.cxx#2996).
Since its values comes from aParam which is initialized in lcl_PrepareQuery, thought there must something here.
http://opengrok.libreoffice.org/xref/core/sc/source/core/data/table3.cxx#2930
Comment 8 Julien Nabet 2016-12-26 18:15:14 UTC
Sorry aParam isn't initialized in lcl_PrepareQuery but upper in the stack, in ScCellRangeObj::filter, from opengrok.libreoffice.org/xref/core/sc/source/ui/unoobj/cellsuno.cxx#5691
Comment 9 QA Administrators 2018-10-22 02:50:20 UTC Comment hidden (obsolete)
Comment 10 Buovjaga 2019-08-11 11:49:21 UTC
Still crashes

Arch Linux 64-bit
Version: 6.4.0.0.alpha0+
Build ID: 37fc9f51a8de11d40632e8cda17ccf1fa4b1f503
CPU threads: 8; OS: Linux 5.2; UI render: default; VCL: gtk3; 
Locale: fi-FI (fi_FI.UTF-8); UI-Language: en-US
Calc: threaded
Built on 6 August 2019
Comment 11 Andreas Heinisch 2020-06-24 15:34:17 UTC
I investigated the error and I am not 100% sure what causes the problem:

The problem starts in https://opengrok.libreoffice.org/xref/core/sc/source/core/data/table2.cxx?r=8cad1620#3825 when nCol reaches the value 64. Then LO enters ScCellValue::release(https://opengrok.libreoffice.org/xref/core/sc/source/core/data/cellvalue.cxx?r=ca16def2#487) and ScColumn::DeleteContent(
https://opengrok.libreoffice.org/xref/core/sc/source/core/data/column3.cxx?r=a9753917&fi=DeleteContent#120). 

The line sc::CellStoreType::position_type aPos = maCells.position(nRow); fails with a call to:

template<typename _CellBlockFunc, typename _EventFunc>
typename multi_type_vector<_CellBlockFunc, _EventFunc>::position_type
multi_type_vector<_CellBlockFunc, _EventFunc>::position(size_type pos)
{
    if (pos == m_cur_size)

where pos = 140715552002943 and m_cur_size = 208.

HOWEVER, if I'll wait some seconds during the debug process, pos has always the correct value of 0 and the macro works.

Is there some timeout or resource waiting problem?
Comment 12 Julien Nabet 2020-06-25 14:18:31 UTC
Created attachment 162405 [details]
bt with debug symbols

Here's an updated bt with master sources updated today.
Comment 13 Julien Nabet 2020-06-25 14:20:56 UTC
bt shows:
Error: attempt to subscript container with out-of-bounds index 64, but 
container only holds 64 elements.

Objects involved in the operation:
    sequence "this" @ 0x0x327c220 {
      type = std::__debug::vector<std::unique_ptr<ScColumn, o3tl::default_delete<ScColumn> >, std::allocator<std::unique_ptr<ScColumn, o3tl::default_delete<ScColumn> > > >;
    }


#5  0x00007fffdc4ee2fc in ScTable::CopyData (this=0x327c220, nStartCol=0, nStartRow=0, nEndCol=1023, nEndRow=0, nDestCol=0, nDestRow=0, nDestTab=0) at sc/source/core/data/table2.cxx:3862
3862	                aCell.release(aCol[nDestX], nDestY);

Luboš/Noel: the bt retrieved today makes me think about some similar bugs you already fixed so thought you might be interested in this one.
Comment 14 Commit Notification 2020-07-01 13:00:41 UTC
Luboš Luňák committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/ee2d2184133b3bf47d38a03b14abab2caa15dad1

don't add a cell to a non-existent column (tdf#104865)

It will be available in 7.1.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 15 Xisco Faulí 2020-07-02 12:54:09 UTC
Hi Luboš Luňák,
LibreOffice still hangs in

Version: 7.1.0.0.alpha0+
Build ID: a330d8eef09a3135bf6ae94b31b7ea944f256bdc
CPU threads: 4; OS: Linux 4.19; UI render: default; VCL: gtk3
Locale: en-US (en_US.UTF-8); UI: en-US
Calc: threaded

For the record, before 
https://git.libreoffice.org/core/commit/ee2d2184133b3bf47d38a03b14abab2caa15dad1 LibreOffice crashes, now it hangs
Comment 16 Luboš Luňák 2020-07-02 13:15:28 UTC
It does not, it's just busy copying all cells that do not equal the criteria, which also all the empty cells in the sheet. If you add a condition that the cell shouldn't be empty, it's fine.
Comment 17 Xisco Faulí 2020-07-02 13:47:52 UTC
(In reply to Luboš Luňák from comment #16)
> It does not, it's just busy copying all cells that do not equal the
> criteria, which also all the empty cells in the sheet. If you add a
> condition that the cell shouldn't be empty, it's fine.

I see, should it be treat as a performance issue in another ticket?
I killed LibreOffice after 16 minutes and the macro was still running. Clicking on stop does nothing...
Comment 18 Noel Grandin 2020-07-02 14:00:04 UTC
(In reply to Xisco Faulí from comment #17)
> 
> I see, should it be treat as a performance issue in another ticket?


Its not a performance issue, as much as: if use you use jumbo sheets you are going to shoot yourself in the foot more easily.

There is very little we can do when people try to do calculations across 16 million rows.
Comment 19 Noel Grandin 2020-07-02 14:00:48 UTC
Oh never mind, ignore me, this is not a jumbo sheet problem.
Comment 20 Commit Notification 2020-07-02 14:14:40 UTC
Luboš Luňák committed a patch related to this issue.
It has been pushed to "libreoffice-7-0":

https://git.libreoffice.org/core/commit/edca1aa90ad1ec2f8959effc7198f227e3be91d4

don't add a cell to a non-existent column (tdf#104865)

It will be available in 7.0.0.1.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 21 Luboš Luňák 2020-07-02 14:53:40 UTC
(In reply to Xisco Faulí from comment #17)
> (In reply to Luboš Luňák from comment #16)
> > It does not, it's just busy copying all cells that do not equal the
> > criteria, which also all the empty cells in the sheet. If you add a
> > condition that the cell shouldn't be empty, it's fine.
> 
> I see, should it be treat as a performance issue in another ticket?

I don't understand BASIC scripting much, but AFAICT the macro checks the entire sheet for cells that are not equal to the given strings. And empty cells also are not equal to those strings. Since the macro asks to copy such data, LO ends up copying pretty much the entire sheet.

I'd probably keep this as closed until an actual user complains (and explains why LO shouldn't be slow in such a case).
Comment 22 Luboš Luňák 2020-07-02 14:55:50 UTC
Actually, maybe that's what the bugreport was originally about and the crash came only later.
Still, I find it questionable to complain that it takes long to copy an entire sheet based on filters (that are probably poorly specified to being with).
Comment 23 Xisco Faulí 2020-07-02 15:34:37 UTC
(In reply to Luboš Luňák from comment #22)
> Actually, maybe that's what the bugreport was originally about and the crash
> came only later.
> Still, I find it questionable to complain that it takes long to copy an
> entire sheet based on filters (that are probably poorly specified to being
> with).

checked it in LibreOffice 3.3 and the behaviour is not the same, so at least it's not a regression
Comment 24 Commit Notification 2020-07-07 09:12:42 UTC
Luboš Luňák committed a patch related to this issue.
It has been pushed to "libreoffice-6-4":

https://git.libreoffice.org/core/commit/6c3a6b9685acad171fda7bc5d44b0dd53b9f7268

don't add a cell to a non-existent column (tdf#104865)

It will be available in 6.4.6.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.