100244 – CRASH: Pivot table seems to cause massive memory leak

Bug 100244 - CRASH: Pivot table seems to cause massive memory leak

Summary: CRASH: Pivot table seems to cause massive memory leak

Status:	NEW

Alias:	None

Product:	LibreOffice
Classification:	Unclassified
Component:	Calc (show other bugs)
Version: (earliest affected)	Inherited From OOo
Hardware:	All All

Importance:	high major
Assignee:	Not Assigned

URL:
Whiteboard:
Keywords:	haveBacktrace, perf

Depends on:
Blocks:	Pivot-Table Memory Crash
	Show dependency tree / graph

Reported:	2016-06-06 21:06 UTC by Bob
Modified:	2024-03-30 20:29 UTC (History)
CC List:	11 users (show)

See Also:	131720 126710
Crash report or crash signature:

Attachments
ods with pivot table that causes memory leak (475.26 KB, application/vnd.oasis.opendocument.spreadsheet) 2016-06-06 21:06 UTC, Bob	Details
Simplified smaller version of the original document (209.92 KB, application/vnd.oasis.opendocument.spreadsheet) 2016-10-09 16:15 UTC, Dennis Francis	Details
bt at random with debug symbols (7.35 KB, text/plain) 2017-08-12 22:55 UTC, Julien Nabet	Details
perf flamegraph with simplified example (66.97 KB, application/x-bzip) 2019-09-24 19:13 UTC, Julien Nabet	Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Bob 2016-06-06 21:06:46 UTC

Created attachment 125520 [details]
ods with pivot table that causes memory leak

Refreshing the pivot table in the attached file causes LO memory consumption to spike.  Similar results between Windows 10 and Ubuntu 14.04 on the 5.1.3.2 release.

Warning:  Be prepared to kill LO when you refresh this pivot table!  If you have a system with LOTS of memory, it will eventually refresh - for me, resident memory reaches 0.05t with virtual over 52g (on linux "top").

A strange/temporary "fix" is to "Edit Layout.." and remove column "item_num" *prior* to Refreshing pivot table.  OR remove the "Data Fields" where a "Difference From" display option is set.

Note:  The source range is only 8300 rows by 12 columns, so not "huge" by any means.

My guess is that its some combination of "Row fields" and the "Difference From" that I'm using in the "Data Fields" that is causing the issue.

-Bob

Comment 1 raal 2016-06-07 09:52:33 UTC

Version: 5.2.0.0.alpha1+; win7  crash with "bad allocation" error

Comment 2 Katarina Behrens (Inactive) 2016-06-10 10:39:34 UTC

Is this a regression? i.e. did it ever work reasonably and only got worse in recent releases?

Comment 3 Bob 2016-06-10 20:24:09 UTC

This is the first time I've experienced this problem - and I've never tried using the "Difference From" display option before now.

I did test it using OpenOffice 3.2 on Ubuntu 14.04, and it behaved badly as well.

Comment 4 Julien Nabet 2016-06-19 08:32:28 UTC

On pc Debian x86-64 with master sources updated yesterday, I got an hang.

I noticed a lot of these before the hang:
warn:sc:4374:1:sc/source/core/data/dptabsrc.cxx:2696: ScDPMember::GetItemData: what data? nDim 15, mnDataId 2830

Comment 5 Dennis Francis 2016-10-09 16:13:43 UTC

I am working on this one and have some updates.

I have created a reduced row and simpler version of the original document(attached) such that it cause large memory usage on refresh of pivot table but fit in my system's RAM (16GB) even when calc run with valgrind-memcheck. It uses about lots of memory, but valgrind's memcheck tool shows no big leaks (all are "definitely lost ones are in KB's").

valgrid's massif tool reports that large number of allocations happen at 
ScDPResultDimension::AddMember(ScDPParentDimData const&) (dptabres.cxx:3961)

However I observed that the high memory usage after pivot table refresh does not go away even after I close the document (with calc still running with no other document). After some code hunting I suspect this behavior is due to holding of ScDPSource object inside ScDPObject object via a uno Reference which may be shared with other objects which do not get de-allocated while closing the document having pivot tables. It might only be released after calc is closed. This would explain the valgrind-memcheck's result.

I will next try to trace the reference count changes of ScDPSource object via gdb.

Calc version - latest master built 7th Oct 2016.
OS : Fedora 24 64 bit.

Comment 6 Dennis Francis 2016-10-09 16:15:31 UTC

Created attachment 127895 [details]
Simplified smaller version of the original document

Comment 7 Julien Nabet 2017-06-05 14:30:56 UTC

Dennis: any update here?

Comment 8 Dennis Francis 2017-06-05 14:57:30 UTC

Sorry, I did not proceed further. I will take a look again soon. Till then I am changing the status to NEW and have reset the assignee, so others have a chance to work on this.

Comment 9 Xisco Faulí 2017-06-19 10:02:48 UTC

it crashes for me when I click on cust_name pivot table

Version: 6.0.0.0.alpha0+
Build ID: 9e4502f0e393d2bc2810488b3ebb0a5c23038436
CPU threads: 1; OS: Windows 6.1; UI render: default; 
TinderBox: Win-x86@42, Branch:master, Time: 2017-06-16_08:52:00
Locale: es-ES (es_ES); Calc: group

In

Version: 6.0.0.0.alpha0+
Build ID: 08f6f9dded1b142b858c455da03319abac691655
CPU Threads: 4; OS Version: Linux 4.8; UI Render: default; VCL: gtk2; 
Locale: ca-ES (ca_ES.UTF-8); Calc: group

it hangs

Comment 10 Julien Nabet 2017-08-12 22:55:39 UTC

Created attachment 135505 [details]
bt at random with debug symbols

On pc Debian x86-64 with master sources updated yesterday, it hangs when clicking on cust_name with the second attachment.

I attached a bt at random

Comment 11 QA Administrators 2018-09-20 02:51:07 UTC Comment hidden (obsolete)

** Please read this message in its entirety before responding **

To make sure we're focusing on the bugs that affect our users today, LibreOffice QA is asking bug reporters and confirmers to retest open, confirmed bugs which have not been touched for over a year.

There have been thousands of bug fixes and commits since anyone checked on this bug report. During that time, it's possible that the bug has been fixed, or the details of the problem have changed. We'd really appreciate your help in getting confirmation that the bug is still present.

If you have time, please do the following:

Test to see if the bug is still present with the latest version of LibreOffice from https://www.libreoffice.org/download/

If the bug is present, please leave a comment that includes the information from Help - About LibreOffice.

If the bug is NOT present, please set the bug's Status field to RESOLVED-WORKSFORME and leave a comment that includes the information from Help - About LibreOffice.

Please DO NOT

Update the version field
Reply via email (please reply directly on the bug tracker)
Set the bug's Status field to RESOLVED - FIXED (this status has a particular meaning that is not
appropriate in this case)

If you want to do more to help you can test to see if your issue is a REGRESSION. To do so:
1. Download and install oldest version of LibreOffice (usually 3.3 unless your bug pertains to a feature added after 3.3) from http://downloadarchive.documentfoundation.org/libreoffice/old/

2. Test your bug
3. Leave a comment with your results.
4a. If the bug was present with 3.3 - set version to 'inherited from OOo';
4b. If the bug was not present in 3.3 - add 'regression' to keyword

Feel free to come ask questions or to say hello in our QA chat: https://kiwiirc.com/nextclient/irc.freenode.net/#libreoffice-qa

Thank you for helping us make LibreOffice even better for everyone!

Warm Regards,
QA Team

MassPing-UntouchedBug

Comment 12 Roman Kuznetsov 2018-09-20 12:02:31 UTC

still repro CRASH without any crashreport in 

Version: 6.2.0.0.alpha0+
Build ID: ace6bbf3da9ae27aca87865b6be887a3aed341fc
CPU threads: 4; OS: Windows 6.1; UI render: default; 
TinderBox: Win-x86@42, Branch:master, Time: 2018-09-20_05:45:56
Locale: ru-RU (ru_RU); Calc: threaded

@Xisco: why isn't it critical?

Comment 13 Xisco Faulí 2018-09-20 18:36:31 UTC

> @Xisco: why isn't it critical?

It's an old one, inherit from OOo times...

Comment 14 Julien Nabet 2019-04-21 08:42:00 UTC

On pc Debian x86-64 with master sources updated today, I gave it a new try with initial Bob's file.

I noticed these logs:
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 0
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 1
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 2
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 3
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 4
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 5
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 6
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 7
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 8
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 9
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 10
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 11
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 12
etc.

+ this bt at random:
#0  0x00007fffddc4f410 in ScDPResultMember::InitFrom(std::__debug::vector<ScDPDimension*, std::allocator<ScDPDimension*> > const&, std::__debug::vector<ScDPLevel*, std::allocator<ScDPLevel*> > const&, unsigned long, ScDPInitState&, bool) (this=0x5556094292b0, ppDim=std::__debug::vector of length 5, capacity 8 = {...}, ppLev=std::__debug::vector of length 5, capacity 8 = {...}, nPos=4, rInitState=..., bInitChild=true)
    at /home/julien/lo/libreoffice/sc/source/core/data/dptabres.cxx:1077
#1  0x00007fffddc556f9 in ScDPResultDimension::InitFrom(std::__debug::vector<ScDPDimension*, std::allocator<ScDPDimension*> > const&, std::__debug::vector<ScDPLevel*, std::allocator<ScDPLevel*> > const&, unsigned long, ScDPInitState&, bool)
    (this=0x5556093d5770, ppDim=std::__debug::vector of length 5, capacity 8 = {...}, ppLev=std::__debug::vector of length 5, capacity 8 = {...}, nPos=3, rInitState=..., bInitChild=true)
    at /home/julien/lo/libreoffice/sc/source/core/data/dptabres.cxx:2846
#2  0x00007fffddc4f444 in ScDPResultMember::InitFrom(std::__debug::vector<ScDPDimension*, std::allocator<ScDPDimension*> > const&, std::__debug::vector<ScDPLevel*, std::allocator<ScDPLevel*> > const&, unsigned long, ScDPInitState&, bool) (this=0x5556093d5660, ppDim=std::__debug::vector of length 5, capacity 8 = {...}, ppLev=std::__debug::vector of length 5, capacity 8 = {...}, nPos=3, rInitState=..., bInitChild=true)
    at /home/julien/lo/libreoffice/sc/source/core/data/dptabres.cxx:1077
#3  0x00007fffddc556f9 in ScDPResultDimension::InitFrom(std::__debug::vector<ScDPDimension*, std::allocator<ScDPDimension*> > const&, std::__debug::vector<ScDPLevel*, std::allocator<ScDPLevel*> > const&, unsigned long, ScDPInitState&, bool)
    (this=0x555605ae9c00, ppDim=std::__debug::vector of length 5, capacity 8 = {...}, ppLev=std::__debug::vector of length 5, capacity 8 = {...}, nPos=2, rInitState=..., bInitChild=true)
    at /home/julien/lo/libreoffice/sc/source/core/data/dptabres.cxx:2846
#4  0x00007fffddc4f444 in ScDPResultMember::InitFrom(std::__debug::vector<ScDPDimension*, std::allocator<ScDPDimension*> > const&, std::__debug::vector<ScDPLevel*, std::allocator<ScDPLevel*> > const&, unsigned long, ScDPInitState&, bool) (this=0x555605ae9af0, ppDim=std::__debug::vector of length 5, capacity 8 = {...}, ppLev=std::__debug::vector of length 5, capacity 8 = {...}, nPos=2, rInitState=..., bInitChild=true)
    at /home/julien/lo/libreoffice/sc/source/core/data/dptabres.cxx:1077
#5  0x00007fffddc556f9 in ScDPResultDimension::InitFrom(std::__debug::vector<ScDPDimension*, std::allocator<ScDPDimension*> > const&, std::__debug::vector<ScDPLevel*, std::allocator<ScDPLevel*> > const&, unsigned long, ScDPInitState&, bool)
    (this=0x5555ec0d1390, ppDim=std::__debug::vector of length 5, capacity 8 = {...}, ppLev=std::__debug::vector of length 5, capacity 8 = {...}, nPos=1, rInitState=..., bInitChild=true)
    at /home/julien/lo/libreoffice/sc/source/core/data/dptabres.cxx:2846
...
etc.

Comment 15 Oliver Brinzing 2019-09-23 17:01:15 UTC

still reproducible with:

Version: 6.4.0.0.alpha0+ (x64)
Build ID: 71ef762f21ada8c25aad2183065478171e985e8c
CPU threads: 4; OS: Windows 10.0; UI render: default; VCL: win; 
Locale: de-DE (de_DE); UI-Language: en-US
Calc: threaded

Comment 16 Xisco Faulí 2019-09-24 07:43:43 UTC

@Julien, is it possible to have a perf chart here ;-) ?

Comment 17 Julien Nabet 2019-09-24 08:29:26 UTC

(In reply to Xisco Faulí from comment #16)
> @Julien, is it possible to have a perf chart here ;-) ?
No pb.
I can also retrieve a Valgrind trace, perhaps as for Flamegraph with enable-symbols build instead of enable-dbgutil build.

Comment 18 Julien Nabet 2019-09-24 19:13:25 UTC

Created attachment 154453 [details]
perf flamegraph with simplified example

On pc Debian x86-64 with master sources updated today + enable-symbols, I retrieved a Flamegraph

Comment 19 Xisco Faulí 2019-09-25 11:19:02 UTC

Hi Noel,
I thought you might be interested in this issue...

Comment 20 Roman Kuznetsov 2020-12-25 16:38:41 UTC

the memory leak is still here (over 5Gb) in

Version: 7.2.0.0.alpha0+ (x64)
Build ID: ad9e04321df25824d2288a2ef1f4275f070f1cf7
CPU threads: 4; OS: Windows 6.1 Service Pack 1 Build 7601; UI render: Skia/Raster; VCL: win
Locale: ru-RU (ru_RU); UI: ru-RU
Calc: CL

but I didn't get any crashes, possibly because I use 64 bit OS and LO

Comment 21 Roman Kuznetsov 2022-06-19 21:47:44 UTC

(In reply to Roman Kuznetsov from comment #20)
> the memory leak is still here (over 5Gb) in
> 
> Version: 7.2.0.0.alpha0+ (x64)
> Build ID: ad9e04321df25824d2288a2ef1f4275f070f1cf7
> CPU threads: 4; OS: Windows 6.1 Service Pack 1 Build 7601; UI render:
> Skia/Raster; VCL: win
> Locale: ru-RU (ru_RU); UI: ru-RU
> Calc: CL
> 
> but I didn't get any crashes, possibly because I use 64 bit OS and LO

Memory leak still here in

Version: 7.5.0.0.alpha0+ / LibreOffice Community
Build ID: e4d23c27288b99c3ed3cfa332ff308b31c01f97d
CPU threads: 4; OS: Linux 5.14; UI render: default; VCL: gtk3
Locale: ru-RU (ru_RU.UTF-8); UI: en-US
Calc: threaded Jumbo

Comment 22 Pavel Kysilka 2024-03-30 10:10:59 UTC

I tested this bug today and the app is consuming about 260MB of RAM.

Version: 24.8.0.0.alpha0+ (X86_64) / LibreOffice Community
Build ID: 13250e1aa589453534d113442da1ac8a2cbb71b9
CPU threads: 128; OS: Linux 6.1; UI render: default; VCL: gtk3
Locale: cs-CZ (cs_CZ.UTF-8); UI: en-US
Calc: threaded

Comment 23 Roman Kuznetsov 2024-03-30 18:26:14 UTC

(In reply to Pavel Kysilka from comment #22)
> I tested this bug today and the app is consuming about 260MB of RAM.
> 
> Version: 24.8.0.0.alpha0+ (X86_64) / LibreOffice Community
> Build ID: 13250e1aa589453534d113442da1ac8a2cbb71b9
> CPU threads: 128; OS: Linux 6.1; UI render: default; VCL: gtk3
> Locale: cs-CZ (cs_CZ.UTF-8); UI: en-US
> Calc: threaded

Pavel did you try just update the pivot table in the file?

the memory leak is still here (over 11 Gb and my Windows 10 just died instead just kill LO process!)

Version: 24.8.0.0.alpha0+ (X86_64) / LibreOffice Community
Build ID: 2146e66d8df2b7b6a2dd868e886cae76aaf7f48b
CPU threads: 16; OS: Windows 10.0 Build 19045; UI render: Skia/Vulkan; VCL: win
Locale: ru-RU (ru_RU); UI: ru-RU
Calc: CL threaded

Comment 24 Pavel Kysilka 2024-03-30 20:29:37 UTC

Roman, you're right. After the refreshing pivot table, Calc consumes 61GB of RAM.

There is a similar error with pivot tables.

https://bugs.documentfoundation.org/show_bug.cgi?id=126710