When i open an ods file (22MB) , the ram used is around 37% off 8 G (total). That file has about 100 columns and 4000 rows with data and formulas. If i select all the columns and rows, then ask for sorting two columns, the ram used climb to 66%. Then i save that job, but the ram used is still the same (expecting it to be lower as the job is done). Continuing working with that file for a while, the ram is not lowered. Then if i close that file, the ram fall to 65 % (again this is not expected). Closing calc, then the ram is finally lowered to initial 37%. Please tell me if i need to provide debugging output, but tell me how to do it.
What kind of file MS?, please test if clearing all direct cell format out of the range with data helps in someway.
It is my own libreoffice's calc filed on ubuntu. Cells are formatted either with: colored columns, and/or 2 data digits, nothing else. Indeed sorting range data cells (only) require less memory, but still increase the initial ram amount, without freeing it after saving (which is expected).
[Automated Action] NeedInfo-To-Unconfirmed
Can you attach the file after clear or modify private information.
The file has nothing special builtin. Simply data and some formulas using if(and(...)) I have also tested some other files with calc, and also writer's files (~50 pages)where i have ran some 'find & replace' feature. and i have got the same issue with each used files: ram is not recovered after the asked job is done. That means probably cant be blamed, but libreoffice-core, or event outside (mutter ? or else)
Have you test with a clean profile? Menu/Help/Restart in Safe Mode
Safe mode does not get better result. I also have reproduced that issue with different pcs and other calc/writer files. So the question is: which of libreoffice or system (gnome-shell on xorg session) or their dependencies is/are in charge of ram dynamic usage inside calc/writer ?
Thank you for reporting the bug. Please attach a sample document, as this makes it easier for us to verify the bug. (Please note that the attachment will be public, remove any sensitive information before attaching it. See https://wiki.documentfoundation.org/QA/FAQ#How_can_I_eliminate_confidential_data_from_a_sample_document.3F for help on how to do so.) I have set the bug's status to 'NEEDINFO'. Please change it back to 'UNCONFIRMED' once the requested document is provided.
Dear polo, This bug has been in NEEDINFO status with no change for at least 6 months. Please provide the requested information as soon as possible and mark the bug as UNCONFIRMED. Due to regular bug tracker maintenance, if the bug is still in NEEDINFO status with no change in 30 days the QA team will close the bug as INSUFFICIENTDATA due to lack of needed information. For more information about our NEEDINFO policy please read the wiki located here: https://wiki.documentfoundation.org/QA/Bugzilla/Fields/Status/NEEDINFO If you have already provided the requested information, please mark the bug as UNCONFIRMED so that the QA team knows that the bug is ready to be confirmed. Thank you for helping us make LibreOffice even better for everyone! Warm Regards, QA Team MassPing-NeedInfo-Ping
As explained/exposed earlier, there is no special relationship about the file (osd/doc/txt/pdf) used: it is reproducible everytime on different configs/versions with systems running gnome/systemd/mutter on xorg session. The simplest fact is shown via System-monitor applet, when: - soffice is loaded: take note of % ram used - then open a task (calc/writer or else); do some work and save; note ram % - close that task; note ram % again : you see that ram is not freed - then close soffice, and finally ram is freed. As asked via comment 8, it should be first needed to determine the source of that problem. If you have suggestions to narrow down that issue, please let me know. But i'm quite sure everyone can detect that problem on similar config (at least).
Feedback: Tested LO 6.0.7 : no ram issue seen after a spike due to huge sorting; a couple seconds after that task is done, the used ram level come back to the previous one. Tested newer LO 6.2.4 and 6.34 against the same huge ods file, and made the same sorting job, as described by Polo, the used ram level climb as expected, but then the level dont set back as it might be. Works on Ubuntu Bionic with 6.0.7, but dont on Disco and Eoan with newer versions.
(In reply to duceil from comment #12) > Feedback: > > Tested LO 6.0.7 : no ram issue seen after a spike due to huge sorting; a > couple seconds after that task is done, the used ram level come back to the > previous one. > > Tested newer LO 6.2.4 and 6.34 against the same huge ods file, and made the > same sorting job, as described by Polo, the used ram level climb as > expected, but then the level dont set back as it might be. > > Works on Ubuntu Bionic with 6.0.7, but dont on Disco and Eoan with newer > versions. @duceil Please attach a sample document, as this makes it easier for us to verify the bug. (Please note that the attachment will be public, remove any sensitive information before attaching it. See https://wiki.documentfoundation.org/QA/FAQ#How_can_I_eliminate_confidential_data_from_a_sample_document.3F for help on how to do so.)
Created attachment 160639 [details] mem usage while loading and closing files, For the Bug hunting Session 7.0.0.0.a1+: tried mem usage and release, see pic, files with random values, each 1.000.000 cells (A1:CV10000) filled with =CHAR(INT(RAND()*26+1+96))&CHAR(INT(RAND()*26+1+96))&CHAR(INT(RAND()*26+1+96))&CHAR(INT(RAND()*26+1+96))&CHAR(INT(RAND()*26+1+96)) loaded three such files - marks 1, 2, 3, constructed a fourth one - mark 4, subsequently sorted them by col A and B ascending - mem spikes at 5, 6, 7, 8, (useless on random data, i know! that) opened one new empty file - mark 9, released the first four, without saving - marks A, B, C, D, left is one empty! file, not even any undo actions valid for it, blocking 2 GB of memory, memeory freed with close of program - mark E, did i / the OP miss some memory retention 'timeouts'? or is it really a mem leak?
(In reply to b. from comment #14) Please attach the sample file you used. Else everybody has to rebuild it themselfs.. And thanks for testing Calc.. not quite my expertise
impressive, re-checked, even with a writer doc in the background to keep soffice open, no spike, special win 8.1 problem? (i'm on 7)
sorry, prev. comment wrong thread :-(
it's faster to copy and paste that formula: =CHAR(INT(RAND()*26+1+96))&CHAR(INT(RAND()*26+1+96))&CHAR(INT(RAND()*26+1+96))&CHAR(INT(RAND()*26+1+96))&CHAR(INT(RAND()*26+1+96)) in that area: A1:CV10000 than to manage a 10 mb download ...
Created attachment 160929 [details] Example file
Created attachment 160938 [details] Example file
1. open Calc 2. Tools -> Options -> Advanced -> Open Expert Configuration -> Search for Undo Set undo steps 100 back to 0 3. Close LibreOffice and restart 4. Open the attached file 5. Copy Column D 6. Start pasting column by column from E to S 7. Memory usage increases to for me 569 MB 8. Delete all the content -> Still 569 MB 9. Close the document -> 164 MB Version: 7.0.0.0.alpha1+ (x64) Build ID: f9790da286f2d2fa47f1748f8cfa6172c6622ca3 CPU threads: 4; OS: Windows 6.3 Build 9600; UI render: Skia/Raster; VCL: win; Locale: de-CH (nl_NL); UI: en-US Calc: CL I have no clue why the memory is allocated until close of the document.. it's not undo information Same can be observed in 4.4.7.2
@Julien Do you have any insight in memory management of Calc? Not sure if it's really leaking.. looks more like a delayed release. Memory is freed eventually (after closing).. But the current behavior can add up pretty quick.. And undo isn't even enabled (assuming the setting works as expected)
(In reply to Telesto from comment #22) > @Julien > Do you have any insight in memory management of Calc? Not sure if it's > really leaking.. looks more like a delayed release. Memory is freed > eventually (after closing).. > > But the current behavior can add up pretty quick.. And undo isn't even > enabled (assuming the setting works as expected) Sorry, I don't have any insight and don't think Flamegraph may help. Except Eike, I don't know who may help. There was Kohei some years ago but he works only on mdds if I well understood. About leaks, there was a tool on Mac (I don't remember the name and since I don't have a Mac anymore, can't tell).
@Mike Not to nag.. but more as a question.. should this be open as bug? Comment 0/ Comment 14
(In reply to Telesto from comment #24) Hi! Although this might need further analysis, I agree that not releasing memory *at all* after an operation (as 7-8 from c#14 might hint) must be treated as a bug. Yet, it needs investigating: is that really not released e.g. on timer? is sorting repeatedly on the same data (e.g. asc->desc->asc->...) produce higher levels after the spike each time? etc. But if it's released subsequently, without unloading documents or exiting program, then IMO it needs much more substantial ground to be called a bug. That would mean proper memory handling, and only if that results in actual problems, the strategy might need to be revised.
Created attachment 161088 [details] Screenshot @Mike, Small question: how long is a timer based release 'allowed' to take.. 4 minutes after closing the document is fine?
1. Open attachment 160938 [details] 2. Copy Column D 3. Paste one by one until column S 4. CTRL+Z everything 5. CTRL+Y everything 6. Copy an empty cell (clearing the clipboard) 7. Open a new calc sheet 8. Close the one attached file without saving 9. Do nothing.. 1 GB in use.. won't be released.. The release is - surely - triggered by copying something to the clipboard. Making edits to the new Calc file (typing) has no impact... as far I noticed.. Opening a new file doesn't do anything either. Found in 7.0 and in 4.4.7.2 and in Version: 4.3.7.2 Build ID: 8a35821d8636a03b8bf4e15b48f59794652c68ba but not in Versie: 4.1.0.4 Build ID: 89ea49ddacd9aa532507cbf852f2bb22b1ace28 Adding bibisectrequest assuming this is reproducible on Linux
Bisected to: author Kohei Yoshida <kohei.yoshida@collabora.com> 2013-10-10 20:24:21 -0400 committer Kohei Yoshida <kohei.yoshida@collabora.com> 2013-10-11 12:14:27 -0400 commit 7333881bb7b04f7e4e2a28638024ae82a9c14e81 (patch) tree 97548f94ab918d502b45a5dda40ece5ad4117617 parent 6255be7ca294d350143290c343673f264f42220c (diff) Formula tokens, formula cells and formula interpreters to use shared strings. https://cgit.freedesktop.org/libreoffice/core/commit/?id=7333881bb7b04f7e4e2a28638024ae82a9c14e81 Not actually the release oft memory.. but in general the memory usage Before this commit +/-200 after 1 GB. Even the calculation is faster before..
@Noel I'm horrible opportunistic here.. And not even sure if Calc is you're thing.. but have done some nice changes improving memory usage in general: So maybe a clue why this commit is causing a massive increase in memory usage https://cgit.freedesktop.org/libreoffice/core/commit/?id=7333881bb7b04f7e4e2a28638024ae82a9c14e81 Formula tokens, formula cells and formula interpreters to use shared strings.
@Telesto that particular commit is intended to _lower_ memory usage in most situations, by creating a shared pool of string objects to use, instead of every single cell and formula using it's own string object. However, we never ever remove strings from that pool, so in this situation it leads to memory that this not freed until close. We could flush that pool periodically, but it is not obvious when to flush that pool, in a way which will not cause performance regressions for some people.
(In reply to Noel Grandin from comment #30) > @Telesto that particular commit is intended to _lower_ memory usage in most > situations, by creating a shared pool of string objects to use, instead of > every single cell and formula using it's own string object. > > However, we never ever remove strings from that pool, so in this situation > it leads to memory that this not freed until close. > > We could flush that pool periodically, but it is not obvious when to flush > that pool, in a way which will not cause performance regressions for some > people. I'm not knowing anything to give any advice here.. There are two flaws, IMHO * The pool isn't flushed periodically * The pool isn't flushed properly after closing the document either. It needs some bizarre trigger to do so (like copying something to the clipboard) Also no clue how a pool functions. Lacking the knowledge about smart pointers, unique pointers etc.. Nor when unused strings can be useful.. Or how to track unused and used strings.. just emptying a pool on timer doesn't sound smart.. I assume there are more strings stored than actually present in the visual presentation in Calc itself. Is there no intelligent compiler feature to track unused strings.. as a sort of garbage collector. To flush things out after being unused for say 30 seconds.. Dumping everything into the pool until close, isn't doesn't sound like a proper solution either.. However not sure if the pool gets filled because of the specifics of the bug doc in question.. or this being a general behavior
@Noel: (In reply to Noel Grandin from comment #30) > ... that particular commit is intended to _lower_ memory usage in most > situations, by creating a shared pool of string objects to use, instead of > every single cell and formula using it's own string object. > > However, we never ever remove strings from that pool, so in this situation > it leads to memory that this not freed until close. hi, thanks, that explains a lot ... is the referencing to this list 'uni-' or 'bidirectional'? when i think about it ... it's a clear advantage as long as it doesn't become a killer, and will become a killer once it's grown too big ... thus dangerous for big sheets / projects is it possible / a good idea to implement it similar to linux hardlinks in filesystems, track referencing items and require the referencers to delete the reference as soon as it is no longer needed, or to check from the list whether the referencing objects are still 'alive'? and release the referenced string once all referencers are dead ... ?
Only sharing my own feeling here. Apps like Calc/Writer/... are ran under Soffice ombrella That report suggest that Soffice is not dealing with apps's ram usage and does not care about jobs saved/closed. Indeed decisions have to be taken about the 'undo' possibilities (how many ? ) Maybe a custom user setting can definitely register the job done: 1) press icon for example when the job is saved, then freeing job's ram. 2) ram freed if that job is closed (Soffice does not need to store such data after the job has ended) 3) Soffice definitely purge the ram used by the being closed app (calc/writer/...)
Not related to "shared strings" - so bit off topic - and probably inherent-ed - but there something odd going on with the undo/redo 'information pool': bug 133553 Memory usage has a steep curve even with the undo/redo steps set to 1
Luboš Luňák committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/861fbd998f2b526c2aea073c9471613bf728fa75 purge shared string pool if ScDocument is closed (tdf#125428) It will be available in 7.1.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Luboš Luňák committed a patch related to this issue. It has been pushed to "libreoffice-7-0": https://git.libreoffice.org/core/commit/b0556da3a0966c9c68c6a57909524058d0bb5e07 purge shared string pool if ScDocument is closed (tdf#125428) It will be available in 7.0.0.1. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Luboš Luňák committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/5b2a153779b68e8454a66973579512fe17e376d5 do not call purge() on string pool too often (tdf#125428) It will be available in 7.4.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.