Description: Usign coversion throught UNO API cause memory and handle leak. Simple program in Java attached Steps to Reproduce: 1. Run Libre Office as service: soffice.exe --invisible --headless --norestore --nologo --nodefault --nolockcheck --nocrashreport --nofirststartwizard --accept="socket,host=localhost,port=2002,tcpNoDelay=1;urp;StarOffice.Service" 2. Call conversion DOCX to PDF 3. GOTO point "2" and start observe memory leak.. Actual Results: Memory Leak, handle leak (on x64 and x86, on standard instalation and on portable version too) Expected Results: No memory nad handles leak Reproducible: Always User Profile Reset: Yes Additional Info: Attached ZIP with simple program to reproduce bug. Just: 1 (optional) complile LibreConverter.java to get LibreConverter.jar (already included) - 1_compile.bat 2. run LibreOffice - 2_runLibre.bat 3. run conversion in loop - 3_test.bat
Created attachment 155370 [details] Simple test program to reproduce error (extract to c:\test)
i can confirm a growing memory consumption with: Version: 6.2.8.2 (x64) Build-ID: f82ddfca21ebc1e222a662a32b25c0c9d20169ee CPU-Threads: 4; BS: Windows 10.0; UI-Render: Standard; VCL: win; Gebietsschema: de-DE (de_DE); UI-Sprache: de-DE Calc: after ~100 conversion: ~400 mb (growing) with: Version: 6.1.6.3 (x64) Build-ID: 5896ab1714085361c45cf540f76f60673dd96a72 CPU-Threads: 4; BS: Windows 10.0; UI-Render: Standard; Gebietsschema: de-DE (de_DE); Calc: after after ~100 conversion: ~190 mb (constant)
this may have been started with: https://gerrit.libreoffice.org/plugins/gitiles/core/+/b85ff98383942360901b8242cf77366782400426 gerrit.libreoffice.org / core / b85ff98383942360901b8242cf77366782400426 commit b85ff98383942360901b8242cf77366782400426 [log] author Jan-Marek Glogowski <glogow@fbihome.de> Mon Oct 22 18:34:06 2018 +0000 committer Jan-Marek Glogowski <glogow@fbihome.de> Wed Oct 24 11:52:43 2018 +0200 tree 7f895e035daaacdf0679dcb4236cdb188b70a034 parent 4b46826ec2219935ebcf86ed6e6db73910122e72 [diff] Change PDFWriterImpl into an OutputDevice It actually changes it into a VirtualDevice and should just be a refactoring. We get rid of the crude stuff in a follow up patch, While at it unfriend PDFWriterImpl from OutputDevice. Change-Id: Id43731ad076690292c30f9f3e05ff0dd58edc5e5 Reviewed-on: https://gerrit.libreoffice.org/62201 Tested-by: Jenkins Reviewed-by: Jan-Marek Glogowski <glogow@fbihome.de> /cygdrive/d/sources/bibisect/bibisect-win32-6.2 $ git bisect bad c4662cd95478c4b66911fa35c7cf96580493ee97 is the first bad commit commit c4662cd95478c4b66911fa35c7cf96580493ee97 Author: Norbert Thiebaud <nthiebaud@gmail.com> Date: Wed Oct 24 04:20:12 2018 -0700 source b85ff98383942360901b8242cf77366782400426 source b85ff98383942360901b8242cf77366782400426 :040000 040000 e016a3e9d86d1ef6bf240ea79dc0ea1f89123fa0 bdb481f623e6aa66b5b082d09da2305d23a81ecf M instdir /cygdrive/d/sources/bibisect/bibisect-win32-6.2 $ git bisect log # bad: [35a87a66cfc6dfb661f6fed49fb32c081dd26bc7] source d250c94d78ac7e79753aa30f869db919b01fc450 # good: [b0a56ec98b1368cb5e3e531e0b3f69565af91609] source 3a801799536e6870f2fb111b1cc00b9575a35a39 git bisect start 'master' 'oldest' # good: [7f7b05d44d7d7f13cc9c865963a72e555b516b3d] source 1b88de0a07180661c4d5d6706247d546d06bc983 git bisect good 7f7b05d44d7d7f13cc9c865963a72e555b516b3d # bad: [1c3155d561cb094926cd19aa514856dfb4e23c5e] source a626bdd56d7116efa57e65403ad51b56657148c3 git bisect bad 1c3155d561cb094926cd19aa514856dfb4e23c5e # good: [189df060b92de369d2cc8f0dba64ed5c53df7055] source 67e5201cc6caee52d8e37e98a0d535c085c19cb4 git bisect good 189df060b92de369d2cc8f0dba64ed5c53df7055 # good: [226d4eb9c28947a08cf78473d47bd7bf8c67bf5f] source 70198d4f7ffc7b3139cf34764b0e6bb6971489c6 git bisect good 226d4eb9c28947a08cf78473d47bd7bf8c67bf5f # bad: [48f7f098ccf9ae592eade9bc77c903fff21d247c] source 094e7b6a1028620c2b1503de8b51dc6a2482e290 git bisect bad 48f7f098ccf9ae592eade9bc77c903fff21d247c # good: [d3ee1ef07ed55a040171b2c161ce85d1f02666d4] source 9deabca91c8fd899fd162f4a16a1061793e8a10e git bisect good d3ee1ef07ed55a040171b2c161ce85d1f02666d4 # bad: [f8d019eb12c6f43aaea36b88ccec5d60ee64a4a8] source 199998361c3987f3bcdc26501b5f017d8965a22b git bisect bad f8d019eb12c6f43aaea36b88ccec5d60ee64a4a8 # good: [3876760123ad93d7f137ab9fa72feb23f6b2e5e2] source 2629aac31142449312f77c5843ea209cc810acb4 git bisect good 3876760123ad93d7f137ab9fa72feb23f6b2e5e2 # bad: [083afc8a8fc91cb9a5d4d6946b2836b473fcc61b] source 7579a33a2e19b55e9448fccdf3ce8dccfbce3478 git bisect bad 083afc8a8fc91cb9a5d4d6946b2836b473fcc61b # good: [a0cf33787d1e3eba1189efd92a513a5ff09e1bbb] source 4b46826ec2219935ebcf86ed6e6db73910122e72 git bisect good a0cf33787d1e3eba1189efd92a513a5ff09e1bbb # bad: [d3f2000ac8bd1d00240e2eaa5a5d3473ac6a214e] source 548e60a8c47e270aba79a5f4e5911cbb35462814 git bisect bad d3f2000ac8bd1d00240e2eaa5a5d3473ac6a214e # bad: [22348c5b1c5928b32630e9d49ec2e6cecd464e5d] source f8e06f7b77a6286d2c41bbc76f06a768c76cd87a git bisect bad 22348c5b1c5928b32630e9d49ec2e6cecd464e5d # bad: [c4662cd95478c4b66911fa35c7cf96580493ee97] source b85ff98383942360901b8242cf77366782400426 git bisect bad c4662cd95478c4b66911fa35c7cf96580493ee97 # first bad commit: [c4662cd95478c4b66911fa35c7cf96580493ee97] source b85ff98383942360901b8242cf77366782400426
Thanks for the bisecting and the nice test program! Had to adapt it to Linux, but that was rather minimal work. There is now https://gerrit.libreoffice.org/#/c/82562/, which fixes the rather massive memory leak of the PDF writer. But even with that patch you will definitely notice, that it just slows down the memory exhaustion, which seems to happen because of massive harfbuzz shape plan caching and how LO's own font and glyph cache keeps unused font instances alive. Also GlyphCaches byte-based accounting is really broken in any way, as it knows nothing about the harfbuzz cache. There are three problem areas: * The global GlyphCache in the application-wide GenericUnixSalData * The individual ImplFontCache per OutputDevice * The harfbuzz internal shape plan caching (hb_face_t => shape_plans) All these keeps the hb_font_t instances alive, even if they are not referenced anymore, which also keeps the hb_face_t instances alive. Running a grep on the harfbuzz code reveals, that the cache is rather unrestricted and just cleared in hb_face_destroy. Otherwise it seem just to grow (see also https://harfbuzz.github.io/shaping-plans-and-caching.html). That's why nearly all of my leaks - after the patch is applied - are rooted in the hb_shape_full call (which is the only one in LO) from GenericSalLayout::LayoutText. My current guess is, we should disable the harfbuzz cache and correctly cache and account its shape plans inside LO, if that is possible from the APIs point of view. I'll check how other software integrates harfbuzz and does caching.
Jan-Marek Glogowski committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/4dd87ccdab80eb094cede538e3d742148df3880a tdf#128434 really free the VclPtr<PDFWriterImpl> It will be available in 6.4.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Created attachment 155807 [details] Valgrind soffice log of 500 documents generated by the test program with LO shutdown This is a log of my modified test program run against the Valgrind soffice, which just creates 500 documents and the calls xDesktop.terminate() to shut down LO. As you can see in the log there are many lost blocks with a multiple of 500, which have the hb_shape_full in it, as I already wrote in the the previous comment. I guess most of the 500*n calls is some per-document lost block, which isn't freed on document close.
Jan-Marek Glogowski committed a patch related to this issue. It has been pushed to "libreoffice-6-4": https://git.libreoffice.org/core/commit/f4a2bc0da65695d9744e8b4be20a09c03fb196e0 tdf#128434 really free the VclPtr<PDFWriterImpl> It will be available in 6.4.0.1. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Could this fix be back-ported to the 6.3 branch ASAP please, so that it is ready for the 6.3.4 code-freeze next week? Many thanks.
Jan-Marek Glogowski committed a patch related to this issue. It has been pushed to "libreoffice-6-3": https://git.libreoffice.org/core/commit/27cfadd5e7a0897ee9fd046ab3e35edfd3aa2369 tdf#128434 really free the VclPtr<PDFWriterImpl> It will be available in 6.3.4. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Created attachment 155867 [details] Another valgrind soffice 500 documents log with additional patches The new log is from a run with three additional patches plugging memory leaks: https://gerrit.libreoffice.org/#/c/82968/ As far as I can tell, it just has one per-document leak left, which I could find with a search for "00 blocks": ==25806== 36,000 bytes in 500 blocks are still reachable in loss record 3,346 of 3,363 ==25806== at 0x483577F: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) ==25806== by 0x7FB7B5D: utl::OEventListenerAdapter::startComponentListening(com::sun::star::uno::Reference<com::sun::star::lang::XComponent> const&) (weak.hxx:85) ==25806== by 0x5B28B6D: basic::ImplRepository::impl_createManagerForModel(std::unique_ptr<BasicManager, std::default_delete<BasicManager> >&, com::sun::star::uno::Reference<com::sun::star::frame::XModel> const&) (basicmanagerrepository.cxx:480) ==25806== by 0x5B29065: basic::ImplRepository::getDocumentBasicManager(com::sun::star::uno::Reference<com::sun::star::frame::XModel> const&) (basicmanagerrepository.cxx:233) ==25806== by 0x60B0E4A: SfxObjectShell::InitBasicManager_Impl() (objxtor.cxx:818) ==25806== by 0x60B0F8C: (anonymous namespace)::lcl_getBasicManagerForDocument(SfxObjectShell const&) (objxtor.cxx:628) ==25806== by 0x60B1279: SfxObjectShell::GetBasicManager() const (objxtor.cxx:662) ==25806== by 0x1B1A120D: SwDoc::GetVbaEventProcessor() (vbaaccesshelper.hxx:47) ==25806== by 0x1B8C4585: SwDocShell::Notify(SfxBroadcaster&, SfxHint const&) (docsh2.cxx:257) ==25806== by 0x64A84B4: SfxBroadcaster::Broadcast(SfxHint const&) (SfxBroadcaster.cxx:49) ==25806== by 0x5F347DD: SfxShell::PutItem(SfxPoolItem const&) (shell.cxx:195) ==25806== by 0x1B8CEC93: InitDrawModelAndDocShell(SwDocShell*, SwDrawModel*) (docshdrw.cxx:69) ==25806== by 0x1B1F0B69: SwDoc::SetDocShell(SwDocShell*) (docnew.cxx:620) ==25806== by 0x1B8CFB9F: SwDocShell::AddLink() (docshini.cxx:417) ==25806== by 0x1B8D1DF0: SwDocShell::InitNew(com::sun::star::uno::Reference<com::sun::star::embed::XStorage> const&) (docshini.cxx:114) ==25806== by 0x60ADA21: SfxObjectShell::DoLoad(SfxMedium*) (objstor.cxx:711) ==25806== by 0x60DC763: SfxBaseModel::load(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) (sfxbasemodel.cxx:1856) ==25806== by 0x618B76A: (anonymous namespace)::SfxFrameLoader_Impl::load(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&, com::sun::star::uno::Reference<com::sun::star::frame::XFrame> const&) (frmload.cxx:691) ==25806== by 0x15FF7D49: framework::LoadEnv::impl_loadContent() (loadenv.cxx:1157) ==25806== by 0x15FF8FDF: framework::LoadEnv::startLoading() (loadenv.cxx:390) ==25806== by 0x15FF9497: framework::LoadEnv::loadComponentFromURL(com::sun::star::uno::Reference<com::sun::star::frame::XComponentLoader> const&, com::sun::star::uno::Reference<com::sun::star::uno::XComponentContext> const&, rtl::OUString const&, rtl::OUString const&, int, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) (loadenv.cxx:171) ==25806== by 0x16017DCA: framework::Desktop::loadComponentFromURL(rtl::OUString const&, rtl::OUString const&, int, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) (desktop.cxx:621) ==25806== by 0x49CE457: gcc3::callVirtualMethod(void*, unsigned int, void*, _typelib_TypeDescriptionReference*, bool, unsigned long*, unsigned int, unsigned long*, double*) (callvirtualmethod.cxx:133) ==25806== by 0x49CD374: cpp_call(bridges::cpp_uno::shared::UnoInterfaceProxy*, bridges::cpp_uno::shared::VtableSlot, _typelib_TypeDescriptionReference*, int, _typelib_MethodParameter*, void*, void**, _uno_Any**) (uno2cpp.cxx:233) ==25806== by 0x49CDC65: unoInterfaceProxyDispatch (uno2cpp.cxx:413) ==25806== by 0x1191B4C0: binaryurp::IncomingRequest::execute_throw(binaryurp::BinaryAny*, std::vector<binaryurp::BinaryAny, std::allocator<binaryurp::BinaryAny> >*) const (incomingrequest.cxx:236) ==25806== by 0x1191BB86: binaryurp::IncomingRequest::execute() const (incomingrequest.cxx:79) ==25806== by 0x11920832: request (reader.cxx:85) ==25806== by 0x566B1F6: cppu_threadpool::JobQueue::enter(long, bool) (jobqueue.cxx:107) ==25806== by 0x566B87F: cppu_threadpool::ORequestThread::run() (thread.cxx:165) ==25806== by 0x566C759: threadFunc (thread.hxx:185) ==25806== by 0x4888F40: osl_thread_start_Impl(void*) (thread.cxx:235) ==25806== by 0x4EE8FA2: start_thread (pthread_create.c:486) ==25806== by 0x4AE74CE: clone (clone.S:95) Some listener for the StarBasic handler. I'll probably check that next week, but I never worked in that code. Hopefully it's something obvious. But we seem to be down to 72 bytes per document, which is much better then I initially hoped in a short term. And it has to survive all the unit tests. Happy weekend, everyone.
Any chance these three additional patches for memory leaks could be back-ported to the 6.3 branch as well, please?
Jan-Marek Glogowski committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/e6aac0b637d583d3cfb893276f813ff5aa1ade17 tdf#128434 try garbage collect ImplFontCache fonts It will be available in 6.5.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Jan-Marek Glogowski committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/f8e1f8652255cadd80a991aa3e059ee631b333b8 tdf#128434 correctly release fonts in destructors It will be available in 6.5.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Jan-Marek Glogowski committed a patch related to this issue. It has been pushed to "libreoffice-6-4": https://git.libreoffice.org/core/commit/325005697155853891ce4f23e7349931e748d7e7 tdf#128434 try garbage collect ImplFontCache fonts It will be available in 6.4.0.1. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Jan-Marek Glogowski committed a patch related to this issue. It has been pushed to "libreoffice-6-4": https://git.libreoffice.org/core/commit/a00bdc999344db34d5926dc77ed5ca895295b0ee tdf#128434 correctly release fonts in destructors It will be available in 6.4.0.1. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Jan-Marek Glogowski committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/48b23bbfa0271ed327f668933b92d2ae9b99e806 tdf#128434 free the BasicManager event listener It will be available in 6.5.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
If the 6.3 code-freeze hasn't happened yet, could these patches be back-ported, please?
Jan-Marek Glogowski committed a patch related to this issue. It has been pushed to "libreoffice-6-4": https://git.libreoffice.org/core/commit/d8cde1cf69bb170da74018e629e1b65830924e0b tdf#128434 free the BasicManager event listener It will be available in 6.4.0.1. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Jan-Marek Glogowski committed a patch related to this issue. It has been pushed to "libreoffice-6-3": https://git.libreoffice.org/core/commit/083d98d73379f864a0d3cb9695e7c7ad0febce1c tdf#128434 try garbage collect ImplFontCache fonts It will be available in 6.3.4. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Jan-Marek Glogowski committed a patch related to this issue. It has been pushed to "libreoffice-6-3": https://git.libreoffice.org/core/commit/41f2ec60825bc71686f506178733a885373d6ddb tdf#128434 correctly release fonts in destructors It will be available in 6.3.4. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Jan-Marek Glogowski committed a patch related to this issue. It has been pushed to "libreoffice-6-3": https://git.libreoffice.org/core/commit/fb65014f61957bb0e7cf9008b38322ef14e707d6 tdf#128434 free the BasicManager event listener It will be available in 6.3.4. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.