Created attachment 152415 [details] slow_replace.ods Description: In calc, the replacement is slow when converting string to int. Steps to Reproduce: Create an empty calc. I suppose that the decimal separator is "," Fill about 200 000 cells with "A1.2". Replace "A" by "" : time : 5s. Replace "." by "," : time : 4 min with LibreOffice 6.2.4.2. time : 8 min 30s with LibreOffice 6.4.0.0+Master. Actual Results: Slowdown Expected Results: No slowdown Reproducible: Always User Profile Reset: Fresh installation Additional Info: Version: 6.4.0.0.alpha0+ (x64) Build ID: 9a2fbfa3cc1da8bd9388d5b4c780e86f0dccc791 CPU threads: 12; OS: Windows 10.0; UI render: GL; VCL: win; TinderBox: Win-x86_64@42, Branch:master, Time: 2019-06-25_23:12:21 Locale: fr-FR (fr_FR); UI-Language: en-US Calc: CL Version: 6.2.4.2 Build ID: 2412653d852ce75f65fbfa83fb7e7b669a126d64 Threads CPU : 12; OS : Windows 10.0; UI Render : par défaut; VCL: win; Locale : fr-FR (fr_FR); Langue IHM : fr-FR Calc: threaded
After 10 min 1 core at 100% and increasing memory more then 1.5G i killed the test. confirm the problem with Version: 6.4.0.0.alpha0+ Build ID: 54a3044ebd7f946235ba1d0a8af11ad94872afb7 CPU threads: 8; OS: Linux 4.19; UI render: default; VCL: gtk3; Locale: nl-BE (en_US.UTF-8); UI-Language: en-US Calc: threaded
I made some tests. The problem occurs with LibreOffice 5.0.0.1 but not with LibreOffice 4.0.0.1 so it's an old regression. Can someone confirms? Should it be bisected? Version 4.0.0.1 (Build ID: 527dba6f6e0cfbbc71bd6e7b88a52699bb48799) Version: 5.0.0.1 (x64) Build ID: 9a0b23dd0ab9652e0965484934309f2d49a7758e Locale : fr-FR (fr_FR)
I made a bibisect. But I think I did that for a mess. 8e7bade4e7314a340c77edd9042e230f61f0323d is the first bad commit commit 8e7bade4e7314a340c77edd9042e230f61f0323d Author: Matthew Francis <mjay.francis@gmail.com> Date: Sat Sep 5 18:33:36 2015 +0800 source-hash-4c99a427ee4adaeddb2682c192384bad21d9d09b Bibisect: This commit covers the following source commit(s) which failed to build c7bdee8dbd1cf260a8513a0d31b36f90daa70f1c 77ec47356025de4e46f48f94629f896349b0a8e5 e9c5eb60d53204261c7937108bd53e86e46fc2f3 75dec25730c88bdb8eb5e2a3f92689460fa89d29 c008dc483f8c6840803983e7e351cec6fdd32070 46419cd7a2d453c6f252c28dfb9dbfb08605e1c4 dcf04c58456d8285bdcaa2921b6c08d48e09dab3 dabeb3d643c0ebd2cb7fdce333d0e6248b1488ad 8b252f30267d043522ff2cbf2bf42275bb7a6ec6 76ca1529edb9afdb1838ab9a9e01fa231148c1d8 341a4c87e977a2fd3948de6eaa647be4b32e6ebc 3b3b0c04385851f120dc26d26e40f0d1c6344274 cb4a47887df282196c9aa529269d5115306813c1 1fe76b401760e0b31095d0a4d7e877aeee1624dc 6ea53929245eb496d6954ab266636978653f1784 65be1e27254ff1292b3593af42fbfb0235c26fcd b6a6a26698cb56a194c7c99b0ad267e60e3a05d5 3b0c069c9a157c4cd9ec5636c776115af6d9664f 458df361e0761651b6d17f4fb730f996243bc9f3 2a5ea9ee0ba2b38d1810828a9fdf884d9b0fd198 e3b91687590f08438b5a5d4eec72e634b11a8589 57538e5f5002ecc2df95087244cb0431867c443e df90b9eaa16a52d076658ffc2fd1f65f12d1a622 2a1c5aba7640416c78501116dd42d12e74fe4734 66d3f24334e69e1655e83520950c59a0bda095a3 8a39b8ce354bd42325ff61c07cfdc7a150d2925a 91f7e9e02e72b46c881656a6b493fac276a6822b 33a417fc82ce3c5f1b3d435d912eb4b5afc52054 7a522da4f3946006fd325d498845225b8a8836f2 5f188d659e8601e577f3a837c9dd3459761371ac 92a78a052efcb3122932894fb446c62733daad42 9186ae043b7e8afa5730b9d70818360cdfef2201 359f33c5d0d39c4fcc57cba199a0d5b9a66c8fb9 b1391232450af9e81079a2afe2173c422c8e9e3b e639e3068c30bb614c394466f41fa450ee8b2dbb cf02151987b19b12ca0be463732765bd35f54028 2c92a92e2fa2503f0381b89f316f982eda580b6e 21a1bce4301d3d8de9702373c830d2e115223991 878f46727d8bcf1f75d056d9270ef3e2fe0b9d88 bb7d5ce2a8bd1dca51eb627aa2df811541053969 4347e3b15f10784b482544bd6324d3fcd4f0146c ec0080c40cfdb26896537f47a4c2e0439f9afdb1 commit 4c99a427ee4adaeddb2682c192384bad21d9d09b Author: Kohei Yoshida <kohei.yoshida@gmail.com> AuthorDate: Mon Jun 24 16:19:02 2013 -0400 Commit: Kohei Yoshida <kohei.yoshida@gmail.com> CommitDate: Mon Jun 24 16:51:45 2013 -0400 Fix incorrect merge. Change-Id: I1337413e1ee49b7d90671ac517dbb2bd918dbebf opt/program/classes/ScriptFramework.jar | Bin 79686 -> 79686 bytes opt/program/classes/ScriptProviderForBeanShell.jar | Bin 19541 -> 19541 bytes opt/program/classes/ScriptProviderForJava.jar | Bin 10409 -> 10409 bytes .../classes/ScriptProviderForJavaScript.jar | Bin 14973 -> 14973 bytes opt/program/classes/XMergeBridge.jar | Bin 7009 -> 7009 bytes opt/program/classes/bsh.jar | Bin 269036 -> 269036 bytes opt/program/classes/commons-logging-1.1.1.jar | Bin 50499 -> 50499 bytes opt/program/classes/commonwizards.jar | Bin 304542 -> 304542 bytes opt/program/classes/flow-engine-0.9.4.jar | Bin 280485 -> 280485 bytes opt/program/classes/flute-1.1.6.jar | Bin 82841 -> 82841 bytes opt/program/classes/form.jar | Bin 47916 -> 47916 bytes opt/program/classes/hsqldb.jar | Bin 701099 -> 701099 bytes opt/program/classes/js.jar | Bin 621798 -> 621798 bytes opt/program/classes/libbase-1.1.6.jar | Bin 121321 -> 121321 bytes opt/program/classes/libfonts-1.1.6.jar | Bin 207513 -> 207513 bytes opt/program/classes/libformula-1.1.7.jar | Bin 393458 -> 393458 bytes opt/program/classes/liblayout-0.2.10.jar | Bin 801656 -> 801656 bytes opt/program/classes/libloader-1.1.6.jar | Bin 124235 -> 124235 bytes opt/program/classes/librepository-1.1.6.jar | Bin 64066 -> 64066 bytes opt/program/classes/libserializer-1.1.6.jar | Bin 21763 -> 21763 bytes opt/program/classes/libxml-1.1.7.jar | Bin 76344 -> 76344 bytes opt/program/classes/officebean.jar | Bin 45577 -> 45577 bytes opt/program/classes/query.jar | Bin 16638 -> 16638 bytes opt/program/classes/report.jar | Bin 58485 -> 58485 bytes opt/program/classes/reportbuilder.jar | Bin 295704 -> 295704 bytes opt/program/classes/reportbuilderwizard.jar | Bin 34515 -> 34515 bytes opt/program/classes/sac.jar | Bin 12967 -> 12967 bytes opt/program/classes/sdbc_hsqldb.jar | Bin 10165 -> 10165 bytes opt/program/classes/smoketest.jar | Bin 3666 -> 3666 bytes opt/program/classes/table.jar | Bin 31487 -> 31487 bytes opt/program/classes/unoil.jar | Bin 1687214 -> 1687214 bytes opt/program/classes/xmerge.jar | Bin 151661 -> 151661 bytes opt/program/libeditenglo.so | Bin 3022380 -> 3022435 bytes opt/program/libfreebl3.so | Bin 902154 -> 902154 bytes opt/program/libneon.so | Bin 1968093 -> 1968093 bytes opt/program/libnspr4.so | Bin 734916 -> 734916 bytes opt/program/libnss3.so | Bin 4315872 -> 4315872 bytes opt/program/libnssckbi.so | Bin 989850 -> 989850 bytes opt/program/libnssdbm3.so | Bin 491907 -> 491907 bytes opt/program/libnssutil3.so | Bin 364355 -> 364355 bytes opt/program/libooxlo.so | Bin 9242212 -> 9242212 bytes opt/program/libplc4.so | Bin 51773 -> 51773 bytes opt/program/libplds4.so | Bin 33950 -> 33950 bytes opt/program/libscfiltlo.so | Bin 6420958 -> 6420908 bytes opt/program/libsclo.so | Bin 15456254 -> 15751416 bytes opt/program/libsdfiltlo.so | Bin 761109 -> 761109 bytes opt/program/libsdlo.so | Bin 9688171 -> 9688171 bytes opt/program/libsduilo.so | Bin 794376 -> 794376 bytes opt/program/libsmime3.so | Bin 747159 -> 747159 bytes opt/program/libsmlo.so | Bin 1522669 -> 1522669 bytes opt/program/libsoftokn3.so | Bin 689388 -> 689388 bytes opt/program/libssl3.so | Bin 895667 -> 895667 bytes opt/program/libwriterfilterlo.so | Bin 5558676 -> 5558676 bytes opt/program/setuprc | 2 +- opt/program/versionrc | 2 +- opt/sdk/lib/libsalcpprt.a | Bin 5216 -> 5216 bytes opt/share/Scripts/java/HelloWorld/HelloWorld.jar | Bin 1541 -> 1541 bytes opt/share/Scripts/java/Highlight/Highlight.jar | Bin 4220 -> 4220 bytes opt/share/Scripts/java/MemoryUsage/MemoryUsage.jar | Bin 3194 -> 3194 bytes opt/share/autocorr/acor_af-ZA.dat | Bin 4946 -> 4946 bytes opt/share/autocorr/acor_bg-BG.dat | Bin 1249 -> 1249 bytes opt/share/autocorr/acor_ca.dat | Bin 6383 -> 6383 bytes opt/share/autocorr/acor_cs-CZ.dat | Bin 5940 -> 5940 bytes opt/share/autocorr/acor_da-DK.dat | Bin 7259 -> 7259 bytes opt/share/autocorr/acor_de-DE.dat | Bin 4003 -> 4003 bytes opt/share/autocorr/acor_en-AU.dat | Bin 8212 -> 8212 bytes opt/share/autocorr/acor_en-GB.dat | Bin 8185 -> 8185 bytes opt/share/autocorr/acor_en-US.dat | Bin 8130 -> 8130 bytes opt/share/autocorr/acor_en-ZA.dat | Bin 8035 -> 8035 bytes opt/share/autocorr/acor_es-ES.dat | Bin 4526 -> 4526 bytes opt/share/autocorr/acor_fa-IR.dat | Bin 58323 -> 58323 bytes opt/share/autocorr/acor_fi-FI.dat | Bin 1514 -> 1514 bytes opt/share/autocorr/acor_fr-FR.dat | Bin 4377 -> 4377 bytes opt/share/autocorr/acor_ga-IE.dat | Bin 14029 -> 14029 bytes opt/share/autocorr/acor_hr-HR.dat | Bin 5908 -> 5908 bytes opt/share/autocorr/acor_hu-HU.dat | Bin 10115 -> 10115 bytes opt/share/autocorr/acor_is-IS.dat | Bin 1073 -> 1073 bytes opt/share/autocorr/acor_it-IT.dat | Bin 2093 -> 2093 bytes opt/share/autocorr/acor_ja-JP.dat | Bin 7422 -> 7422 bytes opt/share/autocorr/acor_ko-KR.dat | Bin 7422 -> 7422 bytes opt/share/autocorr/acor_lb-LU.dat | Bin 5131 -> 5131 bytes opt/share/autocorr/acor_lt-LT.dat | Bin 3176 -> 3176 bytes opt/share/autocorr/acor_mn-MN.dat | Bin 4369 -> 4369 bytes opt/share/autocorr/acor_nl-BE.dat | Bin 2802 -> 2802 bytes opt/share/autocorr/acor_nl-NL.dat | Bin 2791 -> 2791 bytes opt/share/autocorr/acor_pl-PL.dat | Bin 7925 -> 7925 bytes opt/share/autocorr/acor_pt-BR.dat | Bin 21780 -> 21780 bytes opt/share/autocorr/acor_pt-PT.dat | Bin 2360 -> 2360 bytes opt/share/autocorr/acor_ru-RU.dat | Bin 4034 -> 4034 bytes opt/share/autocorr/acor_sh-ME.dat | Bin 1371 -> 1371 bytes opt/share/autocorr/acor_sh-RS.dat | Bin 1371 -> 1371 bytes opt/share/autocorr/acor_sh-YU.dat | Bin 1371 -> 1371 bytes opt/share/autocorr/acor_sk-SK.dat | Bin 4985 -> 4985 bytes opt/share/autocorr/acor_sl-SI.dat | Bin 4957 -> 4957 bytes opt/share/autocorr/acor_sr-ME.dat | Bin 1478 -> 1478 bytes opt/share/autocorr/acor_sr-RS.dat | Bin 1478 -> 1478 bytes opt/share/autocorr/acor_sr-YU.dat | Bin 1478 -> 1478 bytes opt/share/autocorr/acor_sv-SE.dat | Bin 2852 -> 2852 bytes opt/share/autocorr/acor_tr-TR.dat | Bin 2522 -> 2522 bytes opt/share/autocorr/acor_vi-VN.dat | Bin 9532 -> 9532 bytes opt/share/autocorr/acor_zh-CN.dat | Bin 7396 -> 7396 bytes opt/share/autocorr/acor_zh-TW.dat | Bin 7396 -> 7396 bytes opt/share/config/images_crystal.zip | Bin 3213504 -> 3213504 bytes opt/share/config/images_hicontrast.zip | Bin 1877237 -> 1877237 bytes opt/share/config/images_oxygen.zip | Bin 3359815 -> 3359815 bytes opt/share/config/images_sifr.zip | Bin 2710046 -> 2710046 bytes opt/share/config/images_tango.zip | Bin 2750162 -> 2750162 bytes opt/share/config/images_tango_testing.zip | Bin 2905779 -> 2905779 bytes opt/share/gallery/computers.sdg | Bin 100334 -> 100334 bytes opt/share/gallery/diagrams.sdg | Bin 632530 -> 632530 bytes opt/share/gallery/education.sdg | Bin 69551 -> 69551 bytes opt/share/gallery/environment.sdg | Bin 162033 -> 162033 bytes opt/share/gallery/finance.sdg | Bin 242719 -> 242719 bytes opt/share/gallery/people.sdg | Bin 344787 -> 344787 bytes opt/share/gallery/sounds.sdg | Bin 4491 -> 4492 bytes opt/share/gallery/sounds.thm | Bin 1457 -> 1457 bytes opt/share/gallery/symbols.sdg | Bin 522278 -> 522278 bytes opt/share/gallery/transportation.sdg | Bin 69959 -> 69959 bytes opt/share/gallery/txtshapes.sdg | Bin 184612 -> 184612 bytes opt/ure/share/java/java_uno.jar | Bin 5147 -> 5147 bytes opt/ure/share/java/juh.jar | Bin 52852 -> 52852 bytes opt/ure/share/java/jurt.jar | Bin 106834 -> 106834 bytes opt/ure/share/java/ridl.jar | Bin 248816 -> 248816 bytes opt/ure/share/java/unoloader.jar | Bin 4304 -> 4304 bytes 124 files changed, 2 insertions(+), 2 deletions(-)
Range of commits https://cgit.freedesktop.org/libreoffice/core/log/?qt=range&q=c7bdee8dbd1cf260a8513a0d31b36f90daa70f1c..ec0080c40cfdb26896537f47a4c2e0439f9afdb1
Adding CC to Kohei Yoshida
Bibisected to the same commit STR 1. Open attachment 161303 [details] 2. CTRL+H 3. Replace X to 2 & Press Replace all (9 seconds prior to the identified commit) minutes after
7:26 min with 6.2.8.2, nothing one could like to work with ...
Repeated replacing changes type of cells, and since mdds groups same types in one vector, changing the first cell of such a vector means the first item in the vector gets erase()-ed -> O(n^2). I'm not sure if there's a good way to handle this and the use case is somewhat niche.
(In reply to Luboš Luňák from comment #8) > Repeated replacing changes type of cells, and since mdds groups same types > in one vector, changing the first cell of such a vector means the first item > in the vector gets erase()-ed -> O(n^2). I'm not sure if there's a good way > to handle this and the use case is somewhat niche. Hi Luboš, thanks for your comments here. Reducing priority to medium
Keeping the new cell values in a separate array buffer, then pushing all these changes to mdds::multi_type_vector as a single call (set() method can take an array of values) may be one way to tackle it. If you change the cell values one by one for a large number of cells, it does get very slow very quickly, as this use case demonstrates. I have historically tackled similar issues this way.
Although(In reply to Kohei Yoshida from comment #10) > Keeping the new cell values in a separate array buffer, then pushing all > these changes to mdds::multi_type_vector as a single call (set() method can > take an array of values) may be one way to tackle it. If you change the > cell values one by one for a large number of cells, it does get very slow > very quickly, as this use case demonstrates. > > I have historically tackled similar issues this way. Although this requires that we know the type of the new cell values ahead of time, which is not always the case, and this particular use case it's a bit tricky.
I have some ideas that *could* help this use case, and it's something we could do on the mdds side. Let me think about this a bit.
Still slow Version: 7.4.0.0.alpha0+ / LibreOffice Community Build ID: 32714f966186d301435d3eb9f7f6950bc9a6bb1e CPU threads: 8; OS: Linux 5.17; UI render: default; VCL: kf5 (cairo+xcb) Locale: fi-FI (fi_FI.UTF-8); UI: en-US Calc: threaded
Luboš Luňák committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/b8aa4d7caf82903b3ba1ff45483756db6835cc60 make SharedString move operator= inline (tdf#126109) It will be available in 7.4.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
After this commit, time goes down from 6min15 down to 6min.
I compare times in version 6.2.4.2, where it whould works good and in latest daily. And difference is only 15 seconds. All time is about 1:45. It works better then for example in version 4.2 where I wait over 3 minutes. Version: 7.4.0.0.alpha1+ (x64) / LibreOffice Community Build ID: bbec710bd25fc5da27636cde73fe4ab23c76904f CPU threads: 8; OS: Windows 10.0 Build 19043; UI render: Skia/Vulkan; VCL: win Locale: cs-CZ (cs_CZ); UI: en-GB Calc: CL
Noel Grandin committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/1cc9f4d972f8a5e1e3f4980942e128dee9a2701c tdf#126109 calc slow when replacing string to number It will be available in 7.5.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Noel Grandin committed a patch related to this issue. It has been pushed to "libreoffice-7-4": https://git.libreoffice.org/core/commit/ceb5b35ca0677375e05f0acd02934045e013c04d tdf#126109 calc slow when replacing string to number It will be available in 7.4.0.0.beta2. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Noel Grandin committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/1a7913c1b57b333223b30194c67c4d8d11e583d2 tdf#126109 calc slow when replacing string to number It will be available in 7.5.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Time after b8aa4d7caf82903b3ba1ff45483756db6835cc60 is 6min (15 May) Time after c0e66cfab12d6c78d68af09498533362724dbd05 is 7min (12 June) Time after 1cc9f4d972f8a5e1e3f4980942e128dee9a2701c is 6m30 (15 June) Time after e4d23c27288b99c3ed3cfa332ff308b31c01f97d is 7min (18 June) I run benchmark twice and time is +/- 5s
Noel Grandin committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/3be3fb9bafc4c2bf17ef3fe4eba0a059199abe24 tdf#126109 calc slow when replacing string to number It will be available in 7.5.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Only 5 sec for . -> , changing in Version: 7.5.0.0.alpha0+ (x64) / LibreOffice Community Build ID: 086055b0d7e44d1d07b3f23af55503e6a3924d87 CPU threads: 8; OS: Windows 10.0 Build 19043; UI render: Skia/Vulkan; VCL: win Locale: ru-RU (ru_RU); UI: ru-RU Calc: threaded Excellent! Thanks Noel!
I confirm. From e4d23c27288b99c3ed3cfa332ff308b31c01f97d (Jun 19) it took 7m50. From 1eaf11278a2f2f81fbff774fef73f72a6b1a4efb (Jun 22) it took 5s. Many thanks for having fixed it and to have catch the problem.