Bug 56355 - MAILMERGE: Creating mailing with more than 1000 records is incredible slow
Summary: MAILMERGE: Creating mailing with more than 1000 records is incredible slow
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Writer (show other bugs)
Version:
(earliest affected)
3.5.6.2 release
Hardware: x86 (IA32) Linux (All)
: medium critical
Assignee: Jan-Marek Glogowski
URL:
Whiteboard: target:4.2.0 lhm-limux
Keywords:
: 79067 (view as bug list)
Depends on: 80823
Blocks:
  Show dependency treegraph
 
Reported: 2012-10-24 14:36 UTC by Rudolf Kollien
Modified: 2014-11-10 15:20 UTC (History)
3 users (show)

See Also:
Crash report or crash signature:


Attachments
Screenshot of the hanging status monitor of mailmerge (113.84 KB, image/png)
2012-10-25 08:24 UTC, Rudolf Kollien
Details
Screenshot what happens when status monitor is moved around (159.24 KB, image/png)
2012-10-25 08:29 UTC, Rudolf Kollien
Details
csv file with 22000 records (766.71 KB, text/csv)
2013-02-15 17:19 UTC, Brenda Granados
Details
printout of mail merge using 22000 records (188.10 KB, application/vnd.oasis.opendocument.text)
2013-02-15 17:20 UTC, Brenda Granados
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Rudolf Kollien 2012-10-24 14:36:54 UTC
Hallo,

we switched over from OOo3.2 to LO 3.5.2 on RHEL (Centos) 5.8 x64. We use(d) the writer's mailmerge feature for printing mass mail. The datasources are csv files.

When we try to print the mailing, LO is incredible slow, resp. hangs completely. This happens when using a file with more than aprox. 500 records. Up to 500 records it takes minutes until the printing dialog appears. The highest amount of records we where able to print was 1500. But therefor we waited more than 6 minutes until the printing dialog appeared. With a "normal" mailing of 21000 records still after 40 minutes nothing happend! Because of this, we are unable to print our mailings. With OOo the same action (from selection dialog to print dialog) takes round about 2 or 3 seconds(!!!).

We tried different types of settings in LO. We increased the amount of memory, we disabled the java engine. We loaded the address records in a postgresql database, we imported the csv file in a Calc document and used this as datasource. But nothing changed. soffice.bin consumes 99.9% of a CPU, the "save monitor" (in the german version "Speichern-Monitor") appears with a transparent background (the rest of the selection dialog shines through) and nothing more happens.

We tried LO 3.6.2.2 with no success (nothing changed). T

As mentioned above we run on:
RHEL (Centos) x64 with kernel 2.6.18-308.16.1.el5
LibreOffice 3.5.6.2 x32 (RPM)
36GB of memory
16 CPUs
Comment 1 Rudolf Kollien 2012-10-25 08:24:12 UTC
Created attachment 69054 [details]
Screenshot of the hanging status monitor of mailmerge

At the situation shown in the screenshot, no more other screen updates from LO are done/possible. Starting an new writer/calc/impress window is possible, but the window isn't shown, no entries/editing is possible.
Comment 2 Rudolf Kollien 2012-10-25 08:29:41 UTC
Created attachment 69055 [details]
Screenshot what happens when status monitor is moved around

This screenshot shows what happens to the screen when you move the "hanging" status monitor around the screen.
Comment 3 Brenda Granados 2013-02-15 17:18:12 UTC
Hi. I first created an .ods spreadsheet with 2000 records, and created a form letter using title, name, address, etc for fields. I printed to file without issue. It took several minutes (5-6), but LibreOffice did not hang, and the print out completed.

I then created a csv file with 22,000 records. I printed this to file, as well. It took 1 hour and 20 minutes, and never froze. The printout dialog constantly showed the next number of record it was printed. (Example: Printing 1000...Printing 1001...Printing 1002). During this time, LibreOffice used anywhere from 40-60% CPU. I was able to use other applications, including other LibreOffice products, except Writer, during this time. Trying to open and edit a Writer document was very sluggish, and I didn't spend too much time with that. It did not cause any crash or freezing. I did experience that moving the print dialog in Libre Office would result in showing the incremental movements.

In opening the resulting printout, LibreOffice Writer was very sluggish, and would use about 100% CPU.
Comment 4 Brenda Granados 2013-02-15 17:19:46 UTC
Created attachment 74889 [details]
csv file with 22000 records
Comment 5 Brenda Granados 2013-02-15 17:20:25 UTC
Created attachment 74890 [details]
printout of mail merge using 22000 records
Comment 6 Brenda Granados 2013-02-15 17:26:38 UTC
I ran my tests using LibreOffice 4.0.1.0+ on 64-bit Ubuntu 12.04 LTS.

I'm not a hardware guru, but here are specs from my System Details, if it's of help:

7.8 GiB
AMD FX(tm)-4100 Quad-Core Processor × 4
Comment 7 Björn Michaelsen 2013-11-07 19:42:01 UTC
Jan-Marek Glogowski at the Freiburg Hackfest created a callgrind trace of mailmerge, its available here:
https://owncloud.documentfoundation.org/Common/QA/Bugzilla/Bugs/56355
Comment 8 Jorendc 2013-11-20 18:08:20 UTC
(In reply to comment #7)
> Jan-Marek Glogowski at the Freiburg Hackfest created a callgrind trace of
> mailmerge, its available here:
> https://owncloud.documentfoundation.org/Common/QA/Bugzilla/Bugs/56355

And submitted a patch http://cgit.freedesktop.org/libreoffice/core/commit/?id=13a8fac05425f9d66c643f25faa49c1f17c62474 :).

@Björn: can we mark this one as fixed?

Kind regards,
Joren
Comment 9 Björn Michaelsen 2013-11-20 20:00:10 UTC
@Jorendc: indeed ;)

Jan-Marek Glogowski committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=13a8fac05425f9d66c643f25faa49c1f17c62474

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 10 Rudolf Kollien 2013-11-21 11:33:29 UTC
Ok. Will test it next week. In the current master as of "master~2013-11-20_23.55.57..." the patch seems still not to be included. The mail merge runs as slow as always. But i took the test to make a note of the elapsed time of the current master and will compare it next week with the patched master.
Comment 11 Jan-Marek Glogowski 2013-11-21 14:44:21 UTC
Just to make it clear - the patch doesn't speed up MM per-se. This just postpones the layouting of the generated _single_ MM document, so it takes constant time per document to generate, instead of the increasing time to relayout the whole combined document for every appended MM document.

Still not sure, if my MM problem is the same then yours (me running MM via Uno from WollMux).
Comment 12 Rudolf Kollien 2013-11-25 13:22:12 UTC
As of the last daily master (4.3.0) from 25. Nov. there is no increased speed. All is the same as initially reported. The only thing changed is, that the printing monitor now (since 4.1.x) shows the progress again. So the user can see work in progress.

Printing a mass mail with 21.000 records still takes more than 90 minutes to bring up the printing dialog and again 90 minutes to start sending the output to the printer.
Comment 13 Daniel 2014-01-06 21:28:40 UTC
Hello

I found that "Format - Bullets and Numbering" in the source document make MM becomes slower an slower with every MM document.

LibreOffice 4.1.3 an 4.2.1 
Windows 7
Comment 14 Cor Nouws 2014-06-17 11:49:47 UTC
as temporary bandage, one can of course split large datasets in more batches.
I did sets with few thousands records some years back. Was indeed slow.
Comment 15 Cor Nouws 2014-06-17 11:50:33 UTC
*** Bug 79067 has been marked as a duplicate of this bug. ***
Comment 16 Michael Meeks 2014-07-10 09:55:23 UTC
> I found that "Format - Bullets and Numbering" in the source document
> make MM becomes slower an slower with every MM document.

Perhaps some lovely issue with outline numbering across all the documents then (?)
Comment 17 Cor Nouws 2014-11-04 20:54:21 UTC
Anyone tested this in a recent version?
thanks,
Cor
Comment 18 Jan-Marek Glogowski 2014-11-05 08:50:43 UTC
It's not fixed yet, but my patch is an improvement for some simple document types. People are still working on this.
Comment 19 Luboš Luňák 2014-11-09 18:59:45 UTC
This bugreport actually does not provide any testcase to reproduce the specific problem, so since I've just speeded up MM quite a bit in master, let's close this, as without further info it cannot be checked if this specific case is still slow or not.
Comment 20 Jan-Marek Glogowski 2014-11-10 08:47:30 UTC
(In reply to Lubos Lunak from comment #19)
> This bugreport actually does not provide any testcase to reproduce the
> specific problem, so since I've just speeded up MM quite a bit in master,
> let's close this, as without further info it cannot be checked if this
> specific case is still slow or not.

There is test document and spreadsheet attached to the duplicate bug 79067 we opened.
Comment 21 Luboš Luňák 2014-11-10 15:20:48 UTC
That one performs reasonably well now.