Bug 103919 - Mailmerge e-mail does not send all merged documents
Summary: Mailmerge e-mail does not send all merged documents
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Writer (show other bugs)
Version:
(earliest affected)
5.2.2.2 release
Hardware: x86-64 (AMD64) Linux (All)
: high major
Assignee: Not Assigned
QA Contact:
URL:
Whiteboard:
Keywords: bibisectRequest, regression
Depends on:
Blocks: Mailmerge
  Show dependency treegraph
 
Reported: 2016-11-14 07:29 UTC by Ferdi
Modified: 2017-03-27 12:02 UTC (History)
7 users (show)

See Also:
Crash report or crash signature:


Attachments
Mailmerge window showing difference in sent documents vs. total number of documents (31.12 KB, image/png)
2016-11-21 07:35 UTC, Ferdi
Details
A hack to maybe fix this bug (1.10 KB, patch)
2016-12-14 12:15 UTC, Alex Kempshall
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ferdi 2016-11-14 07:29:05 UTC
When doing a Mail merge for a newsletter, not all mails are sent, although the wizard does create them all. (in any case when saving the documents)
If there is only one recipient No mail is sent, on the mailserver I get a connect from the machine sending them immediately followed by a disconnect, no data is received.
When there are 2 adresses, only the second gets sent, not the first. I see the connect immediately followed by a disconnect of the first one.
If I have 10 mails, only the first 6 are sent, the last four never reach the mailserver (no connection is made)

This is using HTML mailmerge messages, have not tested pdf or ODF sending. Mainly because I need the mails to be HTML.

I run my own mailserver (postscript). If desired I can attach logging from the server showing the connects/disconnects and the document I am trying to send.

The addresses and names in the document are retrieved from an LibreOffice Base database query.
Comment 1 Ferdi 2016-11-14 19:07:18 UTC
Did some more testing, when e-mailing as ODF document, I get only 5 out of ten mails. The order they arrive is also the same every time.
document 1 comes first, then 5, 4, 3, 2, 6 to 10 never get sent.
unfortunately the window that is shown during sending remains to short on screen to be able to read it The numbering on it in the short glimpse that it is seen seems odd though.
Sending as pdf is even worse, only 3 out of 10 mails are sent.
But now when sending HTML, 9 out of 10 get sent.And they are sent in very random order.
This is not very consistent behaviour. In this manner I can just as well send them all manually. And I intent to sent several hundred to all the people that have an subscription to my newsletter so that would be very inconvenient.
Comment 2 Ferdi 2016-11-15 06:42:16 UTC
Just sent out an mailmerge with 130 e-mail adresses. roughly half get sent.
When looking at the dialog showing the progress it displays the text "Sending paused" constantly.
There is a counter in the dialog, counting the number of documents and the number that are sent. When the number of documents counter has reached the total number of documents in the mailmerge, the dialog closes and presumably sending stops. The number of documents sent counter never reaches the total number of documents.
Comment 3 Ferdi 2016-11-21 07:31:55 UTC
(In reply to Ferdi from comment #2)
> Just sent out an mailmerge with 130 e-mail adresses. roughly half get sent.
> When looking at the dialog showing the progress it displays the text
> "Sending paused" constantly.
> There is a counter in the dialog, counting the number of documents and the
> number that are sent. When the number of documents counter has reached the
> total number of documents in the mailmerge, the dialog closes and presumably
> sending stops. The number of documents sent counter never reaches the total
> number of documents.

After carefully looking the above seems to be occurring every time.
The mailmerge should not end when the total number of documents is reached, but when the total number of documents has been sent!
I attached a screenshot, to show the counter and how the number of sent documents are different from the total number of documents.
Comment 4 Ferdi 2016-11-21 07:35:21 UTC
Created attachment 128909 [details]
Mailmerge window showing difference in sent documents vs. total number of documents

Mailmerge stops when total number of documents is reached, should be when number of sent documents equals the total number of documents.
Now only about half of the total number gets sent.
Comment 5 Buovjaga 2016-11-28 20:00:24 UTC
Could you try with 5.3 beta1?

https://wiki.documentfoundation.org/Installing_in_parallel/Linux
http://dev-builds.libreoffice.org/pre-releases/deb/x86_64/LibreOfficeDev_5.3.0.0.beta1_Linux_x86-64_deb.tar.gz

Set to NEEDINFO.
Change back to UNCONFIRMED, if the problem persists. Change to RESOLVED WORKSFORME, if the problem went away.
Comment 6 Ferdi 2016-12-07 06:57:03 UTC
Thanks for the reply, I had a little holiday, so could not test this yet. Will download and install 5.3.0 beta and test with that. I'll post results here.
Comment 7 Ferdi 2016-12-07 18:42:49 UTC
I installed the latest beta and got it running.
Unfortunately I cannot test this problem because in this version it will not connect to any mailserver at all. I find it impossible to sent any mail.

I am using exactly the same settings that work in 5.2.2 but they do not work in 5.3.0

Also tried other smtp servers that I have access to, no luck they all fail with the message below:

LibreOfficeDev could not connect to the outgoing mail server. Check your system's settings and the settings in LibreOfficeDev. Check the server name, the port and the secure connections settings
--
<class 'LookupError'>: unknown encoding: idna, traceback follows
  File "/home/ferdi/Downloads/LibreOfficeDev_5.3.0.0.beta1_Linux_x86-64_deb/DEBS/Install/opt/libreofficedev5.3/program/mailmerge.py", line 106, in connect
    self.server = smtplib.SMTP(server, port,timeout=tout)
  File "/home/ferdi/Downloads/LibreOfficeDev_5.3.0.0.beta1_Linux_x86-64_deb/DEBS/Install/opt/libreofficedev5.3/program/python-core-3.5.0/lib/smtplib.py", line 251, in __init__
    (code, msg) = self.connect(host, port)
  File "/home/ferdi/Downloads/LibreOfficeDev_5.3.0.0.beta1_Linux_x86-64_deb/DEBS/Install/opt/libreofficedev5.3/program/python-core-3.5.0/lib/smtplib.py", line 335, in connect
    self.sock = self._get_socket(host, port, self.timeout)
  File "/home/ferdi/Downloads/LibreOfficeDev_5.3.0.0.beta1_Linux_x86-64_deb/DEBS/Install/opt/libreofficedev5.3/program/python-core-3.5.0/lib/smtplib.py", line 306, in _get_socket
    self.source_address)
  File "/home/ferdi/Downloads/LibreOfficeDev_5.3.0.0.beta1_Linux_x86-64_deb/DEBS/Install/opt/libreofficedev5.3/program/python-core-3.5.0/lib/socket.py", line 689, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/home/ferdi/Downloads/LibreOfficeDev_5.3.0.0.beta1_Linux_x86-64_deb/DEBS/Install/opt/libreofficedev5.3/program/python-core-3.5.0/lib/socket.py", line 728, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
Comment 8 Ferry Toth 2016-12-07 20:23:27 UTC
I am confirming this for LibO 5.2.2.2 on Ubuntu. I regularly send ~ 80 html formatted messages. 

After upgrading to Ubuntu 16.10 I am getting complaints of people not receiving them. The dialog counts to about 80, making me think that all have actually been sent. A poll among the receivers shows only about half were received.
Comment 9 Cor Nouws 2016-12-09 21:33:43 UTC
I have seen the same problem yesterday in 5.2.x
Comment 10 Ferry Toth 2016-12-11 18:41:59 UTC
I just tried using LibO 5.2.3.2 (aka fresh) from the Libreoffice PPA. I haven't yet polled the receivers but I'm pretty sure the problem still happened.
Just before the sending dialog closed I saw the counter reaching 97 with about half of that actually transferred.
Comment 11 Alex Kempshall 2016-12-14 12:10:21 UTC
I've had this problem. I have a test database with 3 records, only one of them ever gets through.

This is on 5.4.0.0.alpha0+.

I have amended sw/source/ui/dbui/mmoutputtypepage.cxx to put two sleeps in and I now get all 3 messages sent.

Not tried it with more than 3 messages. I shall attach a git diff with my changes.

Hope this helps a developer.
Comment 12 Alex Kempshall 2016-12-14 12:15:45 UTC
Created attachment 129628 [details]
A hack to maybe fix this bug

I would in no way suggest putting sleeps in to solve this bug. This diff is a suggestion to the developers as to where the problem may be. 

I have tested this on Lubuntu 14.04 LTS, Slackware 14.2 64bit and Slackware 14.2 32bit all using freshly compiled 5.4.0.0.alpha0+

Alex
Comment 13 Alex Kempshall 2016-12-30 17:47:44 UTC
The problem appears to be with latency in combination with monitoring the wrong variable.

Currently If I send 63 messages to smtp.gmail.com, Libreoffice gives the appearance of struggling to keep up. It appears to pre-process all the batch of 63 messages quite quickly. In parallel with the pre-processing it sends processed messages to the smtp server this is done at a slow pace. Once it's pre-processed all 63 messages it doesn't wait until all the messages have been sent but just closes the dialog stopping Libreoffice from sending any more messages to smtp.gmail.com. Hence the problem.


If I start an smtp server on my own machine and send the messages to localhost I don't see this problem as the messages have got less distance to travel and keep pace with the pre-processing.

In the past up to and including LO 5.1 the progress dialog was always kept open and had to be closed manually. I have tested a patch that will revert to that behavior. I will submit it to gerret during the week commencing Monday 2nd January 2017.

Alex
Comment 14 Ferdi 2016-12-30 18:30:08 UTC
Thank you Alex! Great work.
Comment 15 Aleksander Kelšin 2017-02-11 14:16:03 UTC
This bug has been around for a while. That's why I keept 5.1.* for now. Today I tried 5.2.5.1 and e-mail is sent just for one record, while other functions (Edit Individual Documents, Save Merged Document and Print Merged Documents) work fine.
Comment 16 Eric 2017-02-18 19:42:44 UTC
Hi,

I can confirm that this bug also happen on Ubuntu 16.04 , LO Version: 5.3.0.3
Build ID: 1:5.3.0~rc3-0ubuntu1~xenial1.1
Threads CPU : 12; Version de l'OS :Linux 4.4; UI Render : par défaut; VCL : gtk2; Moteur de mise en page : nouveau; 
Locale : fr-FR (fr_FR.UTF-8); Calc: group


I've turned on debug in mailmerge.py, and it sends the first email and not the others.

Here is the log  :

PyMailServiceProvider init
PyMailServiceProvider create with <Enum instance com.sun.star.mail.MailServiceType ('SMTP')>
PyMailSMTPService init
python version is: 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609]
PyMailSMTPService getSupportedConnectionTypes
PyMailSMTPService connect
ServerName: relay.dommel.be
Port: 25
False
Timeout: <object object at 0x7fda2419b0c0>
ConnectionType: Insecure
PyMailSMTPService isConnected
PyMailSMTPService sendMailMessage
PyMailSMTPService subject: test
PyMailSMTPService from:  Eric Bechet
PyMailSMTPService from:  xxxxxxx@yyy.com
PyMailSMTPService send to: ('bzbz@email.com',)
PyMailSMTPService flavors len: 1
PyMailSMTPService mimetype is: text/plain; charset=UTF-8; format=flowed
('PyMailSMTPService recipients are: ', dict_keys(['bzbz@email.com']))
send: 'ehlo [127.0.1.1]\r\n'
reply: b'250-amber.schedom-europe.net\r\n'
reply: b'250-STARTTLS\r\n'
reply: b'250-PIPELINING\r\n'
reply: b'250 8BITMIME\r\n'
reply: retcode (250); Msg: b'amber.schedom-europe.net\nSTARTTLS\nPIPELINING\n8BITMIME'
send: 'mail FROM:<xxxx@yyy.com>\r\n'
reply: b'250 ok\r\n'
reply: retcode (250); Msg: b'ok'
send: 'rcpt TO:<bzbz@email.com>\r\n'
reply: b'250 ok\r\n'
reply: retcode (250); Msg: b'ok'
send: 'data\r\n'
reply: b'354 go ahead\r\n'
reply: retcode (354); Msg: b'go ahead'
data: (354, b'go ahead')
send: b'MIME-Version: 1.0\r\nContent-Type: text/plain; charset="utf-8"\r\nContent-Transfer-Encoding: quoted-printable\r\nSubject: test\r\nFrom: =?utf-8?q?Eric_Bechet?= <xxx@yy.com>\r\nTo: bzbz@email.com\r\nX-Mailer: LibreOffice 5.3 via Caolan\'s mailmerge component\r\nDate: Sat, 18 Feb 2017 20:39:13 +0100\r\n\r\n=EF=BB=BFBonjour Eric B=C3=A9chet, habitant Montr=C3=A9al.Votre courriel es=\r\nt dsfsdf@fff.be\r\n    =20\r\nHello\r\n\r\n\r\n.\r\n'
PyMailSMTPService isConnected
PyMailSMTPService disconnect
send: 'quit\r\n'
reply: b'250 ok 1487446761 qp 25915\r\n'
reply: retcode (250); Msg: b'ok 1487446761 qp 25915'
data: (250, b'ok 1487446761 qp 25915')
reply: b'221 amber.schedom-europe.net\r\n'
reply: retcode (221); Msg: b'amber.schedom-europe.net'
Comment 17 Jeff Bennett 2017-03-26 21:49:17 UTC
I can confirm that this problem is still present in Version: 5.3.1.2
Build ID: e80a0e0fd1875e1696614d24c32df0f95f03deb2
CPU Threads: 8; OS Version: Windows 6.2; UI Render: default; Layout Engine: new; 
Locale: en-US (en_US); Calc: group

It seems like it tracks "sent" and "created" and as soon as all the documents are created, the dialog closes and no more emails are sent.  This results in ~half the expected emails actually being sent.  Some go out, some do not.

As a further annoyance, the dialog closes so it becomes impossible to see which emails failed.

If the dialog just didn't auto close, I'm guessing all would be well.
Comment 18 Alex Kempshall 2017-03-27 12:02:53 UTC
I did submit a hack for this which was to simply take out the code that closed the dialog. It got rejected because

"
Patch Set 2: Code-Review-2

Sure it fixes the problem, if the user keeps LO open long enough, but you won't get any feedback _when_ that is. With this patch the MailDispatcher thread is simply never stopped.
"

It's been suggested that -

"
The email progress dialog is created using
VclPtr<SwSendMailDialog> pDlg =
VclPtr<SwSendMailDialog>::Create(pButton, *xConfigItem);

And this dialog actually creates and keeps the mail dispatcher alive, as you can see in struct SwSendMailDialog_Impl.

The main problem seems to be the last endDialog(pButton);, which you
removed. As the pDlg has the pButton as a parent, the destruction of the main dialog also closes the progress dlg, which shuts down the dispatcher.

So a quick fix would probably be to make the SwSendMailDialog modal and execute it before the endDialog. This should block closing the MM wizard. I don't know, if it has a way to interact, but I guess it closes when all mails are send.

A better fix would be to change the parent of SwSendMailDialog to something else, so it won't close on MM wizard close, and will still process the mails in the background. So a user can still use LO while the emails are send, but obviously is also able to close LO.
"