Bug 105288 - MailMerge: when running e-mail using a macro only a few messages get sent
Summary: MailMerge: when running e-mail using a macro only a few messages get sent
Status: VERIFIED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Writer (show other bugs)
Version:
(earliest affected)
5.4.0.0.alpha0+
Hardware: x86-64 (AMD64) Linux (All)
: medium normal
Assignee: Not Assigned
QA Contact:
URL:
Whiteboard: target:5.4.0 target:5.3.1
Keywords:
Depends on:
Blocks: Mail-Merge
  Show dependency treegraph
 
Reported: 2017-01-12 16:02 UTC by Alex Kempshall
Modified: 2017-04-10 09:40 UTC (History)
4 users (show)

See Also:
Crash report or crash signature:


Attachments
This is the starting document the MailMerge (12.19 KB, application/vnd.oasis.opendocument.text)
2017-01-12 16:03 UTC, Alex Kempshall
Details
This is the database for the mailmerge (11.72 KB, application/vnd.oasis.opendocument.database)
2017-01-12 16:04 UTC, Alex Kempshall
Details
The Writer document to be merged and emailed (9.07 KB, application/vnd.oasis.opendocument.text)
2017-04-10 05:19 UTC, Doug Hutcheson
Details
The spreadsheet of test data for the merge (14.12 KB, application/vnd.oasis.opendocument.spreadsheet)
2017-04-10 05:20 UTC, Doug Hutcheson
Details
The Base document containing the macro fncEmailMerge (7.44 KB, application/vnd.oasis.opendocument.database)
2017-04-10 05:21 UTC, Doug Hutcheson
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Alex Kempshall 2017-01-12 16:02:18 UTC
In Writer when attempting to email messages using a macro only a few get sent. This occurs when trying to send emails through smtp.gmail.com. Possibly due to latency issues. Only discovered after implementing a fix for tdf#105071

Steps to reproduce the problem.

1.
Download the files CompanySeedStarted1.odt and CompanySeedTestData1.odb from this bug report. The database CompanySeedTestData1.odb contains 63 entries all with dummy email addresses.

2.
Start LibreOffice writer with the document CompanySeedStarted1.odt downloaded in step 1. Note: this file contains a macro so be mindful that some security steps might need to be taken.

3.
Tools -> Mail Merge Wizard

4.
The "Select starting document" dialog box should appear. The left pane should contain 5 steps. In the right pane select the "Use Current Document" Radio Button. Then press the "Next >>" Button.

5.
The "Select document type" dialog box should appear. Select the "E-mail message" Radio Button. Then press the "Next" Button.

6.
The "Insert address block" dialog box should appear. Press the "Select Different Address List" Button.

7.
The "Select Address List" dialog box should appear. Press the "Add" Button.

8.
The "Open - LibreOffice" dialog should appear. Navigate to directory containing CompanySeedTestData1.odb downloaded in step 1. Once the database has been selected press the "Open" Button.

9.
The "Select Address List" dialog box should re-appear with the database CompanySeedTestData1 and the table tblCompanySeedTestData1 added to the list of available databases. If not already selected - select this database. Press the "OK" Button.

10.
Control should be returned to the "Mail Merge Wizard" dialog. Press the "Finish" Button.

11.
If not already done so. Establish the Mail Merge E-mail settings by navigating from the menu Tools -> Options -> LibreOffice Writer -> Mail Merge E-Mail

11.1
The  "Options LibreOffice Writer Mail Merge E-Mail" dialog should appear. Enter the senders name you want to be called by, the senders email address, the server name, the port number. If not already checked left click the "Use secure connection (SSL) check box. Push the "Server Authentification" Button.

11.2
The "Server Authentification" dialog should appear. Enter the senders email address and the senders password for the smtp server. Push the "OK" button.

11.3
Control will be passed back to the "Options LibreOffice Writer Mail Merge E-Mail" dialog. Push the OK Button. Control will return to the open document.

12.
Run the MailMerge macro by navigating from the menu Tools -> Macros - Run Macro.

13.
The "Macro Selector" dialog should appear.

14.
In the left hand "Library" pane navigate to the macro CompanyStarted1.odt -> Standard -> CompanyStarted1

15.
In the right hand "Macro Name" pane select the macro "Email_Test_0001". Press the "Run" button".

16.
Control should be passed back to the document. Progress will be indicated by the progress bar in the status area. Should process 63 of 63 messages. A "Done" message box should appear.

17.
Though the Mail Merge process indicates that 63 messages have been processed only the first few appear on the server configured in the "Mail Merge E-mail settings" dialog. 


Alex
Comment 1 Alex Kempshall 2017-01-12 16:03:45 UTC
Created attachment 130361 [details]
This is the starting document the MailMerge
Comment 2 Alex Kempshall 2017-01-12 16:04:11 UTC
Created attachment 130362 [details]
This is the database for the mailmerge
Comment 3 Alex Kempshall 2017-01-12 16:16:17 UTC
In MailMerge there are two processes going on in parallel. 

1. 
Format messages and place them on a dispatcher queue. When all messages have been placed on the dispatcher queue the queue is stopped.

2.
Take messages off the dispatcher queue and send and await reply from smtp server.


If step 2 above takes a lot longer than step 1 the queue is closed prematurely.

The code that closes the queue is here in sw/source/uibase/dbui/dbmgr.cxx  SwDBManager::MergeMailFiles

    if( xMailDispatcher.is() )
    {
        xMailDispatcher->stop();
        xMailDispatcher->shutdown();
    }

If this is  changed to 

    while(xMailDispatcher->messageCount());

    if( xMailDispatcher.is() )
    {
        xMailDispatcher->stop();
        xMailDispatcher->shutdown();
    }


Where xMailDispatcher->messageCount() is in sw/source/uibase/dbui/maildispatcher.cxx and looks like

int MailDispatcher::messageCount()
{
    return messages_.size();
}
Comment 4 Alex Kempshall 2017-01-12 16:31:35 UTC
Before the above change I was usually getting  15 of my 62 test messges.

With the change I'm now getting 61 of my 62 messages. So the dispatcher is still getting closed too early, but progress being made.
Comment 5 Alex Kempshall 2017-01-12 17:54:07 UTC
In MailDispatcher::run of added a flag sendMailMessageNotifyListener in sw/source/uibase/dbui/maildispatcher.cxx to ensure that on the last record the dispatcher is not closed prematurely.


            thread_status_guard.clear();
            sendMailMessageNotifyListener_ = true;
            uno::Reference<mail::XMailMessage> message = messages_.front();
            messages_.pop_front();
            message_container_guard.clear();
            sendMailMessageNotifyListener(message);
            sendMailMessageNotifyListener_ = false;
Comment 6 Jan-Marek Glogowski 2017-01-13 14:35:57 UTC
There are two problems with your suggestion:

1. The correct interface to get (async) information from the MailDispatcher is to register a listener.

2. A while loop like "while(xMailDispatcher->messageCount());" will block any event processing of LO. Now for a short time, that's ok, but who knows in advance how many mails still need processing and how long this will take and LO is meanwhile frozen.

It would be nice if you can test private/jmux/tdf#105288. It's currently just compile tested, so I hope it won't deadlock ;-)
Comment 7 Alex Kempshall 2017-01-19 08:35:39 UTC
I've tested private/jmux/tdf#105288 and now get all messages sent.

As I'm now getting all messages sent I've identified another issue! My macro in sending the messages displays a progress bar in the status area. The progress bar should show progress of the sending of 63 messages - which it does. On completion a "Done" message box should appear which it does.

However, the "Done" message box appears some time after the progress bar has indicated that all 63 messages have been sent. I have no evidence, but what the progress bar appears to be doing is tracking the mail merge when it should be tracking the sending of the merged documents via email. Should I raise this as a new bug report or can it be dealt with here?
Comment 8 Buovjaga 2017-01-24 07:41:56 UTC
(In reply to Alex Kempshall from comment #7)
> However, the "Done" message box appears some time after the progress bar has
> indicated that all 63 messages have been sent. I have no evidence, but what
> the progress bar appears to be doing is tracking the mail merge when it
> should be tracking the sending of the merged documents via email. Should I
> raise this as a new bug report or can it be dealt with here?

I think a new report would be best as this already has a lot of text :)
Thanks for your efforts. I guess you can close this as fixed, if you are happy.
Comment 9 Alex Kempshall 2017-01-24 09:39:22 UTC
I'll leave as open until the patch has been applied to master, it's currently in a private branch, and I can retest.

Once It's been applied to master and I've retested I can see whether the other problem is still there. If it is I will raise a separate report and we move forward.
Comment 10 Commit Notification 2017-02-10 10:46:18 UTC
Jan-Marek Glogowski committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=5f2e1eda4532a52fe9f0f9d2017b37d566f25f33

tdf#105288 MM wait until all emails are send

It will be available in 5.4.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 11 Commit Notification 2017-02-13 08:21:44 UTC
Jan-Marek Glogowski committed a patch related to this issue.
It has been pushed to "libreoffice-5-3":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=25b0442d98e69072c961e403f5f03433c7a05b54&h=libreoffice-5-3

tdf#105288 MM wait until all emails are send

It will be available in 5.3.1.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 12 Alex Kempshall 2017-02-15 14:38:23 UTC
Give it a test in Master and in the development build for 5.3.1. 

Looks OK to me.

Thanks
Comment 13 Buovjaga 2017-02-15 16:07:15 UTC
Let's set to verified as Alex tested on both target versions.
Comment 14 Doug Hutcheson 2017-04-08 07:39:01 UTC
I just tested 5.3.3 and the problem is still there. I see the eleven expected emails being generated, according to the dialog, and then being sent, according to the dialog, but only one is actually sent.

Version: 5.3.3.0.0+
Build ID: bb8b699f217dd0d0f25e4efb37ef5380074c61ca
CPU Threads: 8; OS Version: Linux 4.10; UI Render: default; VCL: gtk2; Layout Engine: new; 
TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:libreoffice-5-3, Time: 2017-04-07_03:52:20
Locale: en-AU (en_AU.UTF-8); Calc: group
Comment 15 Doug Hutcheson 2017-04-09 00:26:28 UTC
(In reply to Doug Hutcheson from comment #14)

I should add that I tried it with a single recipient and that message was not sent.
Comment 16 Doug Hutcheson 2017-04-10 05:19:22 UTC
Created attachment 132435 [details]
The Writer document to be merged and emailed
Comment 17 Doug Hutcheson 2017-04-10 05:20:12 UTC
Created attachment 132436 [details]
The spreadsheet of test data for the merge
Comment 18 Doug Hutcheson 2017-04-10 05:21:45 UTC
Created attachment 132437 [details]
The Base document containing the macro fncEmailMerge
Comment 19 Doug Hutcheson 2017-04-10 05:24:57 UTC
(In reply to Doug Hutcheson from comment #15)
> (In reply to Doug Hutcheson from comment #14)
> 
> I should add that I tried it with a single recipient and that message was
> not sent.

The three documents submitted above represent an emailmerge solution that works. If I run emailmerge from the wizard/toolbar in 5.3.3, few if any emails are sent. If I run my macro, they all get sent. I have submitted these as a possible work-around for others in the same boat.

Cheers,
Doug
Comment 20 Alex Kempshall 2017-04-10 09:22:44 UTC
This bug report relates to emails sent by macro Which has been fixed by Jan-Marek and tested by myself see previous comments.

What Doug seems to be describing is a problem with sending messages using the mail merge wizard which should actually relate to TDF#103919.
Comment 21 Doug Hutcheson 2017-04-10 09:35:30 UTC
(In reply to Alex Kempshall from comment #20)
> This bug report relates to emails sent by macro Which has been fixed by
> Jan-Marek and tested by myself see previous comments.
> 
> What Doug seems to be describing is a problem with sending messages using
> the mail merge wizard which should actually relate to TDF#103919.

My apologies Alex - you are quite correct and I am out of order.   "8-[