Bug 56618 - MAILMERGE: Using a csv as database creates SOME duplicate pages
Summary: MAILMERGE: Using a csv as database creates SOME duplicate pages
Status: RESOLVED WORKSFORME
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Writer (show other bugs)
Version:
(earliest affected)
3.5.6.2 release
Hardware: Other All
: high critical
Assignee: Not Assigned
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: Mail-Merge
  Show dependency treegraph
 
Reported: 2012-10-31 15:33 UTC by Rudolf Kollien
Modified: 2017-08-02 10:49 UTC (History)
5 users (show)

See Also:
Crash report or crash signature:


Attachments
sample spreadsheet of records (16.24 KB, application/vnd.oasis.opendocument.spreadsheet)
2013-02-28 16:55 UTC, Brenda Granados
Details
Output of 131 labels from source file of 101 records (42.26 KB, application/pdf)
2013-05-01 17:01 UTC, Alex
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Rudolf Kollien 2012-10-31 15:33:24 UTC
Hello,

when a csv text file (separated by ; and "") is used as input source for mailmerge, some of the records are printed twice. 

We have a mailmerge with a csv file with 240 records. The first record defines the field names, the rest are simple mail addresses. We created a LO database file and registered it in writer. When choosing the file as data source for the text document, the right count of records is displayed in the field/record selector of Writer. When we than print the mailmerge (ether direct to the printer or to file(s)) with all records, than some of the records (the page/document) printed twice. To be clear: the data is NOT printed twice on ONE page, a second page with complete same data is generated.

When we load the csv file into a Calc spreadsheet, save the spreadsheet and than select this speradsheet as the input source for the mailmerge, all went normal. Each record is printed once.

- In LO Base, when you view the records, there are the right count.
- In the field/record selector the right count is shown.
- In the created spreadsheet the right count is shown.

When printing direct from the text file, the duplicated records are always the same, so the error is repeatable. 

We thought, there is a error in the csv, but:
1) Base and Calc show the content right
2) Calc imports right
3) Datasource selector of Writer shows right
4) We used this file format with OOo 3.2 with no problems, the creating program wasn't changed.

The error occurs with LO 3.5.6.2 on Linux and also with 3.6.0.4 on Mac. So it seems not to be OS specific.

I can provide the text file to a assigned developer by PM. Please show understanding, that i cannot attach this file viewable to the public. But the data is not so confidential that it cannot be used in a development environment.
Comment 1 Rudolf Kollien 2012-10-31 16:56:44 UTC
Correction!!!!

The field/record selector in Writer shows WRONG count. Sorry, didn't look exactly to the counter. I found, that in the selector dialog the records, which are printed twice are in one block. So it isn't that i get an output like:

Page 23 -> AAAAA
Page 24 -> AAAAA
Page 25 -> BBBBB
Page 26 -> BBBBB
Page 27 -> CCCCC
Page 28 -> CCCCC
....

but

Page 23 -> AAAAA
Page 24 -> BBBBB
Page 25 -> CCCCC
Page 26 -> AAAAA
Page 27 -> BBBBB
Page 28 -> CCCCC

In the field/record selector of Writer you can also see:
EEEEE
RRRRR
AAAAA
BBBBB
CCCCC
AAAAA
BBBBB
CCCCC
XXXXX
ZZZZZ

in the middle of the file.
Comment 2 Rudolf Kollien 2012-10-31 17:44:28 UTC
Correction 2:

Sometimes the field/record selector shows the right count (the number of records in the file) sometimes it shows the count including the double printed records. Could not figure out when which happens.
Comment 3 Alex 2012-12-03 04:50:50 UTC
This problem looks very much like what I am experiencing.
I am using LibreOffice 3.5.4.2 on Ubuntu Linux 12.04 LTS.
I have registered a directory on a unix server as a database of csv formatted files used for creating packaging labels. 

When we open the label template file that is linked with a particular database table in that directory, it shows exactly 30 records too many in the resource manager.  The extra records will print as a restart of the database at what whould have been the end of the label set.  

Selecting another table in the resource manager and then reselecting the proper table, correctly shows the proper number of records in the table and prints as expected.
Comment 4 Alex 2012-12-03 05:04:49 UTC
Correction:

This all happens after we go to print and say yes to mail merge.
Comment 5 Rudolf Kollien 2012-12-03 08:19:44 UTC
Hello,

our counter of wrong records is 30 too. I tought, the number of duplicated records are random. But it seems, this are symptomatically.

We currently help us by sorting the data an ascending field descending and vise versa. Doing a sort on the data eliminates the duplicates. But on input data with more than 20.000 records, this is not a real solution. And not always there is a field we can do a ascending/descending.
Comment 6 franzkraft11 2012-12-19 18:10:07 UTC
(In reply to comment #5)
> Hello,
> 
> our counter of wrong records is 30 too. I tought, the number of duplicated
> records are random. But it seems, this are symptomatically.
> 
> We currently help us by sorting the data an ascending field descending and
> vise versa. Doing a sort on the data eliminates the duplicates. But on input
> data with more than 20.000 records, this is not a real solution. And not
> always there is a field we can do a ascending/descending.

Hello!

I experienced the same behavior: a block of 30 records, from 12th to the 41th, are duplicated.
If I click on the 'Refresh' button, the incorrect records disappear.
Comment 7 Brenda Granados 2013-02-28 16:54:04 UTC
Hi, I cannot confirm the 30 records result, but I seem to get double records when I use the mail merge feature as well. I have tried it using csv files that contain 29 records (resulted in 57 printed) and 245 records (resulted in 486 printed). I tried to sort and a query to select distinct entries, but that didn't help. All these are also true for me:

- In LO Base, when you view the records, there are the right count.
- In the field/record selector the right count is shown.
- In the created spreadsheet the right count is shown.

I obtained these results using LibreOffice 4.0.1.0+ on Ubuntu 12.04 LTS.
Comment 8 Brenda Granados 2013-02-28 16:55:17 UTC
Created attachment 75700 [details]
sample spreadsheet of records
Comment 9 Domingo Sacristan 2013-03-15 08:20:25 UTC
I can confirm this bug not only by mailmerge with csv text file but generally.

Libreoffice Version: 4.0.1
Platform: MS windows XP, windows 7

Testcase:

1. print  document with mailmerge fields
2. If the amount of records > 41, normally a block of records between 0 and 41  are duplicated.
3. We help us as well by sorting the data new
Comment 10 Alex 2013-05-01 16:08:46 UTC
This problem is still present in version 4.0.2.2 (Build ID: 400m0(Build:2)).
Clicking the refresh button at the top fixes it, but seeing as it was just freshly loaded, I wouldn't think a refresh would be necessary.
Comment 11 Alex 2013-05-01 17:01:38 UTC
Created attachment 78731 [details]
Output of 131 labels from source file of 101 records

I ran 2 test cases, generating serial numbered labels.  In both cases, the 1st 42 records were printed as expected and then records were restarted at record 13 then running out through the end.  In both cases, the mail merge dialog initially shows 41 records.  Clicking the >| button changes that to 131 records.  Both cases generated 30 extra records. 
To demonstrate this, I created a file of 100 serial numbered records with 1 title record at the beginning, making 131 records, and printed to a PDF which I have attached.  Because the records are numbered, it is easy to see what the program is doing.
Clicking REFRESH at the beginning of the print process, where you can select records from the source, fixes the problem.
I suggest that the program do an automatic refresh when the mailmerge dialog is loaded as most users might expect the counts to be correct at this point.
Comment 12 Alex 2013-05-01 18:16:51 UTC
Just tried the above test with 301 records and got 331 labels in output.  Just as before, the sequence stops at record 42 and restarts the printing with record 13.
Comment 13 Björn Michaelsen 2014-10-10 01:01:37 UTC
Mailmerge has recently been refactored heavily on master, could you please test a recent daily build, details see:

 https://wiki.documentfoundation.org/QA/Testing_Daily_Builds#Where_can_I_get_Daily_Builds.3F
Comment 14 Alex Thurgood 2015-01-03 17:40:21 UTC Comment hidden (no-value)
Comment 15 QA Administrators 2016-01-17 20:02:45 UTC Comment hidden (obsolete)
Comment 16 Rudolf Kollien 2016-01-20 11:30:54 UTC
Will test as soon as possible. Because of holidays, it will delay till end of January.
Comment 17 Timur 2017-07-19 13:33:49 UTC
@Rudolf, @Alex: please retest with current LO versions.
Comment 18 Rudolf Kollien 2017-08-02 10:16:00 UTC
Tested with current version of LO on Xubuntu. Now Mail Merge works as expected even with big csv databases. As of me, the bug can be closed.
Comment 19 Timur 2017-08-02 10:49:51 UTC
Thank you. Than it's WFM. 
What one might experience with CSV is Bug 108825 on Linux and Bug 109219 on Windows.