Created attachment 74058 [details]
text file to use as data source (in utf-8)

Created attachment 74059 [details]
database which feeds from the text file

Created attachment 74061 [details]
odt file with the first column of the database added as database field

Created attachment 74062 [details]
context.xml from inside of the testing file

if you issues a:
od -c testing_file_content.xml

you can see the raw content of the file and the BOM inside of it with the database field name:
0007100   x   t   :   c   o   l   u   m   n   -   n   a   m   e   =   "
0007120 357 273 277   C   o   l   u   m   n   1   "       t   e   x   t

Rainer - do you have any ideas about this one?

@Joel:
Only reading tells me nothing, and I have no time for puzzling around.

@Robert:
Can you see what the problem might be and how to make it reproducible for non-database-mailmerge-experts?

@pasqual milvaques:
Please attach such test kits zipped as 1 single attachment!
Thank you for your report – unfortunately important information is missing.
May be hints on <http://wiki.documentfoundation.org/BugReport> will help you to find out what information will be useful to reproduce your problem? 
Please add all information requested in following:
- Write a meaningful Summary describing exactly what the problem is
- Explain what a "BOM" is
- Contribute a document related step by step instruction containing every 
  key press and every mouse click how to reproduce your problem 
  (similar to example in Bug 43431), Here nobody has time to puzzle how to reproduce the problem with your documents.
– if possible contribute an instruction how to create a sample document 
  from the scratch
- add information 
  -- what EXACTLY is unexpected (in step by step instruction)
  -- and WHY do you believe it's unexpected (cite Help or Documentation!)
  -- concerning your PC (video card, ...)
  -- concerning your Operating System (Version, Distribution, Language)
  -- concerning your LibO version (with Build ID if it's not a public release)
     and localization (UI language, Locale setting)
  –- Libo settings that might be related to your problems 
  -- how you launch LibO and how you opened the sample document
  –- Whether your problem persists when you renamed your user profile 
     before you launch LibO (please see
     <https://wiki.documentfoundation.org/UserProfile#User_profile_location>)
  -- Whether that worked in more early versions
  -- everything else crossing your mind after you read linked texts

@Rainer,

isn't a special database-problem, but a problem for reading text-files with base. I have ha a look for BOM at wikipedia. It is a mark at the beginning of a text-file in utf-16 and utf-32-code; it seems to be optional in utf-8-code. And this is what I read in http://en.wikipedia.org/wiki/Byte_order_mark :
"Java does not support UTF-8 with BOM and does not intend to implement it in future releases."
Could be a problem described in this bug.

@Robert:
Thank you for research, sounds plausible. As soon as reporter contributet step by step instruction I will do further tests.

Created attachment 75215 [details]
testcase files including a procedure to reproduce the problem

In the tar.gz I include all the files needed to reproduce the problem and a text document called testcase_procedure.odt in which I try to detail the steps to reproduce the problem
I hope to answer in it all the questions but if you need more detail say it to me, please

thanks

I have been able to confirm the issue on:
Version 3.6.5.2 
Platform: Bodhi Linux 2.2 x64
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 
As I've been able to confirm this problem on an earlier release I am changing the version number as version is the earliest version that we can confirm the bug, we use comments to say that the bug exists in newer versions as well.

Marking as:

New (confirmed)
Normal - can prevent high quality work under certain circumstances
Low - unfortunately this one probably isn't affecting many users at all. Furthermore, there may be wacky workarounds such as bringing it into spreadsheet, saving it as a ods, then bringing into base ( I know not ideal, just a possible workaround for the time being.

Thanks so much for the clear and concise instructions.


+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 
LibreOffice is powered by a team of volunteers, every bug is confirmed (triaged) by human beings who mostly give their time for free. We invite you to join our triaging by checking out this link:
https://wiki.documentfoundation.org/QA/BugTriage

There are also other ways to get involved including with marketing, UX, documentation, and of course developing -  http://www.libreoffice.org/get-help/mailing-lists/. 

Lastly, good bug reports help tremendously in making the process go smoother, please always provide reproducible steps (even if it seems easy) and attach any and all relevant material

*** Bug 60336 has been marked as a duplicate of this bug. ***

I'm not sure whether this is a bug. Have you set the character set to "UTF-8" in Edit > Database > Properties? I can use the database with the text file for a mail merge without problem on Windows7. The field name has no strange characters.

regina, changing the encoding of the database in the way you have said in windows makes the utf8 characters legible but the bom is not ignored just in linux. that at the end can create problems in some special situations

checked in windows 8 with libreoffice 4.0.2

the problem reported in bug 60336 can't be addressed in this way also, the xml file indicates that it's in utf-8 but bom it's not ignored creating an erroneous behavior

I have observed that the option to choose the encoding is not present in the database wizard at creation time, it will be a nice improvement to have this option present in the wizard as it's a bit hard to find it(I didn't notice that it existed) and requires a reopening of the database to be applied

thanks

On pc Debian x86-64 with master sources updated today, I could reproduce this.

I noticed these 2 commits about BOM:
- http://cgit.freedesktop.org/libreoffice/core/commit/?id=f38277dc0337df15f3ea689096a2c18a03354a61
- http://cgit.freedesktop.org/libreoffice/core/commit/?id=5eb408a3bb8df204452f0b931a254dad5f0cf35b

Then, we need a code pointer to know where to start for this case.

Ok found a start to dig, see:
http://opengrok.libreoffice.org/xref/core/sw/source/uibase/fldui/fldmgr.cxx#1010

Unwinding on gdb, I found different places to put a fix:
- http://opengrok.libreoffice.org/xref/core/sw/source/ui/fldui/flddb.cxx#187
- http://opengrok.libreoffice.org/xref/core/sw/source/uibase/dbui/dbtree.cxx#420
still digging brings to svtools module
- http://opengrok.libreoffice.org/xref/core/svtools/source/contnr/treelist.cxx#464
but wonder if this last one is really good option.

About the fix in itself:
I thought about using/copying (?) http://opengrok.libreoffice.org/xref/core/l10ntools/source/lngmerge.cxx#lcl_RemoveUTF8ByteOrderMarker method

Finally, even if we fix this one, should we consider too other encodings UTF-16 and UTF-32 (considering too Little/Big Endian part), see http://en.wikipedia.org/wiki/Byte_order_mark

Lionel: any thoughts?

I have verified that the problem is still present in LibreOffice 5.1.0.2, tested in Windows 10 (32 bits). There has not been any behaviour change in the bug

2 things which might help:
- tdf#63673 which has been fixed in 5.4.0
- tdf#44291 which has just been fixed in master sources.

Any update here?

Bug is still present in 
Version: 7.1.1.2 / LibreOffice Community
Build ID: fe0b08f4af1bacafe4c7ecc87ce55bb426164676
CPU threads: 8; OS: Linux 5.10; UI render: default; VCL: kf5
Locale: cs-CZ (cs_CZ.UTF-8); UI: en-US
Calc: threaded

I am on Manjaro KDE

Dear pasqual milvaques,

To make sure we're focusing on the bugs that affect our users today, LibreOffice QA is asking bug reporters and confirmers to retest open, confirmed bugs which have not been touched for over a year.

There have been thousands of bug fixes and commits since anyone checked on this bug report. During that time, it's possible that the bug has been fixed, or the details of the problem have changed. We'd really appreciate your help in getting confirmation that the bug is still present.

If you have time, please do the following:

Test to see if the bug is still present with the latest version of LibreOffice from https://www.libreoffice.org/download/

If the bug is present, please leave a comment that includes the information from Help - About LibreOffice.
 
If the bug is NOT present, please set the bug's Status field to RESOLVED-WORKSFORME and leave a comment that includes the information from Help - About LibreOffice.

Please DO NOT

Update the version field
Reply via email (please reply directly on the bug tracker)
Set the bug's Status field to RESOLVED - FIXED (this status has a particular meaning that is not 
appropriate in this case)


If you want to do more to help you can test to see if your issue is a REGRESSION. To do so:
1. Download and install oldest version of LibreOffice (usually 3.3 unless your bug pertains to a feature added after 3.3) from https://downloadarchive.documentfoundation.org/libreoffice/old/

2. Test your bug
3. Leave a comment with your results.
4a. If the bug was present with 3.3 - set version to 'inherited from OOo';
4b. If the bug was not present in 3.3 - add 'regression' to keyword


Feel free to come ask questions or to say hello in our QA chat: https://web.libera.chat/?settings=#libreoffice-qa

Thank you for helping us make LibreOffice even better for everyone!

Warm Regards,
QA Team

MassPing-UntouchedBug