Bug 68790

Summary: Other: Build fails on Windows when system locale is "Japanese(Japan)" .
Product: LibreOffice Reporter: Tomofumi Yagi <yagit>
Component: LibreOfficeAssignee: Not Assigned <libreoffice-bugs>
Status: RESOLVED FIXED    
Severity: normal CC: ikuya
Priority: medium    
Version: 4.2.0.0.alpha0+ Master   
Hardware: Other   
OS: Windows (All)   
See Also: https://bugs.freedesktop.org/show_bug.cgi?id=66395
https://bugs.freedesktop.org/show_bug.cgi?id=66246
Whiteboard: BSA target:4.2.0
Crash report or crash signature: Regression By:
Attachments: Full error message
AutoControls.hrc
Patch for adding BOM to .hrc file
Patch for escaping UTF-8 chars

Description Tomofumi Yagi 2013-08-31 13:20:20 UTC
Created attachment 84969 [details]
Full error message

Problem description: 

When building LibreOffice 4.2.0.0.aplha0+(master), build fails in dbaccess module.

I'm using Windows 7 Ultimate and Visual Studio 2012 Express for Windows Desktop.

When using "--with-lang=en-US ja" autogen switch, build fails.
When using "--with-lang=ALL" switch, build fails,too.
But when not using "--with-lang=" switch, build succeeds.

Please refer Bug 66395 - Compilation error in Windows with UTF-8 unfriendly codepage( https://bugs.freedesktop.org/show_bug.cgi?id=66395 )


Steps to reproduce:

1. Open "Control Panel" -> "Regional and Language Options"
2. Select "Administrative" tab
3. Click "Change system locale".
4. Select "Japanese(Japan)" (Japanese version Windows has this value as default).
5. Click "OK" and restart computer.
6. Start to build libreoffice.
 
Build will fail with following message.
-----------------------------------------------------------------------
$ /opt/lo/bin/make 
/opt/lo/bin/make -j 8 -rs -f C:/lomas/core/Makefile.gbuild
[build DEP] LNK:Library/idbu.lib
[build ALM] animations
[build ALM] basebmp
[build ALM] basegfx
[build ALM] binaryurp
[build ALM] cli_ure
[build ALM] clucene
[build ALM] comphelper
[build ALM] configmgr
[build RDB] postgresql-sdbc
[build ALM] cppu
[build ALM] cppuhelper
[build CXX] dbaccess/source/ui/dlg/ConnectionHelper.cxx
[build CXX] dbaccess/source/ui/dlg/ConnectionPage.cxx
[build CXX] dbaccess/source/ui/dlg/ConnectionPageSetup.cxx
[build CXX] dbaccess/source/ui/dlg/DBSetupConnectionPages.cxx
[build CXX] dbaccess/source/ui/dlg/detailpages.cxx
[build CXX] dbaccess/source/ui/dlg/directsql.cxx
[build CXX] dbaccess/source/ui/dlg/dlgattr.cxx
[build CXX] dbaccess/source/ui/dlg/dlgsave.cxx
[build CXX] dbaccess/source/ui/dlg/dlgsize.cxx
(snip)
Note: including file:  C:\lomas\core\dbaccess\source\ui\inc\commontypes.hxx
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc : warning C4819: The file contains a character that cannot be represented in the current code page (932). Save the file in Unicode format to prevent data loss
Note: including file: C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc
Note: including file:  C:\lomas\core\dbaccess\inc\dbaccess_helpid.hrc
Note: including file:  C:\lomas\core\dbaccess\source\ui\inc\dbu_dlg.hrc
Note: including file:  C:\lomas\core\dbaccess\inc\dbaccess_helpid.hrc
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc(159) : error C2001: newline in constant
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc(370) : warning C4819: The file contains a character that cannot be represented in the current code page (932). Save the file in Unicode format to prevent data loss
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc(456) : error C2162: expected macro formal parameter
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc(460) : error C2162: expected macro formal parameter
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc(461) : error C2162: expected macro formal parameter
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc(468) : error C2162: expected macro formal parameter
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc(469) : error C2001: newline in constant
Note: including file: C:\lomas\core\include\connectivity/CommonTools.hxx
Note: including file: c:\lomas\core\dbaccess\source\ui\dlg\DriverSettings.hxx
Note: including file:  C:\lomas\core\dbaccess\source\inc\dsntypes.hxx
Note: including file:  C:\lomas\core\include\svl/poolitem.hxx
Note: including file: C:\lomas\core\dbaccess\source\ui\inc\dbadmin.hxx
Note: including file:  C:\lomas\core\include\sfx2/tabdlg.hxx
Note: including file:  C:\lomas\core\dbaccess\source\inc\dsntypes.hxx
Note: including file:  c:\lomas\core\dbaccess\source\ui\inc\IItemSetHelper.hxx
Note: including file:   C:\lomas\core\include\sal/types.h
Note: including file:   C:\lomas\core\include\comphelper/uno3.hxx
Note: including file:   C:\lomas\core\dbaccess\source\inc\dsntypes.hxx
Note: including file:  C:\lomas\core\include\comphelper/uno3.hxx
Note: including file: C:\lomas\core\include\comphelper/types.hxx
make[1]: *** [C:/lomas/core/workdir/wntmsci14/CxxObject/dbaccess/source/ui/dlg/ConnectionPage.o] Error 2
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc : warning C4819: The file contains a character that cannot be represented in the current code page (932). Save the file in Unicode format to prevent data loss
Note: including file: C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc
Note: including file:  C:\lomas\core\dbaccess\inc\dbaccess_helpid.hrc
Note: including file:  C:\lomas\core\dbaccess\source\ui\inc\dbu_dlg.hrc
Note: including file:  C:\lomas\core\dbaccess\inc\dbaccess_helpid.hrc
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc(159) : error C2001: newline in constant
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc(370) : warning C4819: The file contains a character that cannot be represented in the current code page (932). Save the file in Unicode format to prevent data loss
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc(456) : error C2162: expected macro formal parameter
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc(460) : error C2162: expected macro formal parameter
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc(461) : error C2162: expected macro formal parameter
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc(468) : error C2162: expected macro formal parameter
C:\lomas\core\workdir\wntmsci14\SrsTemplatePartTarget\dbaccess\AutoControls.hrc(469) : error C2001: newline in constant
make[1]: *** [C:/lomas/core/workdir/wntmsci14/CxxObject/dbaccess/source/ui/dlg/detailpages.o] Error 2
make: *** [build] Error 2
-----------------------------------------------------------------------
              
Operating System: Windows (other)
Version: 4.2.0.0.alpha0+ Master
Comment 1 Tomofumi Yagi 2013-08-31 13:24:33 UTC
Created attachment 84970 [details]
AutoControls.hrc
Comment 2 Tomofumi Yagi 2013-08-31 14:32:22 UTC
Created attachment 84972 [details]
Patch for adding BOM to .hrc file

This problem is that MSVC interprets UTF-8 source code without BOM as local codepage when system locale is "Japanese(Japan)".
I attach a workaound patch in order to clarify the cause.
This patch tweaks transex3.
Tweaked transex3 outputs a *.hrc file with BOM.  
*.hrc files with BOM avoid this problem.
Comment 3 Tomofumi Yagi 2013-09-06 13:22:04 UTC
Created attachment 85353 [details]
Patch for escaping UTF-8 chars

I attached a patch for escaping UTF-8 chars in .hrc file.

After all,I think that *.hrc file with BOM is better.
 
Escaping UTF-8 chars method has two disadvantages.

One is that Converted strings are very long.
This often occurs preprocessor error(the input line is too long).

Two is that it makes code unreadable.

I show an example.

1) Text[ en-US ] = "~Host name";
2) Text[ ja ] = "ใƒ›ใ‚นใƒˆๅ(~H)"; // String of Japanese(UTF-8)
3) Text[ ja ] = "\xe3\x83\x9b\xe3\x82\xb9\xe3\x83\x88\xe5\x90\x8d(~H)"; // Escaped string of Japanese(UTF-8)
Comment 4 Commit Notification 2013-09-14 16:52:37 UTC
Tomofumi Yagi committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=fff70bf98c7a5a63aa0db11e93a3512c6a9a9359

fdo#68790: fix build error on Win when system locale=="Japanese(Japan)"



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 5 Tomofumi Yagi 2013-09-20 12:18:27 UTC
Thank you so much for evaluation and merging of the patch.