Created attachment 134291 [details] encoding mismatched of LO5.4rc1 There may be some problem on importing DBF file with Thai (TIS-620) encoding into LibreOffice 5.4 rc1 Calc. The effect haven't occured before. My laptop :Lenovo T430 OS : Lubuntu 14.04 Libreoffice version: Version: 5.4.0.1 Build ID: 962a9c4e2f56d1dbdd354b1becda28edd471f4f2 CPU threads: 4; OS: Linux 3.13; UI render: default; VCL: gtk2; Locale: en-US (en_US.UTF-8); Calc: group I also attached 2 files from different LO version to show the comparision. The correct one is from LibreOffice 5.3.4.2
Created attachment 134292 [details] correct result of LO5.3.4.2
Thank you for reporting the bug. To be certain the reported issue is not related to corruption in the user profile, could you please reset your Libreoffice profile ( https://wiki.documentfoundation.org/UserProfile ) and re-test? I have set the bug's status to 'NEEDINFO'. Please change it back to 'UNCONFIRMED' if the issue is still present
I'll give it a try, might be a regression from my patches about dbf. Would it be possible you attach the dbf file so I could give it a try? Without it, it could be far more difficult to debug this.
Created attachment 134302 [details] DBF file with items of TiS-620 Thai characters I attached here the file with more records contained the TIS-620 Thai characters for your investigation.
On pc Debian x86-64 with master sources updated yesterday, I could reproduce this. With LO 5.2.7 (package from Debian), I don't reproduce this. I confirm the bug + regression. hexdump shows this: 0000000 7503 1b06 2046 0000 00c1 0063 0000 0000 0000010 0000 0000 0000 0000 0000 0000 1b00 0000 1B isn't listed in encodings
Argh my fix for tdf#55631 is wrong. It seems except the case encoding can be read from the header of dbf file, that it'll always be RTL_TEXTENCODING_IBM_850 even if the user selects another encoding. :-( I'm gonna try to understand why the first patch https://cgit.freedesktop.org/libreoffice/core/commit/?id=9fe9685627c51926459a897594ead9f64deee579 was considered wrong (see https://gerrit.libreoffice.org/#/c/38627/)
Ok, 850 is transformed to "DONTKNOW" so LO tries to read encoding from header. In addition here, since 0x1b isn't listed in the switch, the header encoding can't be used even 850 is selected. I didn't find any website indicating 1b would correspond to "Thai ISO 8859-11/TIS 620". Viruch: could you run some hexdump <file>| head -10 to see if all dbf files containing TIS 620 encoding have this 1b at second line? (at the same exact location as the already quoted example: 0000000 7503 1b06 2046 0000 00c1 0063 0000 0000 0000010 0000 0000 0000 0000 0000 0000 1b00 0000 ) If yes, we could start to add it on a switch.
Attached here are the hexdump result of 6 different files with TIS-620 encoding inside. As I opened these files by "hexedit", it seems to be that beginning from the position of 0000020 of every files are the starting position of the "Field_Names" of each .DBF file. So some of them would look very indentical due to their similar "Filed_Name".
Created attachment 134332 [details] hexdump of DBF file that contains TIS-620 encoding
Created attachment 134333 [details] hexdump of DBF file that contains TIS-620 encoding
Created attachment 134334 [details] hexdump of DBF file that contains TIS-620 encoding
Created attachment 134335 [details] hexdump of DBF file that contains TIS-620 encoding
Created attachment 134336 [details] hexdump of DBF file that contains TIS-620 encoding
Created attachment 134337 [details] hexdump of DBF file that contains TIS-620 encoding
Lionel Elie Mamane committed a patch related to this issue. It has been pushed to "master": http://cgit.freedesktop.org/libreoffice/core/commit/?id=7f1465a9599e9665159dd2d823a6e9064cca5703 tdf#108789 and others: overhaul DBase files encoding handling It will be available in 6.0.0. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Lionel Elie Mamane committed a patch related to this issue. It has been pushed to "libreoffice-5-4": http://cgit.freedesktop.org/libreoffice/core/commit/?id=6e0eafe576436ec229c6d90f654ff1b11ff9bdfd&h=libreoffice-5-4 tdf#108789: branch 5.4 only It will be available in 5.4.0.2. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
It seems to be that the bug was already solved in the latest daily build version as tested on Lubuntu 14.04 Version: 5.4.0.1.0+ Build ID: 6e0eafe576436ec229c6d90f654ff1b11ff9bdfd CPU threads: 4; OS: Linux 3.13; UI render: default; VCL: gtk2; TinderBox: Linux-rpm_deb-x86@71-TDF, Branch:libreoffice-5-4, Time: 2017-07-04_07:33:47 Locale: en-US (en_US.UTF-8); Calc: single
Created attachment 134485 [details] 2017-07-04 daily built test screen shot
(In reply to Viruch Hemapanpairo from comment #17) > It seems to be that the bug was already solved in the latest daily build > ... There's still https://gerrit.libreoffice.org/#/c/39449/ for 5.3 branch but let's put this one to FIXED.
Julien Nabet committed a patch related to this issue. It has been pushed to "libreoffice-5-3": http://cgit.freedesktop.org/libreoffice/core/commit/?id=91bafeff8b8a195d9ecc242c0def413361161d79&h=libreoffice-5-3 tdf#108789 quick fix for 5.3 branch only It will be available in 5.3.5. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Eike Rathke committed a patch related to this issue. It has been pushed to "master": http://cgit.freedesktop.org/libreoffice/core/commit/?id=22ae038a56b85e86219922c2759544545f2d813d Fix crash when saving new spreadsheet as dBase/.dbf, tdf#108789 follow-up It will be available in 6.0.0. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Eike Rathke committed a patch related to this issue. It has been pushed to "libreoffice-5-4": http://cgit.freedesktop.org/libreoffice/core/commit/?id=e059303c2dc6ecac5247c315f5b452d346512c12&h=libreoffice-5-4 Fix crash when saving new spreadsheet as dBase/.dbf, tdf#108789 follow-up It will be available in 5.4.1. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Eike Rathke committed a patch related to this issue. It has been pushed to "libreoffice-5-4-0": http://cgit.freedesktop.org/libreoffice/core/commit/?id=a9950f79867c4a7c2d4c542db754c5d24028fadf&h=libreoffice-5-4-0 Fix crash when saving new spreadsheet as dBase/.dbf, tdf#108789 follow-up It will be available in 5.4.0. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.