Bug 55631 - FILEOPEN: CALC does not import DBASE with CP850 encoding correctly
Summary: FILEOPEN: CALC does not import DBASE with CP850 encoding correctly
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
3.5.0 RC1
Hardware: Other All
: medium normal
Assignee: Julien Nabet
URL:
Whiteboard: target:6.0.0 target:5.4.0.1 target:5.3.5
Keywords: bibisected, bisected, regression
Depends on:
Blocks:
 
Reported: 2012-10-04 16:09 UTC by tbb
Modified: 2017-06-27 18:41 UTC (History)
6 users (show)

See Also:
Crash report or crash signature:


Attachments
Testfile dBase. Try to import with LO-Calc - works up to LO 3.4.6, doesn't since 3.5 (496 bytes, application/x-dbf)
2013-02-03 19:54 UTC, Robert Großkopf
Details

Note You need to log in before you can comment on or make changes to this bug.
Description tbb 2012-10-04 16:09:39 UTC
German umlauts are not decoded correcly.

Cross checked with OOo 3.2.1: It works.
Comment 1 Robert Großkopf 2013-02-03 19:51:11 UTC
I could confirm this bug. If you try to import a dBase-file to Calc, the filter-Dialog appears. You could choose the filter but it doesn't work.
Have tried this with LO 3.4.6 and the first LO 3.5.0 rc1. The bug first appears with LO 3.5 - in LO 3.4.6 and all earlier versions the import works right.
I have set the version to the version where the bug appears first.
The bug isn't solved up to LO 4.0.0.1.
Comment 2 Robert Großkopf 2013-02-03 19:54:58 UTC
Created attachment 74151 [details]
Testfile dBase. Try to import with LO-Calc - works up to LO 3.4.6, doesn't since 3.5

The dialog for Import DBase-files shows "Western Europe(DOS/OS2-850/International)". This import-filter doesn't work in Calc since 3.5. In earlier versions it works.
Comment 3 Buovjaga 2014-10-22 05:20:15 UTC
Western Europe(DOS/OS2-437/US) imports them correctly. Win 7 64-bit 4.3.2.2.
Comment 4 Robert Großkopf 2014-10-22 06:14:02 UTC
(In reply to Beluga from comment #3)
> Western Europe(DOS/OS2-437/US) imports them correctly. Win 7 64-bit 4.3.2.2.

You are right. It is possible to import it this way. But the original code is IBM850. So the filter starts correctly with "DOS/OS2 850 international" but couldn't import it with this filter.
Opened the file with AOO and "DOS/OS2 850 international" - right coding.
Opened the file with KWrite (Linux/KDE) and changed to IBM850 - right coding.
It also works right in the first LO-versions with "DOS/OS2 850 international".

There seems to be something wrong with the filter. LO recognizes the right filter automatically as OOo and AOO does, but could be the filter isn't linked internal the right way.
Comment 5 Urmas 2014-10-22 09:33:26 UTC
It's interesting: that encoding works fine on the files with (V)FP codepage mark.
Comment 6 Matthew Francis 2014-12-04 02:10:49 UTC
In bibisect-43all:
 d202b17d88ecb0b608d81518624021c832c7dfdb is the first bad commit
commit d202b17d88ecb0b608d81518624021c832c7dfdb
Author: Bjoern Michaelsen <bjoern.michaelsen@canonical.com>
Date:   Wed Apr 25 07:28:24 2012 +0200

    source-hash-ce97851773a06103504972eb2771eecd7dd81e36
    
    commit ce97851773a06103504972eb2771eecd7dd81e36
    Author:     David Tardon <dtardon@redhat.com>
    AuthorDate: Mon Feb 6 19:12:02 2012 +0100
    Commit:     David Tardon <dtardon@redhat.com>
    CommitDate: Mon Feb 6 19:12:02 2012 +0100
    
        fix typo

:100644 100644 e1c1d62aa980fee004430f920cdbe3fd1ce79bf0 9acf11b8f6f5e26b03649767813ac42f72c38e1b M	autogen.log
:100644 100644 c14237a7b6ebde67a83585c9b057c78710e08ea2 db4232175b715b6c7f322b17041f56a9145e1622 M	ccache.log
:100644 100644 c407d12366338584cbcebf2197cd7fcdcf1c522b 1b83a94159f8aa22e004b5dc2ebe1895b32a2724 M	commitmsg
:100644 100644 3be616510b5296b5ae2f5c154a6c51f7ba49bf24 cc9f341a09ba536bb41d4219c5b7f5dd219d7cc6 M	dev-install.log
:100644 100644 637e789a93608b99c13fec9e598c2e7a4c454c6d 08ab33c46c34b7b9b0f8b7f21161ad1c1a2ed59a M	make.log
:040000 040000 c47ba9e6977c3c8a957b11ec3f8b85cfa50362af f87831ea583aaccb888e681ce264cc1e4e44d3aa M	opt

# bad: [423a84c4f7068853974887d98442bc2a2d0cc91b] source-hash-c15927f20d4727c3b8de68497b6949e72f9e6e9e
# good: [65fd30f5cb4cdd37995a33420ed8273c0a29bf00] source-hash-d6cde02dbce8c28c6af836e2dc1120f8a6ef9932
git bisect start 'latest' 'oldest'
# bad: [e02439a3d6297a1f5334fa558ddec5ef4212c574] source-hash-6b8393474974d2af7a2cb3c47b3d5c081b550bdb
git bisect bad e02439a3d6297a1f5334fa558ddec5ef4212c574
# bad: [8f4aeaad2f65d656328a451154142bb82efa4327] source-hash-1885266f274575327cdeee9852945a3e91f32f15
git bisect bad 8f4aeaad2f65d656328a451154142bb82efa4327
# bad: [369369915d3582924b3d01c9b01167268ed38f3b] source-hash-45295f3cdceb4c289553791071b5d7f4962d2ec4
git bisect bad 369369915d3582924b3d01c9b01167268ed38f3b
# good: [351622aec2dff3cc3bbbb020ad0097c4322d2a21] source-hash-2c4537471c932b65e6f72e41881b505c4bbad12c
git bisect good 351622aec2dff3cc3bbbb020ad0097c4322d2a21
# bad: [378efb6e51212a05d1bd4b85c916eec5753c1744] source-hash-d453788ac0476cc02b929b0907718ca771d6d956
git bisect bad 378efb6e51212a05d1bd4b85c916eec5753c1744
# bad: [1a3c4b54a8782fe0f4bdba221e87012a92e4d323] source-hash-a330f38093e2643a26239557050561afae9ff23d
git bisect bad 1a3c4b54a8782fe0f4bdba221e87012a92e4d323
# good: [cf86b7f14a98d2d81a5cd93507acb35ff6775d8b] source-hash-85c6244b85b29c1d2bb9d89b62e9512dd65378b5
git bisect good cf86b7f14a98d2d81a5cd93507acb35ff6775d8b
# good: [bc87fae0fc661b44769d71e41a0e8ce3dac3e857] source-hash-f176c9ba7be7f3051a52b9f57b56124038c0cfd6
git bisect good bc87fae0fc661b44769d71e41a0e8ce3dac3e857
# bad: [d202b17d88ecb0b608d81518624021c832c7dfdb] source-hash-ce97851773a06103504972eb2771eecd7dd81e36
git bisect bad d202b17d88ecb0b608d81518624021c832c7dfdb
# good: [9300cbe83880d09cc6d581eb73a92f35f3456b31] source-hash-43c7830b03d141ae11d8617c0fdabefa32dd243c
git bisect good 9300cbe83880d09cc6d581eb73a92f35f3456b31
# first bad commit: [d202b17d88ecb0b608d81518624021c832c7dfdb] source-hash-ce97851773a06103504972eb2771eecd7dd81e36
Comment 7 Robinson Tryon (qubit) 2015-12-13 11:09:41 UTC Comment hidden (obsolete)
Comment 8 Julien Nabet 2016-03-07 23:29:29 UTC
On pc Debian x86-64 with master sources updated yesterday, I could reproduce this.

Reverting this 
https://cgit.freedesktop.org/libreoffice/core/commit/?id=b00c4ec0967f8712d721b31ccb2dd0778c9e973b seems to fix the problem.

Muthu: thought you might be interested in this one.
Comment 9 Xisco Faulí 2016-09-14 15:06:48 UTC
Adding keyword 'bisected' as problematic commit was identified in comment 8
Comment 10 Julien Nabet 2017-06-09 23:24:42 UTC
Let's give it a try with https://gerrit.libreoffice.org/#/c/38620/
Comment 11 Commit Notification 2017-06-10 06:16:02 UTC
Julien Nabet committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=9fe9685627c51926459a897594ead9f64deee579

tdf#55631: DBASE import with CP850 encoding

It will be available in 6.0.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 12 Commit Notification 2017-06-11 07:33:35 UTC
Julien Nabet committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=1cb3100db964be082ce2eb86a20a38193ad51fb1

Revert "tdf#55631: DBASE import with CP850 encoding"

It will be available in 6.0.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 13 Julien Nabet 2017-06-11 07:48:34 UTC
Just for the information, I reverted the initial patch and proposed a new one (see https://gerrit.libreoffice.org/#/c/38659/)
Comment 14 Commit Notification 2017-06-12 17:16:38 UTC
Julien Nabet committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=58bb9da339d61981d45cde8e38a2ce1ae33652c4

tdf#55631: DBASE import (take 2)

It will be available in 6.0.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 15 Commit Notification 2017-06-12 18:07:24 UTC
Julien Nabet committed a patch related to this issue.
It has been pushed to "libreoffice-5-4":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=7bb142d830b86d7d2b84ede2f236312e76956175&h=libreoffice-5-4

tdf#55631: DBASE import (take 2)

It will be available in 5.4.0.1.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 16 Julien Nabet 2017-06-12 20:49:05 UTC
patch for 5.3 branch in review (see https://gerrit.libreoffice.org/#/c/38707/)
Comment 17 Buovjaga 2017-06-16 13:02:05 UTC
Yes, now it offers DOS/OS2 850 international and German characters are OK.

Arch Linux 64-bit, KDE Plasma 5
Version: 6.0.0.0.alpha0+
Build ID: 5c81adc51a05a016e754de7961d3a7bdb4494e01
CPU threads: 8; OS: Linux 4.11; UI render: default; VCL: kde4; 
Locale: fi-FI (fi_FI.UTF-8); Calc: group
Built on June 16th 2017
Comment 18 Commit Notification 2017-06-24 17:51:49 UTC
Julien Nabet committed a patch related to this issue.
It has been pushed to "libreoffice-5-3":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=954931c86cf67876c8324acec31c3e95feeb6b4d&h=libreoffice-5-3

tdf#55631: DBASE import (take 2)

It will be available in 5.3.5.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 19 Julien Nabet 2017-06-24 18:17:03 UTC
Don't hesitate to test, I'm interested in feedback here.