Bug 40713 - FILESAVE problem dBASE Format M
Summary: FILESAVE problem dBASE Format M
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
3.3.3 release
Hardware: x86 (IA32) Linux (All)
: medium normal
Assignee: Julien Nabet
URL:
Whiteboard: target:6.0.0 target:5.4.0.2 target:5.3.5
Keywords:
Depends on:
Blocks:
 
Reported: 2011-09-08 04:36 UTC by linux
Modified: 2017-06-28 01:54 UTC (History)
4 users (show)

See Also:
Crash report or crash signature:


Attachments
dBase DBF files with memo fields.zip (895 bytes, application/zip)
2014-10-26 21:14 UTC, Chris Severance
Details

Note You need to log in before you can comment on or make changes to this bug.
Description linux 2011-09-08 04:36:33 UTC
If you open a *.dbf document with libreoffice calc, it asks you which char-set to choose. In my case "Westeuropa (DOS/OS2-850/international)" is ok and calc opens the file without problems. 

Some columns have the dbf-format "M" (memo) so that more information can be stored as compared to "C,254". This information is lost when you save the file as *.ods. In the new file the columns of the M-format are empty, not even truncated. Please fix the export filter. Thank you.

Tom
Comment 1 Alex Thurgood 2011-09-08 05:37:44 UTC
Tom,

As not everyone has a DBF capable file creator to hand, please provide us with a sample file, so that we can test and try to reproduce.

Thanks,

Alex
Comment 2 Alex Thurgood 2011-09-08 05:40:32 UTC
Also please indicate which (kinds of) information is lost. My understanding of MEMO fields for DBF III (at least, I don't know about DBF IV) was that the data of a MEMO field was actually stored in a separate text file having a DBT extension.


Alex
Comment 3 Alex Thurgood 2011-09-08 05:42:49 UTC
Need more info from poster, and this is more of a feature request than a bug. The behaviour of the DBF export filter has been the same for years, even back to the days of StarOffice. That doesn't mean that it can't be improved, but unless it can be shown to have less functionality in 3.3.3 than in 3.3.2 or 3.3.1 (or prior versions of OOo), say, then it can only ever be a request for enhancement (RFE).


Alex
Comment 4 linux 2011-09-13 06:13:44 UTC
Here are some additional informations regarding import/export of dbf-files into libreoffice-base

1. I opened an existing dbf-file with libreoffice-calc that includes the M memo format and german Umlaute (äüß etc.). During the dBase-import process you are asked the propper code which I selected as "Westeuropa (DOS/OS2-850/international)". The resulting ods-file is ok. It shows the german Umlaute correctly and the M memo-format is ok. It shows all the different Memo-information in the relevant cells.

2. When you save this file as a dbase-file I observed the following in an open Dolphin Window: DURING THE SAVING PROCESS A DBT-FILE IS GENERATED AND ITS SIZE GROWS. AS SOON AS THE SAVING PROCESS IS FINISHED IT DISAPPEARS AND ONLY A DBF-FILE IS STORED. 

3. You need the information stored in the DBT-File in order to show the Information in the M Memo-Format.

I would be very happy if you can fix that bug so that the dbt-file is kept along with the dbf-file. I suppose that this information will help to solve the problem.
Comment 5 Alex Thurgood 2011-09-13 06:58:58 UTC
Hi Tom,


Thanks for the extra information, this is certainly a hint to something that goes wrong somewhere. However, I still have a question :

Could you save from ODS to DBF having memo field types with previous versions of OOo / LibO ? I seem to recall that this was possible with OOo 1.1.x, and possibly 2.x.x versions, but I can't remember if that functionality was lost in later versions.

A sample DBF file with the Memo field that causes the problem in it would be helpful.


Alex
Comment 6 linux 2011-10-10 23:30:47 UTC
bugzilla-daemon@freedesktop.org schrieb:
> https://bugs.freedesktop.org/show_bug.cgi?id=40713
>
> --- Comment #5 from Alex Thurgood <alex.thurgood@gmail.com> 2011-09-13 06:58:58 PDT ---
> Hi Tom,
>
>
> Thanks for the extra information, this is certainly a hint to something that
> goes wrong somewhere. However, I still have a question :
>
> Could you save from ODS to DBF having memo field types with previous versions
> of OOo / LibO ? I seem to recall that this was possible with OOo 1.1.x, and
> possibly 2.x.x versions, but I can't remember if that functionality was lost in
> later versions.
>
> A sample DBF file with the Memo field that causes the problem in it would be
> helpful.
>
>
> Alex
>
>   
Hi Alex,
similar problems appeared in OOo in all versions I used before. I only 
discovered by accident, that libreoffice started to save a dbt file on 
disk because my database is some MB large. It was shown in dolphin and 
the size of the dbt file was growing consistently. When the storage of 
the dbf-file was completed, the dbt file was deleted. This should not 
happen.

I found a *workaround*: When you open a new libreoffice calc file and 
define the colomns in line 1 as DBF-Format with at least one M format 
and one set of data in line 2 and then save the file as dbf instead of 
ods a dbt file is saved and you can add datasets in line 3,4 ... 
Whenever you save this file as dbf it updates the dbt file.

You can work with base and link to the dbf file and save as odb. Then 
the dbt file works and is updated. I did this in order to work with my 
database in dbf format. Working in odb format is extremely slow perhaps 
due to java problems. I still prefere dbf and dbt instead.

But when you try to open the dbf file with calc the again, the memo 
fields are empty ??!

Thanx again
Tom
Comment 7 Björn Michaelsen 2011-12-23 12:37:11 UTC Comment hidden (obsolete)
Comment 8 Björn Michaelsen 2011-12-23 17:00:07 UTC Comment hidden (obsolete)
Comment 9 Florian Reisinger 2012-08-14 14:03:18 UTC Comment hidden (obsolete)
Comment 10 Florian Reisinger 2012-08-14 14:04:13 UTC Comment hidden (obsolete)
Comment 11 Florian Reisinger 2012-08-14 14:08:45 UTC Comment hidden (obsolete)
Comment 12 Florian Reisinger 2012-08-14 14:10:47 UTC Comment hidden (obsolete)
Comment 13 Commit Notification 2013-06-12 09:38:08 UTC Comment hidden (off-topic)
Comment 14 Eike Rathke 2013-06-12 09:53:03 UTC Comment hidden (off-topic)
Comment 15 Chris Severance 2014-10-26 21:14:01 UTC
Created attachment 108475 [details]
dBase DBF files with memo fields.zip

Bug: LO-Calc save as DBF fails to create a memo file when columns contain a memo field. The DBF file is unusable because it contains memo references to a memo file that doesn't exist.

Language is not relevant to this bug. Use CP850. This bug applies to all known versions of LO-Calc up to and including 4.3.2.2

Included are two pairs of test files, one created in dBase III and one created in Foxpro. LO-Calc imports both correctly, correctly detecting whether the memo is the dBase DBT or the Foxpro FPT. This means that code to read memo fields exists within LO-Calc.

The bug lies in LO-Calc save as DBF. To test that you do not need the test files though they do serve as a handy reference. Create a new spreadsheet and paste this in using semicolon (;) as the column delimiter.

MESSAGE,C,10;TEXT,M
"MESSAGE";"Some memo text!"
"MESSAGE2";"More memo text!"

Save as DBF: test.dbf

Expected result: TEST.DBF, TEST.DBT or TEST.FPT
Actual result: TEST.DBF

The first thing we notice is that LO-Calc does not ask us what memo format we want. Foxpro can read DBT but dBase3 cannot read FPT. It is acceptable to always write a DBT but a better implementation would ask which format is desired or have a "Settings" option that specifies which to use. Note that there are other memo implementations not available to me.

The next thing we note is that a 140 byte DBF file is created but no DBT or FPT is created. The memo text has been lost on save. We can use Linux hexdump to see what's inside the dbf.

http://www.dbase.com/KnowledgeBase/int/db7_file_fmt.htm

hexdump -C test.dbf
00000000  83 0e 0a 1a 02 00 00 00  61 00 15 00 00 00 00 00  |........a.......|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  4d 45 53 53 41 47 45 00  00 00 00 43 00 00 00 00  |MESSAGE....C....|
00000030  0a 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  54 45 58 54 00 00 00 00  00 00 00 4d 00 00 00 00  |TEXT.......M....|
00000050  0a 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  0d 20 4d 45 53 53 41 47  45 20 20 20 30 30 30 30  |. MESSAGE   0000|
00000070  30 30 30 30 30 31 20 4d  45 53 53 41 47 45 32 20  |000001 MESSAGE2 |
00000080  20 30 30 30 30 30 30 30  30 30 32 1a              | 0000000002.|

If we view the DBF in a hex viewer we see that the first byte is 0x83. The 0x80 signifies that the DBF has standard memo fields. We also see on the right that the field "TEXT" is marked with an M for memo. We also see that the file pointers stored in the DBF data for the memo texts are 0000000001 and 0000000002 which are likely right. It seems that all of the code necessary for producing a DBF and it's memo file is present in LO-Calc.

LO-Calc just doesn't create the memo file. LO-Calc can read the memo files but doesn't create them. Attempts to open the DBF with a missing memo file fail with errors.

Foxpro: MEMO file is missing/invalid.
dBase3: .DBT file cannot be opened.
Comment 16 Joel Madero 2014-11-05 03:19:02 UTC
I don't see evidence that this bug was independently confirmed by QA team - moving to UNCONFIRMED. Thanks all for your understanding.
Comment 17 Urmas 2014-11-06 03:51:37 UTC
That is strange: apparently DBT file is saved only in case the user changes the default codepage the DBF filter offers.
Comment 18 Alex Thurgood 2015-01-03 17:40:48 UTC Comment hidden (no-value)
Comment 19 QA Administrators 2016-01-17 20:04:01 UTC Comment hidden (obsolete)
Comment 20 Chris Severance 2016-01-18 12:44:15 UTC
LO Calc Linux 5.0.4.2

Bug is still present. Someone claimed that picking a non default language would make the .dbt file appear. I can't reproduce that. What I did find is that if save as once, then save as again, yes to overwrite, the .dbt file appears.
Comment 21 QA Administrators 2017-03-06 14:19:36 UTC Comment hidden (obsolete)
Comment 22 Chris Severance 2017-03-06 16:24:54 UTC
LO Calc Linux 5.3.0

Bug continues unabated.
Comment 23 Julien Nabet 2017-06-18 20:21:59 UTC
On pc Debian x86-64 with master sources updated some days ago, I could reproduce this.
Comment 24 Julien Nabet 2017-06-23 22:16:48 UTC
patch on review for master here:
https://gerrit.libreoffice.org/#/c/39189/

I gave a try with:
>MESSAGE,C,10;TEXT,M
>"MESSAGE";"Some memo text!"
>"MESSAGE2";"More memo text!"
>Save as DBF: test.dbf
from comment 15
Comment 25 Commit Notification 2017-06-24 07:22:50 UTC
Julien Nabet committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=36d91a65ab2db0c4c81e09771f6b44e1905122a0

tdf#40713: dBASE, don't lose dbt file

It will be available in 6.0.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 26 Commit Notification 2017-06-24 13:10:25 UTC
Julien Nabet committed a patch related to this issue.
It has been pushed to "libreoffice-5-4":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=a8dc650eaae7d35b944b62560056bda0423d3f96&h=libreoffice-5-4

tdf#40713: dBASE, don't lose dbt file

It will be available in 5.4.0.2.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 27 Commit Notification 2017-06-24 17:51:40 UTC
Julien Nabet committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=ee60c441dc73100b5041f4d0fb1675fade00d82c

Related tdf#40713, use DecodeMechanism::Unambiguous

It will be available in 6.0.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 28 Commit Notification 2017-06-24 17:51:58 UTC
Julien Nabet committed a patch related to this issue.
It has been pushed to "libreoffice-5-3":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e9cb227d3fd65ea46ffb926f2babb1926e5f2133&h=libreoffice-5-3

tdf#40713: dBASE, don't lose dbt file

It will be available in 5.3.5.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 29 Commit Notification 2017-06-24 17:53:13 UTC
Julien Nabet committed a patch related to this issue.
It has been pushed to "libreoffice-5-4":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=611299a16f9b21914d37788b700af0e0ba9a5aa0&h=libreoffice-5-4

Related tdf#40713, use DecodeMechanism::Unambiguous

It will be available in 5.4.0.2.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 30 Julien Nabet 2017-06-24 18:16:39 UTC
I put this one to FIXED but I'm interested in feedback here.
Indeed, I just fixed the case from comment 24/comment 15.

Just for information, I don't have dBASE, it'd be interesting to have some files with different xBASE versions.
Indeed it's quite a mess to find real versions,
see
- https://www.clicketyclick.dk/databases/xbase/format/dbf.html#DBF_NOTE_1_TARGET
- https://msdn.microsoft.com/en-us/library/aa975386(v=vs.71).aspx
Comment 31 Chris Severance 2017-06-27 18:32:48 UTC
The patch is working in 5.3.4-2.
Comment 32 Julien Nabet 2017-06-27 19:28:17 UTC
(In reply to Chris Severance from comment #31)
> The patch is working in 5.3.4-2.
Are you sure about the version? The patch isn't on 5.3.4 branch (or I missed something)
Comment 33 Chris Severance 2017-06-28 01:54:49 UTC
It will be a short time before the patched version comes through the repos so I applied the patch manually and compiled. Some patches look different so I picked the patch for the closest one: 5.3.5.