Bug 100043 - Works Database: dates earlier than 1970-01-01 are not handled correctly
Summary: Works Database: dates earlier than 1970-01-01 are not handled correctly
Status: RESOLVED FIXED
Alias: None
Product: Document Liberation Project
Classification: Unclassified
Component: General (show other bugs)
Version:
(earliest affected)
unspecified
Hardware: x86-64 (AMD64) Windows (All)
: medium normal
Assignee: Not Assigned
URL:
Whiteboard: libwps
Keywords:
: 100044 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-05-25 08:35 UTC by Bernard Robins
Modified: 2018-03-08 11:34 UTC (History)
6 users (show)

See Also:
Crash report or crash signature:


Attachments
Records of UK Para killed 1944 and died since. (1.38 MB, application/vnd.oasis.opendocument.spreadsheet)
2016-05-25 15:43 UTC, Bernard Robins
Details
Copy of Works 6 Database that does not copy correctly (4.33 MB, application/x-ole-storage)
2016-05-26 05:40 UTC, Bernard Robins
Details
Dates testcase (5.00 KB, application/octet-stream)
2016-05-29 10:41 UTC, Urmas
Details
Long text and linebreaks testcase (6.50 KB, application/x-ole-storage)
2016-05-29 10:42 UTC, Urmas
Details
LO inport from Works 6 DB (1.38 MB, application/vnd.oasis.opendocument.spreadsheet)
2016-05-29 15:49 UTC, Bernard Robins
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Bernard Robins 2016-05-25 08:35:13 UTC
Importing a Microsoft Works 6 database into LO spreadsheet, some dates are not importing correctly. I have a lot of dates in 1944 and this is defaulting to 30 December 1899. (The 2 character date field is set to 1920).
Comment 1 Cor Nouws 2016-05-25 15:13:47 UTC

*** This bug has been marked as a duplicate of bug 100044 ***
Comment 2 Cor Nouws 2016-05-25 15:14:10 UTC
apologies!
Comment 3 Cor Nouws 2016-05-25 15:14:50 UTC
Can you please attach a test file, Bernard?
Comment 4 Bernard Robins 2016-05-25 15:43:57 UTC
Created attachment 125284 [details]
Records of UK Para killed 1944 and died since.
Comment 5 V Stuart Foote 2016-05-25 19:44:37 UTC
We'd need a sample of the original Works 6 database.
Comment 6 V Stuart Foote 2016-05-25 19:53:24 UTC
(In reply to Bernard Robins from comment #0)
Here is the related Works database import issue from bug 10044...
 
>Importing a Microsoft Works 6 database into LO spreadsheet, 
>some of the text from MW6 is not being fully imported, this 
>seems to happen if the field being imported contains more 
>than around 200 characters. After import the "word wrap" is
>not working correctly, displaying some boxes correctly and 
>others with a small red arrow pointing to the right (I can't 
>find any reference to what this means).
Comment 7 V Stuart Foote 2016-05-25 19:58:33 UTC
*** Bug 100044 has been marked as a duplicate of this bug. ***
Comment 8 Bernard Robins 2016-05-26 05:40:34 UTC
Created attachment 125288 [details]
Copy of Works 6 Database that does not copy correctly
Comment 9 Cor Nouws 2016-05-28 10:54:43 UTC
Hi Bernard,

I simply do File > Open and then your WDB (thanks for attaching).

(In reply to Bernard Robins from comment #0)
> I have a lot of dates in 1944 and this is
> defaulting to 30 December 1899. (The 2 character date field is set to 1920).

I do not see date 1889 after importing.

(In reply to Bernard Robins from comment 100044 / #0)
> some of the text
> from MW6 is not being fully imported, this seems to happen if the field
> being imported contains more than around 200 characters.

Can you point to some fields where that is the case?

> being imported contains more than around 200 characters. After import the
> "word wrap" is not working correctly, displaying some boxes correctly and

You have to set that yourself (and may want to use cell styles for this)

> others with a small red arrow pointing to the right (I can't find any
> reference to what this means).

that means that the cell is not wide enough to display the contents.

Maybe it's good to try to get help on a user forum too/first?
See some suggestions here
  http://www.libreoffice.org/get-help/community-support/
Comment 10 Urmas 2016-05-29 10:26:13 UTC
Tested with May 10 version:

All dates earlier than 1970-01-01 are corrupted and appear as 1899-12-30.
All text fields larger than 255 characters are cut at this value (Works Database stores the rest of the text in a record with id 0x36, which is apparently not supported).
Comment 11 Urmas 2016-05-29 10:41:53 UTC
Created attachment 125369 [details]
Dates testcase
Comment 12 Urmas 2016-05-29 10:42:31 UTC
Created attachment 125370 [details]
Long text and linebreaks testcase
Comment 13 Bernard Robins 2016-05-29 15:49:07 UTC
Created attachment 125377 [details]
LO inport from Works 6 DB

Works DB6 allows you to save a file as .cvs which seems to have solved the basic problem of dates and large amounts of text.

I have attached LO Calc of the updated file.

LO FORMATTING: It seems you have to go into "View - Toolbars - Text formatting" to get the text format bar to show. There were dots to the left of the bar, I right clicked and selected "lock" the toolbar. I expected this to then remain on the top of the page but it does not. If you select "print option" the text format bar is removed and you have to go through the above each time you select a different option and want to return to text formatting. The "lock toolbar" dots have now disappeared. BUG or ME ?

Some of the records in LO were showing a red arrow on the right side of the cell after "Wrap Text" had been selected. To format this correctly you have to select the column then go to "Format - Row - Optimal Height - Take out default value tick mark - Select OK". 
Could this be incorporated into the "format cells" option automatically when selecting the "Wrap Text" box ?

You will note in the attached file that some boxes still have the "red arrow", even after being formatted as above. This seems to be when the cell holds 3 or more lines of text and the bottom line is just one word. The only way I have found to get round this is to insert extra text into the cell to make the sentence longer.
Comment 14 David Tardon 2016-05-30 06:36:44 UTC
(In reply to Urmas from comment #12)
> Created attachment 125370 [details]
> Long text and linebreaks testcase

Could you open a separate bug for this?
Comment 15 osnola 2016-05-30 09:27:20 UTC
Hello, I test on OSX using a 32bit and a 64bit version of libwps and I can not reproduce the date's problem :-~ 

I suppose that the problem is that we use gmtime in WKSContentListener::CellContent::double2Date 
( https://sourceforge.net/p/libwps/code/ci/master/tree/src/lib/WKSContentListener.cpp ) and that on Windows, date before 1970 are not supported by gmtime ( for instance http://stackoverflow.com/questions/21869602/mongodb-date-issue-when-before-1970 ). As I do not have access on computer with Windows, if someone has some idea how to fix that on Windows, let me know....

Note:
- concerning the structure with id=0x36, I will look if I can parse it...
Comment 16 osnola 2016-05-30 10:29:39 UTC
(In reply to osnola from comment #15)
> ...
> I suppose that the problem is that we use gmtime in
> WKSContentListener::CellContent::double2Date 
> (
> https://sourceforge.net/p/libwps/code/ci/master/tree/src/lib/
> WKSContentListener.cpp ) and that on Windows, date before 1970 are not
> supported by gmtime ( for instance
> http://stackoverflow.com/questions/21869602/mongodb-date-issue-when-before-
> 1970 ). As I do not have access on computer with Windows, if someone has
> some idea how to fix that on Windows, let me know....
> 
I have added a patch to do the conversion by hand:
https://sourceforge.net/p/libwps/code/ci/818d9c4f527f08965e17ac9146286f3171622f5d/
can someone on Windows check if this solve the date's problem ?
Comment 17 osnola 2016-05-31 07:11:12 UTC
(In reply to osnola from comment #15)
>
> Note:
> - concerning the structure with id=0x36, I will look if I can parse it...

I add a new patch in libwps to try parsing these structures https://sourceforge.net/p/libwps/code/ci/ed353bb70c56fe1cb53a7dfe338200564d06e9dc/ ...
Comment 18 osnola 2016-06-01 07:01:47 UTC
(In reply to osnola from comment #17)
>
> I add a new patch in libwps to try parsing these structures
> https://sourceforge.net/p/libwps/code/ci/
> ed353bb70c56fe1cb53a7dfe338200564d06e9dc/ ...

Hello, 
as I am trying to compile a libwps version with emscripten: http://kripken.github.io/emscripten-site/ ,
you can try if these changes correct the seen problems in http://www.loria.fr/~alonso/tmp/wps2odt.html (a temporary link). Normally, if this "test page" works, you can choose a file to be converted and then open the result in LibreOffice or save it...

Notes:
- the interface must be improved a lot :-~
- I try to "save" the converted result with FileSaver.js: https://github.com/eligrey/FileSaver.js , so some browsers are not supported :-~
- if the file to be converted is very big as "0.\ PARA\ ALL.WDB", the conversion can take one or two minutes and as I have not implemented a progress bar, you may see a message saying that the script is not responding do you want to abort it/continue :-
Comment 19 David Tardon 2018-03-07 12:43:30 UTC
@Bernard, @Urmas: Could either of you verify this has indeed been fixed?
Comment 20 Xisco Faulí 2018-03-08 11:34:58 UTC
(In reply to David Tardon from comment #19)
> @Bernard, @Urmas: Could either of you verify this has indeed been fixed?

Since there's a commit fixing this issue, let's put it to RESOLVED FIXED.
@Bernard, @Urmas, change it back to NEW if the issue is still reproducible in master...