Bug 101077 - FILEOPEN: WK1 Lotus 1-2-3 filter gives numbers 100000000 times higher than they are
Summary: FILEOPEN: WK1 Lotus 1-2-3 filter gives numbers 100000000 times higher than th...
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: filters and storage (show other bugs)
Version:
(earliest affected)
5.1.0.2 rc
Hardware: All Linux (All)
: medium normal
Assignee: David Tardon
URL:
Whiteboard: target:5.3.0 target:5.2.2 target:5.1.6
Keywords: bibisected, regression
Depends on:
Blocks:
 
Reported: 2016-07-22 20:53 UTC by MCbx
Modified: 2016-08-18 23:01 UTC (History)
4 users (show)

See Also:
Crash report or crash signature:


Attachments
WK1 file (4.63 KB, application/vnd.lotus-1-2-3)
2016-07-22 20:53 UTC, MCbx
Details
Screenshot of imported sheet (37.31 KB, image/png)
2016-07-22 21:28 UTC, m_a_riosv
Details
Screenshot of possibly bug (143.38 KB, image/png)
2016-07-22 22:28 UTC, MCbx
Details

Note You need to log in before you can comment on or make changes to this bug.
Description MCbx 2016-07-22 20:53:32 UTC
Created attachment 126368 [details]
WK1 file

LibreOffice 5.1.4.2, on Debian Jessie x64.
The attached WK1 file is generated by Lotus 1-2-3 embedded in HP 200LX portable computer (probably version 2.x) used as data collector.
When I open the file in LibreOffice, all numbers in sheet are exactly 100000000 times larger than they really are in Lotus. In spreadsheet I can see that the problem is both with integer numbers as well as floating-point ones.
Generally, this is a rather serious problem as it may distort data when converting file.

On LO 5.0.1.2 running on Windows XP everything works well and the same file opens with normal numbers.

P.S. 
In attached document headings like "int" "float" have nothing with data type itself.
Comment 1 m_a_riosv 2016-07-22 21:28:16 UTC
Created attachment 126369 [details]
Screenshot of imported sheet

I think I can't see such behavior.

What character set do you select at opening?

Western Europe (DOS/OS2-850) is what I have selected.
Comment 2 MCbx 2016-07-22 22:28:30 UTC
Created attachment 126370 [details]
Screenshot of possibly bug
Comment 3 MCbx 2016-07-22 22:30:59 UTC
I left "Western Europe (DOS/OS2-850)" selected. I tried with codepage 852 too (as I use it in portable) but this file should not have any diacritized characters at all. Added attachment with my presenting of this sheet.
Comment 4 m_a_riosv 2016-07-23 07:43:58 UTC
Please try resetting the user profile, sometimes solves strange issues.
https://wiki.documentfoundation.org/UserProfile
Usually it's enough renaming/deleting the file "user/registrymodifications.xcu",  it affects all the options in Menu/Tools/Options, and the files "user/basic/dialog.xlc" and "scrip.xlc" are overwritten, additionally custom colors in "user/config/standard.soc" are lost.
Comment 5 MCbx 2016-07-23 10:42:02 UTC
Started with a new profile, the same situation. I moved proper profile as I got default icons when started LO.
Comment 6 MCbx 2016-07-23 11:22:11 UTC
Hello again, I made some more tests.
1. WinXP, LO 5.1.4 Portable -> No problem
2. Debian Jessie x32, in VM, MATE Desktop, LO 4.3.3 -> No problem
3. Debian Jessie x32, in VM, MATE Desktop, LO 5.1.4 -> Problem occurs
And what we knew before:
4. WinXP, LO 5.0.1.2 -> No problem
5. Debian Jessie x64, TDE Desktop, LO 5.1.4.2 -> Problem occurs
Generally this is seems to be related to Linux versions.
Comment 7 MM 2016-07-23 13:29:37 UTC
Unconfirmed with v5.1.5.1 under windows 7 x64.
Confirmed with v5.1.5.1 under ubuntu 16.04 x64.
Confirmed with v5.2.0.3 under ubuntu 16.04 x64.
Comment 8 Aron Budea 2016-08-05 01:12:05 UTC
(In reply to MM from comment #7)
> Confirmed with v5.2.0.3 under ubuntu 16.04 x64.

Hm, I couldn't reproduce this with the exact same LibreOffice/OS combination. Strange...
Comment 9 MCbx 2016-08-05 10:53:48 UTC
Maybe this is a problem with OS locale settings? I don't know, I'm just throwing an idea from other projects in which the problem "comma vs dot" persisted and returned for a long time. 

What Linux locale settings do You have?

My locale settings is Polish, so the comma is used as decimal separator and dot as helper. Notice that files
Comment 10 m_a_riosv 2016-08-05 11:30:25 UTC
My locale is Spanish, comma as separator. Open fine for me with several character set, so maybe Linux specific.

Please test if it happens with OpenGL and OpenCL disabled.
Menu/Tools/Options/LibreOffice/View
Menu/Tools/Options/LibreOffice/OpenCL
Comment 11 MCbx 2016-08-05 15:18:05 UTC
Tried with hardware acceleration, OpenGL and OpenCL turned off, the same result.
In fact, my Debian system has Intel GMA945, I don't think it even supports hardware acceleration.
Comment 12 Aron Budea 2016-08-06 02:01:55 UTC
(In reply to MCbx from comment #9)
> What Linux locale settings do You have?

Good idea there.
My Ubuntu's Language and Support / Regional Formats setting was set to English USA, and in that case no matter what I set in Calc as locale, the numbers okay.
Once I changed that to Hungarian (my locale, where also comma is the decimal separator), I could reproduce the issue. Setting locale in LibreOffice didn't matter in this case, either.

Also checked with older versions (with the "buggy" settings):
Reproduced with LO 5.1.0.3.
Not reproduced with LO 5.0.0.5.
=> regression
Comment 13 MM 2016-08-06 14:35:32 UTC
Indeed, setting the regional settings to English USA solves the problem.
Comment 14 MCbx 2016-08-10 19:18:24 UTC
Hello
Added some details...

5.0.1.1 not reproduced
5.0.5.1 not reproduced
5.1.0.1 not reproduced
5.1.0.3 REPRODUCED.
5.1.0.2 REPRODUCED.

It happened between 5.1.0.1 and 5.1.0.2.

Because my bisection locks on single commits (using "till52" repository launches "5.3" dev version?), I just did it manually, by downloading old releases between 5.0 and 5.1.0.3 and going towards buggy version. To verify I used this Debian Jessie VM with MATE.
Comment 15 Aron Budea 2016-08-11 01:36:43 UTC
Thanks for triaging the bug further. I adjusted the version to the earliest known offending one. The range between 5.1.0.1 and 5.1.0.2 includes roughly a month of commits between 2015.12.15 and 2016.01.13.

You wrote your bisection locked on single commits, what did you mean by that? Were you following the steps from [1]? If till52's repo includes the branch point and version change, then the "latest" commit might show version 5.3, but all the other ones should show 5.2.

[1] https://wiki.documentfoundation.org/QA/Bibisect/Linux#Bibisecting_the_Bug
Comment 16 MCbx 2016-08-11 13:52:14 UTC
I still get the same hash in window's title and in bisect log while testing LO and then announcing git bisect good/bad, it looks like the same version is launched again and again. 

When I checkout latest, I get:
mcbx@mcbx:~/Pobrane/LO/lo-linux-dbgutil-daily-till52$ git checkout latest
error: pathspec 'latest' did not match any file(s) known to git.

I can checkout only to oldest. git tag gives me only "oldest". Using "lo-linux-dbgutil-daily-till52.git", OS is Debian Jessie x64.
Comment 17 Aron Budea 2016-08-11 16:13:45 UTC
I'm afraid I have little bibisecting experience, and none with the newer dbgutil repos, raal, can you please assist?
Comment 18 MCbx 2016-08-11 21:20:37 UTC
Hello again!
It looks like there is no "latest" point to do bisection, but downloaded "till52" repository is in "latest" state.
If I try:
git tag -a latest -m "latest ver"
I'm getting the missing tag. Now after running binary, I land in 5.3, then checked out oldest, there is 5.1, checked back to latest. 
Then I decided to start bisection:
git bisect start latest oldest
Git responded me:
Bisecting: 91 revisions left to test after this (roughly 7 steps)

It should be noted in Wiki, about these missing tags, as step-by-step instructions won't work if there is no tag. 
At least for someone who uses Git preconfigured in IDE  with a nice "in" and  "out" buttons :) 
Here are the results:
 641e9c41e8408515bf6b62b94c585c648f12251d is the first bad commit
commit 641e9c41e8408515bf6b62b94c585c648f12251d
Author: Miklos Vajna <vmiklos@collabora.co.uk>
Date:   Fri Jan 1 05:37:14 2016 +0100

    2016-01-01: source-hash-e9598378b55cc05c95bd3f410c396bd44a74341d

:100644 100644 3612d53ed8f838773a19b9fd7094aa5896a80c5f 9201032ac44bf2bafae4249dcc92b97ff3230e8d M      build-info.txt
:040000 040000 03e1af3ce3c56493bc0c41671fb15dd21c70341a e70d88db0eb53af8dbad39d5107ef350d91b3277 M      opt


And bisection log:
# bad: [d3acb9dfa21e2597d0affe3f04fe18b2022b9026] 2016-05-26: source-hash-a042951ad4db2b84021e1d43361511dec998ce82
# good: [69eedb44a433e3be69137a83238a4184785c752f] 2015-11-25: source-hash-7289a140fc68dc898ba2b2357cc960968195f236
git bisect start 'latest' 'oldest'
# bad: [7d1c6fda154836547542c6d741f34a158dd1102c] 2016-02-24: source-hash-f84b8c03462238b821724b7f504ad141c83fcf8f
git bisect bad 7d1c6fda154836547542c6d741f34a158dd1102c
# bad: [6dd89e633720c3f49a4f6eae79311db185759525] 2016-01-09: source-hash-85ac3cd63f6720ff2d3c4b7491f4ad296219fa53
git bisect bad 6dd89e633720c3f49a4f6eae79311db185759525
# good: [4106e1e219154f968de1ba085fa836ee35fb1aa1] 2015-12-17: source-hash-15614c847dde4c52a4974022e5523c9c4fea856b
git bisect good 4106e1e219154f968de1ba085fa836ee35fb1aa1
# good: [56952997ec279fb6050589e00aa21428996850b2] 2015-12-28: source-hash-4b57845388624251b121a3198ea9117a2b81ba14
git bisect good 56952997ec279fb6050589e00aa21428996850b2
# bad: [d7a80dc64f9af14f651b157795c3a1fc1b21c05b] 2016-01-03: source-hash-91f41c3ec23eb063873db8a03c3f6806e2e68af8
git bisect bad d7a80dc64f9af14f651b157795c3a1fc1b21c05b
# good: [2d3c337e431c0a6a79897ee30b973e385204d61e] 2015-12-31: source-hash-150ddbd6bf0aa92ee7b5677302362e7c3759a0ca
git bisect good 2d3c337e431c0a6a79897ee30b973e385204d61e
# bad: [10d08f9257a5b466836966da5f4a3fd652f94d76] 2016-01-02: source-hash-c1258abe50f1508ea0f628ff963bc1914ab86b67
git bisect bad 10d08f9257a5b466836966da5f4a3fd652f94d76
# bad: [641e9c41e8408515bf6b62b94c585c648f12251d] 2016-01-01: source-hash-e9598378b55cc05c95bd3f410c396bd44a74341d
git bisect bad 641e9c41e8408515bf6b62b94c585c648f12251d
# first bad commit: [641e9c41e8408515bf6b62b94c585c648f12251d] 2016-01-01: source-hash-e9598378b55cc05c95bd3f410c396bd44a74341d
Comment 19 Aron Budea 2016-08-11 21:51:30 UTC
Excellent, thanks!
Let's see if I'm interpreting this right, a couple of commits from 2015.12.31 could be at fault, and this seems to be be the most likely candidate:
"upload libodfgen 0.1.6"
https://cgit.freedesktop.org/libreoffice/core/commit/?id=027b98f457e9573510caec25b45d758574332628

Based on some googling, libodfgen is the WordPerfect importer, that's why it could be responsible.

Adding David to CC.
Comment 20 David Tardon 2016-08-12 08:50:35 UTC
(In reply to Aron Budea from comment #19)
> Let's see if I'm interpreting this right, a couple of commits from
> 2015.12.31 could be at fault, and this seems to be be the most likely
> candidate:

If you're not sure, don't use the 'bisected' keyword. That is only applicable if a commit has been positively identified to cause the problem.
Comment 21 MCbx 2016-08-12 09:14:58 UTC
> Based on some googling, libodfgen is the WordPerfect importer, that's why it
> could be responsible.

In file import window I see "libwps" and indeed there is Lotus WK... importer in libwps. Source code of libwpd (libodfgen) in SourceForge contains importers for WordPerfect WP... files.
Comment 22 MCbx 2016-08-12 09:55:00 UTC
One more thing: I've checked out to oldest, then copied libwpd and libwps from the opt/program directory to temp, checked out latest and overwritten new libaries by old ones. In this alignment bug is still present.
Comment 23 Commit Notification 2016-08-12 11:21:17 UTC
David Tardon committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=8d26a169794083f047a57e5c8d3f5da0aaab2583

tdf#101077 make double->str conv. locale-agnostic

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 24 Aron Budea 2016-08-14 07:07:29 UTC
Thanks for the quick fix, David. I can confirm the issue doesn't occur anymore in the following daily build:

Version: 5.3.0.0.alpha0+
Build ID: f8b734a4e2b235c12e86d84c7691e39d05786032
CPU Threads: 1; OS Version: Linux 4.4; UI Render: default; 
TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2016-08-13_10:26:01
Locale: hu-HU (en_US.UTF-8); Calc: group


I'm kind of puzzled by Comment 22, but I guess now we don't need to know why that was the case.
Comment 25 MCbx 2016-08-14 09:30:28 UTC
Thanks a lot for the fix! I was waiting for nightly build to check, but I was looking at dev repo.
Comment 26 Commit Notification 2016-08-18 20:00:35 UTC
David Tardon committed a patch related to this issue.
It has been pushed to "libreoffice-5-2":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=f28ca7a3c0e846b1e516b31a981e814e3768a7d9&h=libreoffice-5-2

tdf#101077 make double->str conv. locale-agnostic

It will be available in 5.2.2.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 27 Commit Notification 2016-08-18 23:01:38 UTC
David Tardon committed a patch related to this issue.
It has been pushed to "libreoffice-5-1":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=fc79186d80ffa8e734727c555456165af6dd51c4&h=libreoffice-5-1

tdf#101077 make double->str conv. locale-agnostic

It will be available in 5.1.6.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.