Bug 110997 - Some forms stall with 100% on open - It's a date field problem
Summary: Some forms stall with 100% on open - It's a date field problem
Status: VERIFIED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Base (show other bugs)
Version:
(earliest affected)
5.4.0.3 release
Hardware: x86-64 (AMD64) All
: high major
Assignee: Lionel Elie Mamane
URL:
Whiteboard: target:6.0.0 target:5.4.1
Keywords: haveBacktrace, regression
Depends on:
Blocks:
 
Reported: 2017-08-01 15:44 UTC by tim
Modified: 2017-08-15 15:58 UTC (History)
4 users (show)

See Also:
Crash report or crash signature:


Attachments
Self-contained test database (11.77 KB, application/vnd.oasis.opendocument.database)
2017-08-01 20:24 UTC, tim
Details
Backtrace from lldb session (15.46 KB, text/plain)
2017-08-02 08:06 UTC, Alex Thurgood
Details

Note You need to log in before you can comment on or make changes to this bug.
Description tim 2017-08-01 15:44:56 UTC
Description:
I have some quite complex forms with several subforms, which when opened stall, with CPU at 100%. This is new in 5.4.0.3 (both direct from Libreoffice and via ubuntu ppa).  Version 5.3.4.2 is OK.

Steps to Reproduce:
1. Open a form from another form providing a filter
2. Form displays, but goes grey
3. Waits forever, CPU at 100%

Actual Results:  
Not sure what I can add. 

Expected Results:
Nothing - just hangs.


Reproducible: Always

User Profile Reset: No

Additional Info:
This happens on non-trivial forms.  I don't know exactly what the crucial factor is.  Some, mainly simpler, forms open OK. My database is in Mariadb on another system on my LAN. I have tried both MYSQL (native) and MYSQL (JDBC)

It would not be at all easy to provide a demo, since the forms are complex, with macros, filtering dialogs, and data resident on another system.

IF you have any ideas how I can diagnose this hang, please let me know. I'm willing to install a debug version of LO if needs be.


User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.112 Safari/537.36 Vivaldi/1.91.867.48
Comment 1 tim 2017-08-01 17:17:46 UTC
I starting stripping one of the forms down, removing field by field, and after assuming it was due to the complexity, it turns out to be a date field - no macros and so on, just a date based on a timestamp field in a table, taht used to work and now loops forever.  Intriguingly the time field based on the same data doesn't cause a hang.

When I get time (a few hours probably) I'll run up a demonstration.
Comment 2 Xisco Faulí 2017-08-01 17:24:49 UTC Comment hidden (obsolete)
Comment 3 tim 2017-08-01 20:24:52 UTC
Created attachment 135055 [details]
Self-contained test database

The attached file has a form which hangs in 5.4.0.3 but does not in 5.3.4.2.

It turns out that something has changed in the underlying data from which my mariadb database is regularly copied which means the (integer) date is, I believe, invalid (so that's my problem to sort out).  Nonetheless it shouldn't hang forever.

Further, when trying to resolve this I tried to change the format of the control ('replace with').  That didn't work in 5.4.0.3 but does in 5.3.4.2.  I guess that is a separate bug.
Comment 4 Alex Thurgood 2017-08-02 07:56:59 UTC
Confirming also on 

Version: 5.4.0.3
Build ID: 7556cbc6811c9d992f4064ab9287069087d7f62c
Threads CPU : 8; OS : Mac OS X 10.12.6; UI Render : par défaut; 
Locale : fr-FR (fr_FR.UTF-8); Calc: group
Comment 5 Alex Thurgood 2017-08-02 08:04:58 UTC
On my master debug-enabled build, attempting to open the form in a lldb session shows this:

Process 1237 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSEGV
    frame #0: 0x000000012ee0b2b4
->  0x12ee0b2b4: movl   (%rsi), %eax
    0x12ee0b2b6: leaq   0xf8(%rbp), %rsi
    0x12ee0b2bd: vmovdqu %ymm0, (%rsi)
    0x12ee0b2c1: vmovdqu %ymm7, 0x20(%rsi)

Continuing in the session allows the form to be displayed.

Attaching bt from sigsegv and further session output.
Comment 6 Alex Thurgood 2017-08-02 08:06:03 UTC
Created attachment 135061 [details]
Backtrace from lldb session
Comment 7 Lionel Elie Mamane 2017-08-02 09:14:04 UTC
(In reply to Alex Thurgood from comment #5)
> On my master debug-enabled build, attempting to open the form in a lldb
> session shows this:

> Process 1237 stopped
> * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSEGV

> Continuing in the session allows the form to be displayed.

> Attaching bt from sigsegv and further session output.

Alex, when Java is used, some SIGSEGVs are "normal", as in they are used internally by Java, caught and treated by Java. So, when Java is being used, any SIGSEGV that does not actually crash/abort LibreOffice (meaning it was treated by Java) should be ignored.

Unless you are diagnosing a crash/abort caused by a SIGSEGV, I recommend to setup lldb to ignore SIGSEGVs altogether. Not sure how to do it in lldb, but in gdb the command is "handle SIGSEGV noprint nostop pass".
Comment 8 Lionel Elie Mamane 2017-08-02 09:15:18 UTC
(In reply to Alex Thurgood from comment #5)

> Continuing in the session allows the form to be displayed.

For complete clarity, does the form display correctly or do you get the same "looping forever with 100% CPU" problem?
Comment 9 Alex Thurgood 2017-08-02 10:05:00 UTC
(In reply to Lionel Elie Mamane from comment #7)

> 
> Unless you are diagnosing a crash/abort caused by a SIGSEGV, I recommend to
> setup lldb to ignore SIGSEGVs altogether. Not sure how to do it in lldb, but
> in gdb the command is "handle SIGSEGV noprint nostop pass".

OK, will have a look to see if I can integrate that somewhere...
Comment 10 Alex Thurgood 2017-08-02 10:07:37 UTC
(In reply to Lionel Elie Mamane from comment #8)
> (In reply to Alex Thurgood from comment #5)
> 
> > Continuing in the session allows the form to be displayed.
> 
> For complete clarity, does the form display correctly or do you get the same
> "looping forever with 100% CPU" problem?

Sorry, yes, I should've been clearer - the form loads completely, no looping or 100% CPU usage. It is as if the form has loaded normally - whether this is because the problem is inexistent in master, or whether the debug session allows it to load nonetheless, I have no idea, and I don't have a debug 5432 build to test with which I might be able to compare.
Comment 11 tim 2017-08-02 10:25:54 UTC
(In reply to tim from comment #3)
> Created attachment 135055 [details]
> Self-contained test database
> 
> The attached file has a form which hangs in 5.4.0.3 but does not in 5.3.4.2.
> 
> It turns out that something has changed in the underlying data from which my
> mariadb database is regularly copied which means the (integer) date is, I
> believe, invalid (so that's my problem to sort out).  Nonetheless it
> shouldn't hang forever.
> 
> Further, when trying to resolve this I tried to change the format of the
> control ('replace with').  That didn't work in 5.4.0.3 but does in 5.3.4.2. 
> I guess that is a separate bug.
I'll open a new bug report for the problem that 'replace with' doesn't work.
Comment 12 Lionel Elie Mamane 2017-08-02 10:45:02 UTC
The problem comes from a date very far in the future, namely approximately in the year 4 111 224. The LibreOffice Date structure cannot handle years after 32767.
It is a bug that this leads to an infinite loop.

The third column of some table or query is of type INTEGER, and you interpret this as a date (because you bound a Date control to that column?). The value of that column (in some row) is 1501593742. Since you request a date out of it, this is understood as a number of days after 1/1/1900.
Comment 13 tim 2017-08-02 10:58:50 UTC
(In reply to Lionel Elie Mamane from comment #12)
> The problem comes from a date very far in the future, namely approximately
> in the year 4 111 224. The LibreOffice Date structure cannot handle years
> after 32767.
> It is a bug that this leads to an infinite loop.
> 
> The third column of some table or query is of type INTEGER, and you
> interpret this as a date (because you bound a Date control to that column?).
> The value of that column (in some row) is 1501593742. Since you request a
> date out of it, this is understood as a number of days after 1/1/1900.
Thanks. I realised after raising the report that the integer content had changed (the data comes from another database originally) - it's now a unix timestamp - so treating it just as a date won't work.  I have now worked round that for my own purposes (using a macro to translate - I tried sql functions but not seemed to work for me).

However, something has changed from 5.3.4.2, which shows it as a date in 2200 (obviously wrong), to 5.4.3.2 which loops.  Others might get caught by having invalid data causing loops.  It took me quite a while to work out that the loop might be caused by something this simple (my original form as 30 or so fields on it, 7 sub-forms and so on).

I realise this is not exactly a high priority, but endless loops are puzzling to users :)
Comment 14 Lionel Elie Mamane 2017-08-02 13:44:33 UTC
Your "utimestamp" field of type "integer" seems to be meant as a number of seconds since 1/1/1970 00:00 midnight (aka "UNIX timestamp").

The "SQL way", if you mean a timestamp, is to have a field of type timestamp / datetime.

If you "force" LibreOffice to interpret a number as a date or time, then the integer part of a number is "days since 1/1/1900" (or 30/12/1899 or something like that, depending on what application (Calc, Base, ...), what setting, ...) and the fractional part of the number is the time in the day, where 0 is 00:00 midnight, 0.5 is 12:00 noon and 0.9999999... is 23:59:59.99999...
Comment 15 Lionel Elie Mamane 2017-08-02 13:46:20 UTC
(In reply to tim from comment #13)
> However, something has changed from 5.3.4.2, which shows it as a date in
> 2200 (obviously wrong), to 5.4.3.2 which loops.  Others might get caught by
> having invalid data causing loops.

Yes, as I wrote, the fact that it creates an infinite loop is a bug. No discussion about that.

> I realise this is not exactly a high priority,

It is. Endless loop is not acceptable (unless the user programmed an endless loop in Basic or something like that).
Comment 16 Commit Notification 2017-08-02 13:51:31 UTC
Lionel Elie Mamane committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=0ccbd58834259cd7724edacde283f4f0f6ed2db0

tdf#110997 protect calls to implBuildFromRelative from year overflow

It will be available in 6.0.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 17 Commit Notification 2017-08-08 13:28:20 UTC
Lionel Elie Mamane committed a patch related to this issue.
It has been pushed to "libreoffice-5-4":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=de8c0152157bb32a6df9b6d4ac4102aef8e8a384&h=libreoffice-5-4

tdf#110997 protect calls to implBuildFromRelative from year overflow

It will be available in 5.4.1.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 18 tim 2017-08-15 15:32:28 UTC
I've just checked in ubuntu pre-release Version: 5.4.1.1
Build ID: 1:5.4.1~rc1-0ubuntu0.17.04.1~lo0
CPU threads: 8; OS: Linux 4.10; UI render: default; VCL: gtk3; 
Locale: en-GB (en_GB.UTF-8); Calc: group

All OK - Thanks.
Comment 19 Julien Nabet 2017-08-15 15:58:00 UTC
Thank you Tim for your feedback.
Let's put this one to VERIFIED then.