Bug Hunting Session
Bug 72987 - [firebird db integration] big-endian LibreOffice unable to open little-endian embedded firebird db (found 2944.512, support 11.2)
Summary: [firebird db integration] big-endian LibreOffice unable to open little-endian...
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Base (show other bugs)
Version:
(earliest affected)
4.3.0.0.alpha0+ Master
Hardware: PowerPC All
: medium normal
Assignee: Rene Engelhard
URL:
Whiteboard: target:6.2.0 target:6.1.4
Keywords:
Depends on:
Blocks: Database-Firebird-Default
  Show dependency treegraph
 
Reported: 2013-12-23 14:06 UTC by Douglas Mencken
Modified: 2018-10-16 11:39 UTC (History)
8 users (show)

See Also:
Crash report or crash signature:


Attachments
doesn't work yet (12.50 KB, patch)
2014-01-08 12:28 UTC, Caolán McNamara
Details
no need for rebuild indexes (5.47 KB, patch)
2015-03-23 13:51 UTC, Popa Adrian Marius
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Douglas Mencken 2013-12-23 14:06:17 UTC
I got

##Failure Location unknown## : Error
Test name: FirebirdTest::testIntegerDatabase
An uncaught exception of type com.sun.star.sdbc.SQLException
- firebird_sdbc error:
*unsupported on-disk structure for file /private/var/folders/+o/+oY8OrlkHRyUosC
PilWtLE+++TI/-Tmp-/lu35xfyn.tmp/firebird.fdb; found 2944.512, support 11.2
caused by
'isc_attach_database'

Failures !!!
Run: 2   Failure total: 1   Failures: 0   Errors: 1

Error: a unit test failed, please do one of:

export DEBUGCPPUNIT=TRUE            # for exception catching
export CPPUNITTRACE="gdb --args" # for interactive debugging on linux
...

when I tried to build LibreOffice 9e118293188607ac883ed65754aaccc9ef2c6abb 4.3.0.0alpha+ master.

It looks like that current firebird is unable to correctly handle I/O on Big Endian architectures.

I'm forced to make --disable-firebird-sdbc builds until it is fixed.
Comment 1 Caolán McNamara 2014-01-02 20:18:28 UTC
hmm, firebird saves using the native platform endianness and alignment as far as I can see so its a platform-dependent file format. Which rather sucks for something to haul around in the embedded database case where the db can be opened on any arbitrary system. The exact failure in this case is big-endian ppc is unable to open the little-endian sample database.

One solution might be to not store the .fbd in the odb but instead always "backup" to the odb and "restore" from the odb which would use the platform-neutral .fbk archival format IIRC
Comment 2 Andrzej Hunt 2014-01-02 21:26:50 UTC
(In reply to comment #1)
> One solution might be to not store the .fbd in the odb but instead always
> "backup" to the odb and "restore" from the odb which would use the
> platform-neutral .fbk archival format IIRC

Yes, that's the plan (discussed on IRC but otherwise undocumented).

The backup API is unfortunately somewhat unintuitive, but there shouldn't
be any huge issues replacing the storage format -- however I have no
idea what sort of impact this will have on saving/opening times.
Comment 3 Caolán McNamara 2014-01-08 12:28:17 UTC
Created attachment 91663 [details]
doesn't work yet

A prototype of backing up the db instead of saving it in native format.

Catch is it appears that one has to actually login to the services api with a username/password which means firebird wants to look up those in /usr/local/firebird/security2.fdb, sigh... So presumably we would have to bundle a security2.fdb and somehow tell firebird to use that to auth against unless there is some other way to workaround that problem of isc_service_attach. 

Hopefully get to debug firebird a bit more to see what options are available here.
Comment 4 Andrzej Hunt 2014-01-13 07:53:21 UTC
(In reply to comment #3)
> Created attachment 91663 [details] [review]
> doesn't work yet
> 
> A prototype of backing up the db instead of saving it in native format.
> 
> Catch is it appears that one has to actually login to the services api with
> a username/password which means firebird wants to look up those in
> /usr/local/firebird/security2.fdb, sigh... So presumably we would have to
> bundle a security2.fdb and somehow tell firebird to use that to auth against
> unless there is some other way to workaround that problem of
> isc_service_attach. 
> 
> Hopefully get to debug firebird a bit more to see what options are available
> here.

Ahh sorry, looks like we're duplicating work here, I've also got a
(non-working) alternative patch in gerrit ;)
( https://gerrit.libreoffice.org/#/c/7299/ )

However it doesn't actually work, as I can't convince the backup service to
work.

Based on the docs, as we're using embedded mode, we specifically *shouldn't* pass a username/password in the spb (i.e. the same as when connecting to the
db -- no username/pass for embedded mode), and I'm not getting any authentication errors with my version suggesting that's not where the
problem lies (but I'll try and experiment anyway just in case).


But I have no idea where the problem could be, as the only error output
given by the backup service is:

 gbak:Exiting before completion due to errors

As far as I can tell we're not missing any libraries or other firebird files that might be needed, but I suspect that it might be necessary to debug firebird itself to figure out what's happening.
Comment 5 Popa Adrian Marius 2014-04-28 08:23:38 UTC
I have asked on Firebird devel and no password / access to security2.fdb is required for embedded mode 
Please add a backtrace with the error if possible
Comment 6 Alex Thurgood 2015-01-03 17:39:20 UTC Comment hidden (no-value)
Comment 7 Popa Adrian Marius 2015-03-23 13:34:30 UTC
I will retest and add patches in gerrit 


1. First i will remove rebuildIndexes() it will not be necesarry when we keep fbk format in odb file 

2. I will retest isc_action_svc_backup , it is not necessary to have a password database with firebird embedded and i will add the secondary patch
Comment 8 Popa Adrian Marius 2015-03-23 13:51:43 UTC
Created attachment 114274 [details]
no need for rebuild indexes
Comment 9 Popa Adrian Marius 2015-03-23 15:53:35 UTC
added also to gerrit 

https://gerrit.libreoffice.org/14968
Comment 10 Popa Adrian Marius 2015-03-24 16:10:42 UTC

First atttempt for secondary patch 
https://gist.github.com/mariuz/60d4d1edfe83c0670648

, i will investigate the error bellow 


unknown:0:FirebirdTest::testIntegerDatabase
An uncaught exception of type com.sun.star.uno.RuntimeException
- unexpected UNO exception caught: firebird_sdbc error:
*invalid database handle (no active connection)
caused by
'isc_detach_database'


##Failure Location unknown## : Error
Test name: FirebirdTest::testIntegerDatabase
An uncaught exception of type com.sun.star.uno.RuntimeException
- unexpected UNO exception caught: firebird_sdbc error:
*invalid database handle (no active connection)
caused by
'isc_detach_database'


Failures !!!
Run: 2   Failure total: 1   Failures: 0   Errors: 1

Error: a unit test failed, please do one of:

export DEBUGCPPUNIT=TRUE            # for exception catching
export CPPUNITTRACE="gdb --args"    # for interactive debugging on Linux
export VALGRIND=memcheck            # for memory checking

and retry using: make CppunitTest_dbaccess_firebird_test
Comment 11 Björn Michaelsen 2015-07-06 16:46:23 UTC
Comment on attachment 91663 [details]
doesn't work yet

Mark patches obsolete, as work is ongoing on gerrit now.
Comment 12 Commit Notification 2016-01-05 11:44:02 UTC
Popa Adrian Marius committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=f961fef03906fc059a4a0c008799f68fc22727c1

WIP tdf#72987 Use firebird backup format for .odb no need for rebuild indexes

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 13 Commit Notification 2016-01-05 12:01:57 UTC
Lionel Elie Mamane committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=c88fd50c4e8815546e9830a41bf08edf9f165923

Revert "WIP tdf#72987 Use firebird backup format for .odb no need for rebuild indexes"

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 14 Lionel Elie Mamane 2016-09-07 17:58:36 UTC
fixed by the switch to Firebird "fbk" (backup) format
Comment 15 Olivier Tilloy 2018-09-06 15:55:42 UTC
Looks like the problem is back (or was never fully fixed), see https://salsa.debian.org/libreoffice-team/libreoffice/libreoffice/merge_requests/3#note_42132
Comment 16 Xisco Faulí 2018-09-11 09:41:31 UTC
(In reply to Olivier Tilloy from comment #15)
> Looks like the problem is back (or was never fully fixed), see
> https://salsa.debian.org/libreoffice-team/libreoffice/libreoffice/
> merge_requests/3#note_42132

@Lionel Elie Mamane, @Popa Adrian Marius, @Tamas Bunth, Do you reproduce the problem described above ?
Comment 17 Lionel Elie Mamane 2018-09-11 09:58:07 UTC
You have to understand how the problem was fixed. The problem was fixed by saving (within the odb zip structure) firebird data in an endianess-independent format, called the "backup" format, in a file with extension ".fbk".

From the build log linked in comment 15, it seems the test is being run on a file that has an "fdb" data file, that is the "native", endianess-dependent Firebird format.

Any "old" file that was created by a LibreOffice version that saved firebird data in the fdb format (all these versions have Firebird as _experimental_ _only_ feature) need to be first opened in a newer LibreOffice version running on the _same_ endianess, save the file again (which will save as fbk), and the resulting file will be endianess-independent.

Now, the error comes from an odb file that is in the LibreOffice source tree, for automated QA tests. The question is, what is that test supposed to test?

Case 1: it is supposed to test a normal firebird file as created by the current version, to make sure it works correctly. Then the two ODB files dbaccess/qa/unit/data/firebird_integer_x64le{,_ods12}.odb need to be converted to fbk format, they are in fdb format. Just open them and save them again.

Case 2: it is supposed to test backwards compatibility, that is the fact that newer LibreOffice can still open old-style fdb-data files. Then, this test should only be run on little endian architectures, or a new test file for big endian architecture needs to be created.

As both tests are actually useful, we can actually duplicate the test and do both :)

Andrzej, Tamas, any thoughts on the above?
Comment 18 Rene Engelhard 2018-09-21 16:32:30 UTC
ah, I see.

Working on that...
Comment 19 Rene Engelhard 2018-09-23 17:07:36 UTC
see

--- snip ---
remote: New Changes:
remote:   https://gerrit.libreoffice.org/60916 tdf#72987 run firebird test for little endian only for now
remote:   https://gerrit.libreoffice.org/60917 tdf#72987 split firebird test into a regression test and a "new" test
--- snip ---
Comment 20 Commit Notification 2018-10-03 08:13:25 UTC
Rene Engelhard committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=f104b3cafee63b47a735cfdce0f05dab2eedbb91

tdf#72987 run firebird test for little endian only for now

It will be available in 6.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 21 Commit Notification 2018-10-03 08:14:50 UTC
Rene Engelhard committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=36befb3aca96907a14e71e82497dbb8f03ead5ab

tdf#72987 split firebird test into a regression test and a "new" test

It will be available in 6.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 22 Commit Notification 2018-10-16 10:31:27 UTC
Rene Engelhard committed a patch related to this issue.
It has been pushed to "libreoffice-6-1":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=d5d09bb733842252b3e2d6e7480b211e67136a75&h=libreoffice-6-1

tdf#72987 split firebird test into a regression test and a "new" test

It will be available in 6.1.4.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.