Bug 156471 - framework to support generic/unknown embedded databases in new database wizard
Summary: framework to support generic/unknown embedded databases in new database wizard
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Base (show other bugs)
Version:
(earliest affected)
7.0 all versions
Hardware: All All
: medium enhancement
Assignee: prrvchr
URL:
Whiteboard: target:24.2.0 inReleaseNotes:24.2
Keywords:
Depends on:
Blocks:
 
Reported: 2023-07-26 02:13 UTC by prrvchr
Modified: 2024-04-16 18:56 UTC (History)
5 users (show)

See Also:
Crash report or crash signature:


Attachments
New database Wizard with url sdbc:embedded:hsqldb (39.82 KB, image/png)
2023-07-26 14:15 UTC, prrvchr
Details
New database Wizard with url sdbc:embedded:sqlite (41.47 KB, image/png)
2023-07-26 14:17 UTC, prrvchr
Details

Note You need to log in before you can comment on or make changes to this bug.
Description prrvchr 2023-07-26 02:13:19 UTC
Description:
It would be nice if it was possible to have several urls dedicated to embedded databases, like:
- sdbc:embedded:hsqldb
- sdbc:embedded:firebird
- sdbc:embedded:sqlite
- sdbc:embedded:your_database

This would allow access to different databases in a single odb file.

This would allow both extensions to work simultaneously:
- HsqlDriverOOo (https://prrvchr.github.io/HsqlDriverOOo/)
- SQLiteOOo (https://prrvchr.github.io/SQLiteOOo/)

Normally only the new database wizard needs to be modified to make this possible...

Actual Results:
 

Expected Results:
 


Reproducible: Always


User Profile Reset: No

Additional Info:
Comment 1 Julien Nabet 2023-07-26 10:45:57 UTC
We've already got:
- sdbc:embedded:hsqldb (obviously)
- sdbc:embedded:firebird
Firebird creation database is available only when enabling experimental features.

For "sdbc:embedded:sqlite", why not. There's no SQLite implementation yet in LO but I saw in your weblinks you used JDBC for your extension.

About "sdbc:embedded:your_database", do you mean a kind of generic one? If yes, don't know if it's possible.

Lionel/Robert/Alex: any opinion here?
Comment 2 prrvchr 2023-07-26 11:58:47 UTC
Hi Julien,

Could we not consider that all urls starting with: sdbc:embedded are managed as embedded databases.
After the installed drivers do the job...

> but I saw in your weblinks you used JDBC for your extension.

There is a big advantage, it's the installation that happens while drinking a beer (no more compilation needed)
Comment 3 Lionel Elie Mamane 2023-07-26 13:39:52 UTC
I understand the request as having several embedded databases in the same odb file.

This is limited by a more fundamental design of LibreOffice Base, which is that an .odb file is a connection to one database (be it embedded or not). LibreOffice Base isn't its own Jet-like database engine that could combine several databases and execute queries across several databases. All SQL queries are executed by the one and only database engine that the odb file "connects to".

Unless someone contributes a Jet-like database engine (that is a big work!), it will stay like that.

In theory we could allow storing several different databases within the same .odb file, but at opening one chooses which one one connects to (and then, during that session, one sees only that database, not the others)? But to me that would feel weird for users, and I don't see a use case (as long as one cannot combine data from the several databases).

I'm closing as WONTFIX. If I misunderstood the request, reopen. If you present a use case that I didn't see, reopen. If anyone volunteers to develop a Jet-like database engine that can combine data from several sources, reopen.
Comment 4 Lionel Elie Mamane 2023-07-26 13:50:27 UTC
(In reply to prrvchr from comment #2)

> Could we not consider that all urls starting with: sdbc:embedded are managed
> as embedded databases.

I don't understand what you want to happen exactly technically where. I mean, as a policy, I would certainly "encourage" (if not __require__ on a policy level to accept the patch) any driver to use URIs like sdbc:embedded:foo only for embedded databases (!) and vice versa. But if you want to enforce that technically, I don't see how, and in theory I think it is not possible: we would have to "inspect" the driver code (automatically) to see where it writes the actual database data (is it in the odb file or elsewhere?). That would inter alia solve the halting problem :) (In theory, we could apply some heuristics to guess, but... why? and what when the guess is wrong?)

> After the installed drivers do the job...

Well, the current system is that drivers register the URI patterns (usually prefixes...) they are able to manage, and LibreOffice starts (and delegates work) to the driver that has registered to manage the URI that the odb file is configured with.

So any driver that manages any sdbc:embedded:XXXXXX for any value of XXXXXX will register that URI prefix. So given an odb file configured with URI sdbc:embedded:foobar or sdbc:embedded:foobar:qux or sdbc:embedded:foobar?qux or ..., either there is a driver that manages it and it will work or there is not, and the user will get an error message along the lines of "no driver for sdbc:embedded:foobar".
Comment 5 Alex Thurgood 2023-07-26 14:01:57 UTC
Absent the valid points made by Lionel, with which I concur, if you want people to accept your extension more widely for testing by DB users, it would be better for it to be installable across the whole range of LO OS releases.

I tried installing the jdbcDriverOOo.oxt extension in its latest available version into macOS Arm (aarch64) LibreOffice 7552.

Installation of this oxt fails repeatedly with an error message:

Connector : couldn't connect to pipe "922164caf3b187610ca4e955dd0f62342f58fa68a4dea863f2f26d5bb19": 10

I remember encountering this error message when I first tried to install that extension a few months ago (and gave up on).

If the migration to embedded Firebird has taught us anything, it is that the whole embedded ODB container concept is relatively fragile, and it doesn't take much to upset the container, rendering one's data inaccessible or corrupt.

Having multiple, potentially accessible embedded DBs would need to ensure that the container doesn't get corrupted just because a user decided to change the URL one day to a different backend embedded engine.

The other issue I have is that it relies on Java. The whole point of moving to embedded Firebird (albeit admittedly with somewhat mixed results) was to reduce the dependency on Java for interacting with the database components. Why would we want to introduce another dependency on Java for a different embedded DB engine,
Comment 6 prrvchr 2023-07-26 14:15:53 UTC
Created attachment 188575 [details]
New database Wizard with url sdbc:embedded:hsqldb
Comment 7 prrvchr 2023-07-26 14:17:54 UTC
Created attachment 188576 [details]
New database Wizard with url sdbc:embedded:sqlite
Comment 8 prrvchr 2023-07-26 14:23:39 UTC
> I understand the request as having several embedded databases in the same odb file.

No, only one database per odb file, but the possibility of managing as many integrated databases as it is possible to provide drivers...

An odb file could only contain an hsqldb or sqlite or firebire or ... database.

The wizard should produce the same connection window for any url starting with sdbc:embedded (see attached files)
Comment 9 prrvchr 2023-07-26 14:37:33 UTC
> The other issue I have is that it relies on Java.

In which driver you can manage the nanosecond, time zones and system-versioned temporal table?
I'm afraid none, so thanks to java and hsqldb...
Comment 10 prrvchr 2023-07-26 14:55:43 UTC
> Installation of this oxt fails repeatedly with an error message...

I'm sorry for this problem, but for the moment I have no possibility to test under macOS Arm (aarch64)
Comment 11 Lionel Elie Mamane 2023-07-26 15:40:08 UTC
(In reply to prrvchr from comment #8)
> No, only one database per odb file, but the possibility of managing as many
> integrated databases as it is possible to provide drivers...

> An odb file could only contain an hsqldb or sqlite or firebire or ...
> database.

I don't see how that is not already the case. Built-in in LibreOffice, we have HSQLDB 1.8 and (when experimental features are enabled) Firebird available, and the new database wizard gives the choice between the two. Any patch / extension adding a driver should also add the corresponding choice to the wizard.

Hmm... Maybe you mean that there is no mechanism for an extension to "plug in" into the wizard to add the choice, it can only be done from LibreOffice code proper?

BTW, have you considered integrating your embedded-SQLite and embedded-recent-hypersql to LibreOffice proper rather than having them as extensions? As long as the recent-hypersql one doesn't break compatibility with existing embedded HSQLDB databases (e.g. it uses a separate URI, e.g. sdbc:embedded:hsqldb2) and it manages compatibility across future HSQLDB versions sanely (e.g. it uses a different URI for each incompatible version of the data store of HyperSQL, or something different to discuss), I see no technical obstacle to integration.
Comment 12 Lionel Elie Mamane 2023-07-26 15:41:56 UTC
(In reply to Lionel Elie Mamane from comment #11)

> Hmm... Maybe you mean that there is no mechanism for an extension to "plug
> in" into the wizard to add the choice, it can only be done from LibreOffice
> code proper?

Err, no, your screenshots show that you achieved adding new choices to the wizard from an extension just fine, so not only there is a mechanism, but you found it and it works. So I'm again not sure what you are asking for.
Comment 13 Robert Großkopf 2023-07-26 17:01:42 UTC
At this moment we try to integrate Firebird and will get Base running without Java: https://bugs.documentfoundation.org/show_bug.cgi?id=154426

When adding databases, which will only work with JDBC-connection, we will exclude all those people who have problems with running Java on their system.

Internal databases in LO (at this moment) are usable for creating a database and learning working with Base. But the first step I do when creating is finished: I extract the Firebird file and connect to the external file. So the data will be saved when changing the row and not when closing my Base file - with all the problems of loosing data while file couldn't be closed right way.
Comment 14 prrvchr 2023-07-26 17:48:03 UTC
screenshots: New database Wizard with url sdbc:embedded:hsqldb
Show how an embedded database Wizard should be. There are only two steps in the browser (no step for choosing the connection url)

screenshots: New database Wizard with url sdbc:embedded:sqlite
Does not show a embedded database Wizard. We have three steps in the browser (including one to choose the connection url)

The problem is that to declare a driver in embedded mode the only solution is to use the url: sdbc:embedded:hsqldb.

All else does not work... is this really normal?

> So the data will be saved when changing the row and not when closing my Base file - with all the problems of loosing data while file couldn't be closed right way.

With the HsqlDriverOOo and SQLiteOOo extensions which work in integrated mode (everything is in the odb file) if you insert 2 rows in a table, you kill the soffice daemon, you open LibreOffice again: you have lost nothing... Anyway that's what I checked with HsqlDB.
Comment 15 Lionel Elie Mamane 2023-07-26 17:57:43 UTC
(In reply to Robert Großkopf from comment #13)
> At this moment we try to integrate Firebird and will get Base running
> without Java: https://bugs.documentfoundation.org/show_bug.cgi?id=154426
> 
> When adding databases, which will only work with JDBC-connection, we will
> exclude all those people who have problems with running Java on their system.

While being more independent from Java is a wished enhancement in LibreOffice, that does in my opinion not preclude accepting contributions that make use of Java for alternative databases... If prrvchr contributes Java-using code, I'd rather we have this available than not having the contributed features and alternative databases. That's how volunteer/community/...-driven development works, the project gets what the volunteers/community/... feels like contributing, not necessarily what the project would like.

Another problem with Java/JDBC in LibreOffice is the rather bad performance hit each time C++ code calls Java code on GNU/Linux (and possibly the BSDs, I don't know), this is not limited to LibreOffice, but LibreOffice is hit hard because each individual sdbc call (access to a single column of a single row) becomes a Java/JDBC all so there are many C++ to Java calls. But again, if someone (prrvchr) is happy to contribute AND MAINTAIN new features, and users are happy to use them... don't let them being Java-based get in the way.

IMO people that want better Java-independence in LibreOffice can submit patches to polish Firebird integration, without blocking prrvchr's Java-based alternative database contributions.
Comment 16 Lionel Elie Mamane 2023-07-26 18:07:05 UTC
(In reply to prrvchr from comment #14)
> screenshots: New database Wizard with url sdbc:embedded:sqlite
> Does not show a embedded database Wizard. We have three steps in the browser
> (including one to choose the connection url)

Why do you need the extra step to choose the connection URI? Since you already added an extra entry to the drop-down menu in the choice for "Embedded database", why doesn't making that choice also set the correct URI and that's it?

> The problem is that to declare a driver in embedded mode the only solution
> is to use the url: sdbc:embedded:hsqldb.

Well, that is clearly not true, since the firebird embedded driver doesn't use URI sdbc:embedded:hsqldb but uses sdbc:embedded:firebird.

I feel we are not understanding each other at some level.

>> So the data will be saved when changing the row and not when closing my Base file - with all the problems of loosing data while file couldn't be closed right way.
 
> With the HsqlDriverOOo and SQLiteOOo extensions which work in integrated
> mode (everything is in the odb file) if you insert 2 rows in a table, you
> kill the soffice daemon, you open LibreOffice again: you have lost
> nothing... Anyway that's what I checked with HsqlDB.

LibreOffice has battled with corruption issues in the ZIP structure of the odb file with embedded database since ages. If you have found a robust solution, that works on any mix of changes of the table data and the other content of the odb file (queries, reports, forms, ...),  I'd be very happy to have it applied to all embedded databases (historic HSQLDB, Firebird, ...).
Comment 17 prrvchr 2023-07-26 18:26:36 UTC
> Why do you need the extra step to choose the connection URI? Since you already
> added an extra entry to the drop-down menu in the choice for "Embedded
> database", why doesn't making that choice also set the correct URI and that's it?

It is not me who chose to display the step to choose the connection URI but the Wizard...
I don't need the step to choose the connection URI for an embedded database...

I repeat that the only solution in order to create a new embedded database is to use the url: sdbc:embedded:hsqldb (and maybe also sdbc:embedded:firebird)

And we may criticize Java, but it is still thanks to it that we can use SQLite (the first database to be used in the world) under LibreOffice and in a fairly simple way:
install java
install LibreOffice
install extensions

In my opinion, we can never make it that simple...
Comment 18 prrvchr 2023-07-26 18:38:21 UTC
And then it would be nice to do performance tests on the jdbcDriverOOo driver and the JDBC implementation in UNO...
No idea of the results
Comment 19 prrvchr 2023-07-26 18:52:41 UTC
> LibreOffice has battled with corruption issues in the ZIP structure of the odb file
> with embedded database since ages. If you have found a robust solution, that works > on any mix of changes of the table data and the other content of the odb file
> (queries, reports, forms, ...),  I'd be very happy to have it applied to all
> embedded databases (historic HSQLDB, Firebird, ...).

Everything is explained in the documentation (https://prrvchr.github.io/SQLiteOOo/#how-does-it-work), but if you don't move the odb file after abnormal closing, you have to find the data as it should.
Comment 20 Lionel Elie Mamane 2023-07-26 19:25:15 UTC
(In reply to prrvchr from comment #17)
> It is not me who chose to display the step to choose the connection URI but
> the Wizard...

From glancing at the code of the wizard (dbaccess/source/ui/dlg/dbwiz.cxx), function
IMPL_LINK(ODbTypeWizDialog, OnTypeSelected, OGeneralPage&, _rTabPage, void)
it looks to me like it makes that decision by calling
 m_pCollection->isConnectionUrlRequired(m_eType)
which itself (iun dbaccess/source/core/misc/dsntypes.cxx) seems to just look at whether the driver registers with a pattern of URLs (ending with "*") or with a single URL (not ending with "*"). So the question is what URL or URL pattern the drivers you add register in the xcu file that you ship...

Or maybe I read it wrong, and the choice is made in
ODsnTypeCollection::fillPageIds
in that case, you probably have to add a new value
DST_EMBEDDED_UNKNOWN
in the enum DATASOURCE_TYPE in dbaccess/source/inc/dsntypes.hxx
and then add that to the treatment throughout the dsntypes.cxx file, everywhere there is a switch(eType) and also in function ODsnTypeCollection::determineType

> And we may criticize Java, but it is still thanks to it that we can use
> SQLite (the first database to be used in the world) under LibreOffice and in
> a fairly simple way:
> install java
> install LibreOffice
> install extensions
> 
> In my opinion, we can never make it that simple...

Oh, with some development work, we can make it much easier for the user, just:

install LibreOffice.

"all we have to do" is develop an sdbc driver that calls the SQLite C library directly (not through JDBC), and ship that within LibreOffice. Would be easier for the user, but more development work for us :)

Or even if it is an extension and not integrated into LibreOffice proper, it would still remove one step for the user, it would just be:

install LibreOffice
install extension

But still much more development work for the person(s) that develop the extension.
Comment 21 Lionel Elie Mamane 2023-07-26 19:31:19 UTC
(In reply to prrvchr from comment #19)
> > LibreOffice has battled with corruption issues in the ZIP structure of the odb file
> > with embedded database since ages. If you have found a robust solution, that works > on any mix of changes of the table data and the other content of the odb file
> > (queries, reports, forms, ...),  I'd be very happy to have it applied to all
> > embedded databases (historic HSQLDB, Firebird, ...).
> 
> Everything is explained in the documentation
> (https://prrvchr.github.io/SQLiteOOo/#how-does-it-work), but if you don't
> move the odb file after abnormal closing, you have to find the data as it
> should.

Oh you mean after killing LibreOffice (or crash thereof), the table data is NOT SAVED in the odb file? Well, that is not what we understood from your description in comment 14. Indeed, not trying to save data into the odb file at any time, but only when the odb file is being saved itself is how we avoided in embedded FireBird the corruption issues historically encountered with embedded HSQLDB...
Comment 22 prrvchr 2023-07-26 19:35:15 UTC
The only way to write a database driver for UNO (LibreOffice / OpenOffice) is:
- Create UNO service com.sun.star.sdbc.Driver
- Declare the existence of this driver to LibreOffice / OpenOffice through a Drivers.xcu file (which declares the uri that the driver will use and I confirm not ending with "*")

I have no access to the new database Wizard.

If I declare as uri: sdbc:embedded:sqlite, then:
- This driver is found in the list of integrated drivers (first drop-down list of the new database Wizard)
- It is also found (when it should not be there) in the second drop-down list.
- And especially the number of steps in the wizard is not good: there are three steps...  which prevents the creation of the new embedded database.
Comment 23 Lionel Elie Mamane 2023-07-26 19:39:34 UTC
(In reply to prrvchr from comment #22)
> I have no access to the new database Wizard.

https://www.libreoffice.org/about-us/source-code/

will happily review patches
Comment 24 prrvchr 2023-07-26 19:55:34 UTC
> Or maybe I read it wrong, and the choice is made in
> ODsnTypeCollection::fillPageIds
> in that case, you probably have to add a new value DST_EMBEDDED_UNKNOWN
> in the enum DATASOURCE_TYPE in dbaccess/source/inc/dsntypes.hxx
> and then add that to the treatment throughout the dsntypes.cxx file,
> everywhere there is a switch(eType) and also in function ODsnTypeCollection::determineType

I think you're right, it's all there. And apparently only sdbc:embedded:hsqldb and sdbc:embedded:firebird are taken into account..
Comment 25 Julien Nabet 2023-07-26 19:56:56 UTC
A good start for dev contribution too:
https://wiki.documentfoundation.org/Development/GetInvolved

git grepping or using https://opengrok.libreoffice.org/ may help to find more quickly interesting parts.
Comment 26 prrvchr 2023-07-26 20:05:00 UTC
I would like to try to change this Wizard, I have a local copy of the LibreOffice gerrit, but I must confess that I have never written anything in C, and that I accept any help.
Comment 27 prrvchr 2023-07-26 20:18:13 UTC
Well I made the necessary changes, I compile (vive java and python) and I'll let you know...
Comment 28 prrvchr 2023-07-26 22:16:13 UTC
Good, the wizard is set, however the database is saved as an hsqldb base, this comes from the method getEmbeddedDatabase() line 290 in libreoffice/dbaccess/source/core/misc/dsntypes.cxx.
I'm afraid that it miss the url as a parameter to this function and I don't know where it is called from.
Thank you for your help....
Comment 29 prrvchr 2023-07-26 22:59:24 UTC
it looks like getEmbeddedDatabase() is called in libreoffice/dbaccess/source/ui/dlg/generalpage.cxx line 242.
But I think it's beyond my abilities, I give up...
Comment 30 Julien Nabet 2023-07-27 06:32:46 UTC
For the record, taking a look at:
https://opengrok.libreoffice.org/search?project=core&full=getEmbeddedDatabase&defs=&refs=&path=&hist=&type=&xrd=&nn=1&si=full&si=full

getEmbeddedDatabase:
- is declared in dbaccess/source/inc/dsntypes.hxx (from line 163)
- is implemented in dbaccess/source/core/misc/dsntypes.cxx (from line 290)
- is used in dbaccess/source/ui/dlg/generalpage.cxx (lines 242 and 265)
Comment 31 prrvchr 2023-07-27 09:32:55 UTC
If I follow Lionel's recommendations, I manage to have a Wizard with only two steps as it should be.

On the other hand, I can't understand the usefulness of the getEmbeddedDatabase() function.
It is this function that limits the connection urls to the two urls: sdbc:embedded:hsqldb and sdbc:embedded:firebird

If anyone can explain?
Comment 32 Lionel Elie Mamane 2023-07-27 09:44:52 UTC
From glancing at the code, it seems to me it is only used to set the _default_ value of the "embedded database" choice, not used to create the actual database implementation after the choice is made. If it were, one wouldn't have the choice between HSQLDB and Firebird (when experimental features are enabled).
Comment 33 prrvchr 2023-07-27 10:23:55 UTC
@Lionel

if I badly patch line 295 in file libreoffice/dbaccess/source/core/misc/dsntypes.cxxdsntypes.cxx with:

return "sdbc:embedded:sqlite";

When I choose Embedded SQLite driver in the Wizard when finished the odb is on the right url!!!
Comment 34 prrvchr 2023-07-27 10:49:32 UTC
I don't know anything about C, so what I'm saying is something to be taken with a grain of salt, but it seems that getEmbeddedDatabase() is used for two things:

- retrieve the connection url assigned to the data source (odb file) in file /home/prrvchr/github/libreoffice/dbaccess/source/ui/dlg/generalpage.cxx line 242
- ensure the presence of integrated driver to display or not the choice of integrated database in the wizard in file /home/prrvchr/github/libreoffice/dbaccess/source/ui/dlg/generalpage.cxx line 465 (which seems to me to be an error because this function always returns something anyway...)
Comment 35 prrvchr 2023-07-27 12:34:58 UTC
Alright, I think I messed up.

Lionel's recommendations are sufficient to take into account embedded databases in addition.

How to submit the correction?
Comment 36 Lionel Elie Mamane 2023-07-27 12:47:04 UTC
(In reply to prrvchr from comment #35)
> How to submit the correction?

Thank you for your contribution to LibreOffice. I take the liberty of assigning this bug to you since you take care of it.

Review for merge into LibreOffice happen on
https://gerrit.libreoffice.org/
Add me as reviewer so that I see it.

See
https://wiki.documentfoundation.org/Development/GetInvolved#Prepare_to_submit_patches
and
https://wiki.documentfoundation.org/Development/gerrit/setup
for the setup. And
https://wiki.documentfoundation.org/Development/GetInvolved#4._Submit_the_patch
for how to submit the patch.

Prior to merging the patch, we need a licence statement from you, see https://wiki.documentfoundation.org/Development/GetInvolved#License_statement
Comment 37 prrvchr 2023-07-27 13:11:36 UTC
> Thank you for your contribution to LibreOffice. I take the liberty of assigning
> this bug to you since you take care of it.

All the pleasure is mine, and especially thank to you I could not have done anything without you.

I think ./logerrit setup failed: it didn't generate my SSH key: how to ask it to recreate it?
Comment 38 prrvchr 2023-07-27 13:16:19 UTC
Ok, done you must delete ~/.shh directory...
Comment 39 prrvchr 2023-07-27 13:35:28 UTC
Ok it's done: https://gerrit.libreoffice.org/c/core/+/154989
Comment 40 prrvchr 2023-07-27 13:38:35 UTC
Once reviewed and accepted, to which version of LibreOffice will this correction apply?
Comment 41 Julien Nabet 2023-07-27 13:55:22 UTC
(In reply to prrvchr from comment #40)
> Once reviewed and accepted, to which version of LibreOffice will this
> correction apply?

Master branch (the by default branch when one's submit a patch) corresponds for the moment to the future 24.2 (the version which will be released in 2024/02).

If the patch is pushed, it's possible to cherry-pick it in other branches (eg : 7.6). Of course, it will require extra approvals.

For 7.5 branch, since we're on 7.5.5 version, I think it's far too late for a new feature but I may be wrong.
Comment 42 prrvchr 2023-07-27 14:24:40 UTC
Under Ubuntu 22.04 we are in 7.3.7... We will have to be patient before it arrives on the workstations...

Anyway, thanks for your help, and I'm glad I was able to solve this problem so quickly (thanks to Lionel...)

I think we can close this issue...
Comment 43 Julien Nabet 2023-07-27 15:06:12 UTC
(In reply to prrvchr from comment #42)
> Under Ubuntu 22.04 we are in 7.3.7... We will have to be patient before it
> arrives on the workstations...
> 
> Anyway, thanks for your help, and I'm glad I was able to solve this problem
> so quickly (thanks to Lionel...)
> 
> I think we can close this issue...

You can try https://launchpad.net/~libreoffice/+archive/ubuntu/ppa to have more recent LO versions.
Comment 44 Commit Notification 2023-07-28 07:15:48 UTC
Pierre committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/9c071c148afce0b95a16b36c27b709380d1130b5

tdf#156471: New database wizard - add generic support for embedded databases of unknown type

It will be available in 24.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 45 Julien Nabet 2023-07-28 07:24:53 UTC
Lionel: do you think it's ok for a cherry-pick in 7.6 branch?
Comment 46 prrvchr 2024-01-13 15:17:26 UTC
(In reply to Julien Nabet from comment #45)
> Lionel: do you think it's ok for a cherry-pick in 7.6 branch?

I would be really happy to have access to this change before version 24.2.0
Comment 47 Julien Nabet 2024-01-13 16:01:19 UTC
(In reply to prrvchr from comment #46)
> (In reply to Julien Nabet from comment #45)
> > Lionel: do you think it's ok for a cherry-pick in 7.6 branch?
> 
> I would be really happy to have access to this change before version 24.2.0

Since we're now at 7.6.4, I don't think it worths it.
Indeed, according to https://wiki.documentfoundation.org/ReleasePlan/7.6, next releases for 7.6 are for important bugfixes.

If you don't want to wait for 24.2 official release, you can try the prerelease (see  https://www.libreoffice.org/download/pre-releases/)

Now you can cherry-pick it on gerrit if you want, I'd put +1 since Lionel was ok.
Comment 48 Stéphane Guillou (stragu) 2024-04-15 11:55:19 UTC
- Pierre, can this be closed as "resolved - fixed" or does it need more work?
- Could someone describe the feature on the release notes, I'm sure I would not word it accurately: https://wiki.documentfoundation.org/ReleaseNotes/24.2#Base
Comment 49 prrvchr 2024-04-15 13:04:13 UTC
Yes it can be closed it's finished...
I want to explain what was done, how to do it?
Comment 50 Stéphane Guillou (stragu) 2024-04-15 14:55:01 UTC
(In reply to prrvchr from comment #49)
> Yes it can be closed it's finished...
So you can change the bug status to "resolved - fixed".
> I want to explain what was done, how to do it?
Log into the wiki, edit the page with a sentence. You can base it on other notes, see for example the use of the {{tdf|12345}} template to reference this bug report. Others can fix it up afterwards if needed :)
Thank you!
Comment 51 prrvchr 2024-04-15 15:30:02 UTC
Ok done, thank stragu...
Comment 52 Stéphane Guillou (stragu) 2024-04-15 15:31:12 UTC
(In reply to prrvchr from comment #51)
> Ok done, thank stragu...
Thank you! I fixed up a couple of things.
Comment 53 prrvchr 2024-04-15 15:37:44 UTC
(In reply to Stéphane Guillou (stragu) from comment #52)
> Thank you! I fixed up a couple of things.

I just saw you are much faster than me... thank you
Comment 54 Robert Großkopf 2024-04-16 06:22:49 UTC
One comment: Have tested the extension for SQlite yesterday. Don't know where the difference between LO 24.2.2.2 and older version like 7.6.4 should be.
Then I started today with a comment to a bug and HSQLDB database. Forgotten to uninstall the extension.
Opening an old HSQLDB when extension has been installed leads to loss of all data. No table is available any more in the old HSQLDB. In this special case no problem. Could download the example database again.

I would place a big warning on the download page of the extension: Don't use this extension if you have created HSQLDB databases and will use this databases any more. You will loss all your data.
Comment 55 prrvchr 2024-04-16 07:55:13 UTC
Hi Robert,

Oh yeah, what you're telling me is pretty creepy.

It is true that when opening an odb file, the extensions do not carry out any checks. The odb file is decompressed then the path of this decompression is given to the underlying database. I need to do more checking...

I will try to recognize the presence of at least one file...

Thank you for these tests, it reveals some surprising things
Comment 56 prrvchr 2024-04-16 08:06:27 UTC
A certain problem comes from the fact that these two extensions work on the same url (sdbc:embedded:hsqldb) while waiting for this fix to be implemented.

I think I need to stop this operation and it redirects users to a version 24.2 minimum.
Comment 57 prrvchr 2024-04-16 08:28:51 UTC
I'm working on H2 and Derby in embedded mode. I will have to update both HyperSQLOOo and SQLiteOOo extensions to support the changes in jdbcDriverOOo 1.3.1. I will assign a specific URL to each extension, and I will even create a new URL for HsqlDB in embedded mode in order to differentiate it from the HsqlDB driver 1.8 embedded by LibreOffice. This is what was planned in the end, I will do this as quickly as possible.

Although this problem is rather annoying, I think that the initial database is still in the odb file and intact. Normally even without backup you should be able to find this data.
Comment 58 Robert Großkopf 2024-04-16 08:46:53 UTC
(In reply to prrvchr from comment #57)
> 
> Although this problem is rather annoying, I think that the initial database
> is still in the odb file and intact. Normally even without backup you should
> be able to find this data.

I have opened both databases in a zip program.
This I could see in folder "database":

Both folders contain log, properties and script. But script in the "damaged" database file won't show any table definition.

Files backup and data only exist in the working version of HSQLDB. The "Damaged" version contains a file Test_HSQLDB (which is the name of the database) with the same size as the backup file in the working version. No data file is available there.
Comment 59 prrvchr 2024-04-16 09:15:59 UTC
So this means that there was indeed recompression when the odb file was closed.

I need to prevent this if the connection to the database was unsuccessful...

I will try to reproduce and correct this... Thanks for warning me...
Comment 60 Stéphane Guillou (stragu) 2024-04-16 09:29:03 UTC
@Pierre and @Robert: is there a risk for LO users who don't use the extension? If that's the case, please open a new report to track it separately. Sounds like a serious dataloss issue.
Comment 61 Robert Großkopf 2024-04-16 09:49:02 UTC
(In reply to Stéphane Guillou (stragu) from comment #60)
> @Pierre and @Robert: is there a risk for LO users who don't use the
> extension? If that's the case, please open a new report to track it
> separately. Sounds like a serious dataloss issue.

I have only got dataloss when installed extension and started a internal HSQLDB-database.

Have uninstalled the extension and could open the database without any problem.
Comment 62 Stéphane Guillou (stragu) 2024-04-16 11:57:11 UTC
(In reply to Robert Großkopf from comment #61)
> I have only got dataloss when installed extension and started a internal
> HSQLDB-database.
Thanks for clarifying!
Comment 63 prrvchr 2024-04-16 14:59:48 UTC
I just updated both extensions to solve part of the problem.

A problem remains if you create an HsqlDB embedded 1.8 database (an odt file) then try to open it with the HyperSQLOOo extension (HsqlDB 2.7.2).

HsqlDB opens the old version without knowing how to read it and without any errors... Normally I should expect a SQLexception.

I'm thinking about how to fix this... I'm going to watch with fredt.
Comment 64 prrvchr 2024-04-16 18:56:04 UTC
Well the new version of HyperSQLOOo 1.1.2 is capable of detecting a version problem (hsqldb 1.8), in fact you must connect with the ;create=false option so that the hsqldb driver throws an exception...

I put a warning message about this problem on the HyperSQLOOo home page.

A big thank you to @Robert...