Bug 138715 - Thunderbird Addressbook no longer connectable
Summary: Thunderbird Addressbook no longer connectable
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Base (show other bugs)
Version:
(earliest affected)
7.0.3.1 release
Hardware: x86-64 (AMD64) macOS (All)
: high major
Assignee: Not Assigned
URL:
Whiteboard: target:7.2.0
Keywords:
: 128977 138807 141493 154380 (view as bug list)
Depends on:
Blocks: 116636 Thunderbird-Interoperability
  Show dependency treegraph
 
Reported: 2020-12-07 17:04 UTC by Robert Großkopf
Modified: 2023-06-29 11:52 UTC (History)
14 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Robert Großkopf 2020-12-07 17:04:14 UTC
Since Thunderbird 78 the addressbook changed from abook.mab to abook.sqlite. So a connection couldn't be established with the direct connection in Base.

We have different possibilities:
- Write down "Old Thunderbird Addressbook abook.mab" for the connection instead of "Thunderbird Addressbook". But most of people will update Thunderbird and it won't work any more.
- We could try it with a connection to SQLite - but there is only working connection through ODBC without problems.
- We could remove the entry for "Tunderbird addressbook" in the list of datasources for Base.
Comment 1 [REDACTED] 2020-12-08 16:51:00 UTC
Repro 

Version: 7.0.3.1, Build ID: d7547858d014d4cf69878db179d326fc3483e082
CPU threads: 8; OS: Linux 5.3; UI render: default; VCL: kf5
Locale: en-US (en_US.UTF-8); UI: en-US, Calc: threaded

Thunderbird 78.5.1
Comment 2 Julien Nabet 2020-12-09 19:55:38 UTC
Robert you indeed had warned us about this about 1 year ago in tdf#128977 put in see also.
Badfully, for the moment nobody stepped forward to try to implement Sqlite driver.
Of course, it could be relevant for https://wiki.documentfoundation.org/Development/GSoC/Ideas but it would need a mentor. The pb is Lionel is the only Base dev expert but has less time now.

IMHO, the most reasonable/realistic thing would be to remove Thunderbird support and get rid of Mork at the same time.

Lionel/Alex: I put you in cc because with Robert, you're the most interested by Base issues.

BTW, I've just created a thread about it here:
http://document-foundation-mail-archive.969070.n3.nabble.com/template/NamlServlet.jtp?new_topic

Don't hesitate to comment of course.

Let's increase priority since:
- it's a blocker for those who want to use TB address book whatever env (Windows, Linux, Mac) if TB version is 78 min
- it's a regression (not in terms of code but in terms of functionnality)
- no workaround (except if we consider that downgrading TB is acceptable - not my opinion)
Comment 3 Julien Nabet 2020-12-09 20:21:31 UTC
I noticed abook.mab.bak and history.mab.bak in TB profile but even if we use them in LO:
1) it won't work on machines which never had TB with version < 78
2) it should be just readonly (since I suppose they've been generated only during TB migration). I suppose TB don't use them after migration when running

IMHO, it wouldn't worth it to try this way since it's kindda loose fallback.
Comment 4 Rene Engelhard 2020-12-10 17:43:42 UTC
> - We could try it with a connection to SQLite - but there is only working connection through ODBC without problems.

There is no sqlite driver in LO, so...
Can't we make LO use a system-wide installed SQLite ODBC driver?

Otherwise mork should probably be removed, yes: https://gerrit.libreoffice.org/c/core/+/107574
Comment 5 Robert Großkopf 2020-12-10 18:02:55 UTC
(In reply to Rene Engelhard from comment #4)
> 
> There is no sqlite driver in LO, so...
> Can't we make LO use a system-wide installed SQLite ODBC driver?
> 
There is no problem to use Sqlite through ODBC. Must also work with the Thunderbird addressbook.

There is reported only one (unconfirmed) bug: https://bugs.documentfoundation.org/show_bug.cgi?id=131238

For normal users the connection through ODBC isn't as easy as the direct connection with the old driver for Thunderbird.
Comment 6 Julien Nabet 2020-12-10 19:13:34 UTC
(In reply to Rene Engelhard from comment #4)
> > - We could try it with a connection to SQLite - but there is only working connection through ODBC without problems.
> 
> There is no sqlite driver in LO, so...
> Can't we make LO use a system-wide installed SQLite ODBC driver?
Do you mean to access TB address book by:
- using current entry (but behind the curtain there would need some "plumbering" to use SQLite ODBC - don't know it's feasible and/or not too difficult, just for the record, I wouldn't be able to implement this)
or 
- using ODBC entry explicitely (and removing current TB address book entry) and so add help in addition to release notes to advertise that to use TB addresse book, you'll got to use ODBC

> 
> Otherwise mork should probably be removed, yes:
> https://gerrit.libreoffice.org/c/core/+/107574
That's fast!! :-) I agree with removing Mork but first I would like Lionel's opinion since he's the only dev expert in Base part.
Comment 7 Julien Nabet 2020-12-10 19:16:34 UTC
(In reply to Robert Großkopf from comment #5)
> (In reply to Rene Engelhard from comment #4)
> ...
> There is reported only one (unconfirmed) bug:
> https://bugs.documentfoundation.org/show_bug.cgi?id=131238
There may be more if people use Sqlite ODBC to use their TB address book.

> For normal users the connection through ODBC isn't as easy as the direct
> connection with the old driver for Thunderbird.
+1
Comment 8 Rene Engelhard 2020-12-10 19:50:32 UTC
> For normal users the connection through ODBC isn't as easy as the direct connection with the old driver for Thunderbird.

That's why I said we need (if we want to keep explicit "TB AB" support instead of using ODBC with some "random" datasource which happens to be sqlite and the TB AB) some way to "plumber" LO with a sqlite ODBC driver. (That one imho also can just happen to be on the system imho, if the stuff failed gracefully and tells the user what to do "install SQLIte ODBC driver")
Comment 9 Rene Engelhard 2020-12-10 19:51:54 UTC
https://packages.debian.org/search?keywords=libsqliteodbc for example...
Comment 10 Rene Engelhard 2020-12-10 20:33:03 UTC
*** Bug 138807 has been marked as a duplicate of this bug. ***
Comment 11 Rene Engelhard 2020-12-10 20:34:58 UTC
*** Bug 128977 has been marked as a duplicate of this bug. ***
Comment 12 Commit Notification 2020-12-13 16:30:16 UTC
Rene Engelhard committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/ee9ed2192b56c98e5b8ee9890ddb4c533117332a

tdf#138715 remove mork driver

It will be available in 7.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 13 Julien Nabet 2020-12-13 19:47:17 UTC
Let's remove target since the commit wasn't a fix but just some Mork removing with some Mozillabootstrap.
There are still some pieces of Mozillabootstrap + all thunderbird part which should be removed since it's broken now.
I must recognize I don't understand well what's Mozillabootstrap so can't respond quite confidently here: https://gerrit.libreoffice.org/c/core/+/107574

<digression>I noticed too some Firefox access with wizard assistant from Writer.
I think "Firefox" is wrong here and it had to be Mozilla when it was a Mozilla suite (which included a browser, a mailer, etc.).
Mozilla suite doesn't exist anymore and has been replaced by Seamonkey.
I don't know if it's used broadly and if it still Mork or not.
Anyway, I think it should be removed too.
</digression>

Rene/Stephan: I put you in cc (Rene, obviously because you created the patch to remove biggest part of Mork, Stephan because you know well about published interfaces, how to avoid compatibility breakings, etc.) (of course, feel free to uncc yourself if it bothers you/have no time/whatever...)
Comment 14 Commit Notification 2020-12-14 15:16:41 UTC
Jan-Marek Glogowski committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/c1625e807ff7d94d0598b2aa86735fb94b829288

tdf#138715 remove mork strings and install targets

It will be available in 7.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 15 Xisco Faulí 2020-12-17 15:05:15 UTC
Should this issue be closed as RESOLVED FIXED ?
Comment 16 Julien Nabet 2020-12-17 17:01:13 UTC
(In reply to Xisco Faulí from comment #15)
> Should this issue be closed as RESOLVED FIXED ?

Not yet, there are still some issues to deal with.
Quote from http://document-foundation-mail-archive.969070.n3.nabble.com/ESC-meeting-agenda-2020-12-17-16-00-Berlin-time-tt4292917.html
"
Removing Mork => removing Thunderbird Address book but there's been too
mozab removing (I'm ok with this), should we remove all related idl eg:
offapi/com/sun/star/mozilla/MozillaBootstrap.idl or
offapi/com/sun/star/mozilla/XProfileDiscover.idl ?

Also there's the wrong named "Firebird" Address Data Source in Writer
wizards, should we remove this (or rename it if it works with Mozilla
Suite/Seamonkey but does it worth it?), knowing that the only remaining
entry would be "Other" (not very UI friendly...)?
Of course, all this should be documented in 7.2 release notes."
Comment 17 Commit Notification 2020-12-19 08:53:14 UTC
Julien Nabet committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/746da2e47d92c9de4d6bcec22155c34cece5c331

Related tdf#138715: kill last Mork refs

It will be available in 7.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 18 Julien Nabet 2020-12-19 09:41:32 UTC
The last patch is just removing last pieces of Mork=>removing target
Comment 19 Commit Notification 2020-12-19 17:05:08 UTC
Adolfo Jayme Barrientos committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/help/commit/546911b845a41f1c1a7188985f6237767114a536

tdf#138715 Remove description of “Firefox” import option from Help
Comment 20 Adolfo Jayme Barrientos 2020-12-19 17:07:40 UTC
Whiteboard target can stay — a WONTFIX resolution by removing code is still a resolution ;)
Comment 21 Adolfo Jayme Barrientos 2020-12-19 18:03:19 UTC
Oops, didn’t see comment 16
Comment 22 Julien Nabet 2021-04-05 09:50:26 UTC
*** Bug 141493 has been marked as a duplicate of this bug. ***
Comment 23 Alex Thurgood 2021-04-21 09:02:07 UTC
So what is the solution for macOS ?

Currently, even Collabora Office  

Version : 6.4-20
Build ID : 8f9231b97d06858b598c6fa819546ca3391d11c7
Threads CPU : 8; OS : Mac OS X 11.2.3; UI Render : par défaut; VCL: osx; 
Locale : fr-FR (fr_FR.UTF-8); Langue IHM : fr-FR
Calc: threaded

fails to find a TB abook.mab file (even after pointing LO to the existence of the Thunderbird.app bundle.

Furthermore, ODBC access to anything is also broken on macOS : bug 138990

Trying to use the Contacts.app as the datasource leads to repeatable crash / recovery cycles : bug 126961

There is thus no solution for a Mac user to have access via LO to any of the most commonly used addressbooks.

Proposed solutions on the back of a postage stamp.
Comment 24 Alex Thurgood 2021-06-18 12:57:36 UTC
Bug stil present in

Version: 7.1.4.2 / LibreOffice Community
Build ID: a529a4fab45b75fefc5b6226684193eb000654f6
CPU threads: 4; OS: Mac OS X 10.16; UI render: default; VCL: osx
Locale: fr-FR (fr_FR.UTF-8); UI: fr-FR
Calc: threaded
Comment 25 Heinz 2022-07-30 17:06:35 UTC
Thunderbird 102.1.0 is still not connectable with LibreOffice 7.3.4
Comment 26 Heinz 2022-07-30 17:10:48 UTC Comment hidden (noise)
Comment 27 Heinz 2022-07-30 17:11:36 UTC Comment hidden (obsolete)
Comment 28 Heinz 2022-07-30 17:17:16 UTC Comment hidden (noise)
Comment 29 Robert Großkopf 2022-07-30 17:49:34 UTC
(In reply to Heinz from comment #28)
> Thunderbird 102.1.0 ist still not connectable with LibreOffice 7.3.4

Why did you write 4 comments - all the same intention?

You could connect through ODBC to Thunderbird addressbook, because it's an SQlite database file. Could be we need a better implementation for SQlite.
Comment 30 Alex Thurgood 2022-12-10 13:03:39 UTC
(In reply to Robert Großkopf from comment #29)
> (In reply to Heinz from comment #28)


> > Thunderbird 102.1.0 ist still not connectable with LibreOffice 7.3.4> 
> You could connect through ODBC to Thunderbird addressbook, because it's an
> SQlite database file. Could be we need a better implementation for SQlite.

Not on MacOS.

At least access to the native macOS address book is now working again, but nothing for Thunderbird. 

The irony is that macOS comes with direct sqlite3 support built into the OS.
Comment 31 Alex Thurgood 2022-12-16 09:19:56 UTC
(In reply to Alex Thurgood from comment #30)
> (In reply to Robert Großkopf from comment #29)

> Not on MacOS.
> 
> At least access to the native macOS address book is now working again, but
> nothing for Thunderbird. 
> 
> The irony is that macOS comes with direct sqlite3 support built into the OS.


According to bug 93094, one can now also use the Xerial JDBC driver version >= 4.40.0.0), provided that one can set an advanced property setting in the driver properties, so that might also provide a way forward. Unfortunatel, that won't help for the AppStore version of LO, which has no Java support at all.
Comment 32 Alex Thurgood 2022-12-16 10:19:46 UTC
Further to comment 31, on macOS Ventura 13.0.1 (aarch64), I can report that it is possible to use the Xerial JDBC driver to setup a JDBC connection to create an ODB file that accesses a Thunderbird sqlite address book.

Problems to be resolved

1) The sqlite file is locked against reading data from it if Thunderbird is running. This is true also when using sqlite CLI from the terminal. Apparently, the only way to retrieve the data from the file is to shutdown Thunderbird.

2) Thunderbird stores literally a gazillion different sqlite databases in the profile directory, e.g.

/Users/alex/Library/Thunderbird/Profiles/xxxxxxxxx.default

with a number of versions (v1,v2,v3, etc) for each one.

Which one is the user supposed to choose ?

I had to go through several before I found one that contained any useful data. This is not something that normal users will understand or be able to comprehend, as it involves navigating through a hidden folder (Library) and seeking out an appropriate abook sqlite file from the list of files in that folder. 


3) The sqlite database appears to comprise 3 tables :

list_cards : list (TEXT) PRIMARY KEY; card (TEXT) PRIMARY KEY

lists : uuid (TEXT) PRIMARY KEY; localId (INT); name (TEXT); nickName (TEXT); description (TEXT)

properties : card (TEXT); name (TEXT); value (TEXT)


I found some kind of exploitable data in the "properties" table.
However, the data is not organised in an obviously exploitable way, for example:

the "card" field contains what appears to be a unique character string value (hex?) for any given contact;

the "name" field contains the names of the fields that are found in the address book, each address book field name having its own separate tuple - for example, "DisplayName", "PreferMailFormat","CellularNumber", "PrimaryEmail", "LastModifiedDate", etc;

the "value" contains the character string, or value, that corresponds to the field name of the address book having the assigned card index value.


Presumably, Thunderbird manages to tie all of these tables together in some way internally, so that the user is presented with a homogenous experience.

A basic simple table access to any given abook.sqlite via LO wouldn't do this, and would be difficult to exploit without the added nuts and bolts code to make things useful.
Comment 33 Robert Großkopf 2022-12-16 10:58:15 UTC
(In reply to Alex Thurgood from comment #32)
There a 2 different addressbooks in Thunderbird, which will show the content most people needed:

abook.sqlite → will show the current created data of addressbook. 
All other of this called abook.v2.sqlite, abook.v3.sqlite should be old data backups and not current created books.

history.sqlite → will show all addresses available in Thunderbird, not added to your personal addressbook.

The following code will help for a query, which could be used to work with addresses in abook.sqlite. It will only show "name", "mail" and "group":

SELECT "a"."card", 
"a"."value" "DisplayName", 
( SELECT "value" FROM "properties" WHERE "card" = "a"."card" AND "name" = 'PrimaryEmail' ) "PrimaryEmail", 
"lists"."name" "list" 
FROM "list_cards", "properties" AS "a", "lists" 
WHERE "list_cards"."card" = "a"."card" AND "lists"."uid" = "list_cards"."list" AND "a"."name" = 'DisplayName'
Comment 34 Alex Thurgood 2022-12-16 11:45:53 UTC
(In reply to Robert Großkopf from comment #33)

 
> SELECT "a"."card", 
> "a"."value" "DisplayName", 
> ( SELECT "value" FROM "properties" WHERE "card" = "a"."card" AND "name" =
> 'PrimaryEmail' ) "PrimaryEmail", 
> "lists"."name" "list" 
> FROM "list_cards", "properties" AS "a", "lists" 
> WHERE "list_cards"."card" = "a"."card" AND "lists"."uid" =
> "list_cards"."list" AND "a"."name" = 'DisplayName'

Thanks for the suggestion, but that query returns no results for me with abook.sqlite, probably because that file has no data in list_cards or lists, and the NULL resultset is ignored.



In Thunderbird, my personal addressbook shows something like 857 entries. I don't know why lists and list_cards would be empty, whereas properties isn't.
Comment 35 Robert Großkopf 2022-12-16 13:41:58 UTC
(In reply to Alex Thurgood from comment #34)
> (In reply to Robert Großkopf from comment #33)
> 
> In Thunderbird, my personal addressbook shows something like 857 entries. I
> don't know why lists and list_cards would be empty, whereas properties isn't.

I have only groups for persons in my addressbook. So query would show all results.
Take this with LEFT JOIN:

SELECT "a"."card", "a"."value" "DisplayName", ( SELECT "value" FROM "properties" WHERE "card" = "a"."card" AND "name" = 'PrimaryEmail' ) "PrimaryEmail", "lists"."name" "list" FROM "properties" AS "a" LEFT JOIN "list_cards" ON "list_cards"."card" = "a"."card" LEFT JOIN "lists" ON "lists"."uid" = "list_cards"."list" WHERE "a"."name" = 'DisplayName'
Comment 36 Julien Nabet 2023-03-26 11:48:31 UTC
*** Bug 154380 has been marked as a duplicate of this bug. ***