Bug 121958 - Replace the Bibliography Database by a dialog to manually input new sources and to fetch from online sources via identifiers for citations/bibliography/references
Summary: Replace the Bibliography Database by a dialog to manually input new sources a...
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Writer (show other bugs)
Version:
(earliest affected)
Inherited From OOo
Hardware: All All
: medium enhancement
Assignee: Not Assigned
URL:
Whiteboard:
Keywords:
: 94418 (view as bug list)
Depends on:
Blocks: Bibliography 125045
  Show dependency treegraph
 
Reported: 2018-12-07 12:27 UTC by Pedro
Modified: 2025-03-27 07:43 UTC (History)
11 users (show)

See Also:
Crash report or crash signature:


Attachments
The Manage Sources Dialog from Word (equivalent to Writer's Bibliography Database) (192.47 KB, image/png)
2018-12-07 12:27 UTC, Pedro
Details
Zotero New Item drop-down menu. When creating a new item, the information tab below shifts allows an user to manually introduce information in the relevant fields. (114.70 KB, image/png)
2018-12-07 12:28 UTC, Pedro
Details
Zotero allows to add sources from online origin via identifiers. (98.33 KB, image/png)
2018-12-07 12:32 UTC, Pedro
Details
Current Bibliography Database is a visual and functionality mess (114.30 KB, image/png)
2018-12-07 12:33 UTC, Pedro
Details
Mock-up of a potential dialog to replace Bibliography database 1 (110.97 KB, image/png)
2018-12-07 15:26 UTC, Pedro
Details
Mock-up of a potential dialog to replace Bibliography database 2 (211.32 KB, image/png)
2018-12-07 15:26 UTC, Pedro
Details
Zotero extension integrated by default in Softmaker Textmaker (82.53 KB, image/png)
2022-03-27 16:49 UTC, Pedro
Details
Zotero addon in the menubar (112.28 KB, image/png)
2022-03-27 16:54 UTC, Pedro
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Pedro 2018-12-07 12:27:10 UTC
Created attachment 147343 [details]
The Manage Sources Dialog from Word (equivalent to Writer's Bibliography Database)

The current Biblioghraphy Database to manage bibliography is a complete mess, unfriendly for users and nearly useless to perform any actual citation/bibliography management. It does not alow to select citation style and the excess of columns that are shown is completely overwhelming and visually confusing to edit the citation styles.

I would like to propose that a revamped dialog is adopted that allow an user to more easily identify what sources are available in a document. 
Namely, the system present in Word allows an user to search by relevant info of a citation (author, title, tag or year) and to select a source from a Master list to a selected list.

Furthermore, it provides a preview of how citations and the bibliography looke like with the selected citation style (allthough it does not allow you to select citation style in this dialog - this could be implemented in a re-work of the Writer management system).

An user can select New to manually add new sources or Edit existing sources.

Zotero uses a superior system, with a drop-down menu that allows to specify if it is:
An article in a journal
An article in a scientific journal
A Document
A book
A section of a book
Etc, etc, etc

Furthermore, Zotero also adds sources from online origin via identifiers. These identifier are DOI, PMID, ISBN or arXiv ID. 

When an user provides one of these identifier codes, a new source is created with all the fields with the relevant info present in the identifier source automatically filled.
In the modern world, this is the ESSENTIAL feature for a citation/bibliography manager since it saves an enormous amount of time for an user to fill information about citation/bibliography sources.

Therefore I propose that:
1 - the current Bibliography Database system is replaced by a more UI friendly dialog, 
2 - that it's name is changed to Citation/Bibliography Manager,
3 - that this dialog allows to manually add sources and fill information for these sources,
4 - that this dialog allows an user to use identifiers to automatically add sources from online origin,
5 - that allows an user to select different citation styles and add different citation styles besides the pre-installed ones from the CSL repository.

These are the barebones features of a decent citation/bibliography management systems. 
Software like Mendeley or Zotero offer plenty of other features that are unnecessary in Writer. However, if Writer is to provide a manager that is not stuck in the 19990's then it needs these improvements. Otherwise remove the Bibliography manager and simply include a dialog that directs a user to open source reference managers like Zotero.
Comment 1 Pedro 2018-12-07 12:28:50 UTC
Created attachment 147344 [details]
Zotero New Item drop-down menu. When creating a new item, the information tab below shifts allows an user to manually introduce information in the relevant fields.
Comment 2 Pedro 2018-12-07 12:32:13 UTC
Created attachment 147345 [details]
Zotero allows to add sources from online origin via identifiers.

Adding a source from an online identifier automatically fills the info fields for that source.
Comment 3 Pedro 2018-12-07 12:33:10 UTC
Created attachment 147346 [details]
Current Bibliography Database is a visual and functionality mess
Comment 4 Pedro 2018-12-07 15:26:00 UTC
Created attachment 147352 [details]
Mock-up of a potential dialog to replace Bibliography database 1

Mock of first tab of a potential dialog for a overhauled Bibliography manager.
Comment 5 Pedro 2018-12-07 15:26:23 UTC
Created attachment 147353 [details]
Mock-up of a potential dialog to replace Bibliography database 2
Comment 6 Dieter 2018-12-07 15:36:26 UTC
Great ideas! I support it!
Comment 7 Pedro 2018-12-07 15:38:48 UTC
Besides UXEval this requires DevEval since it would involve a substantial amount of work.

For devs:
- Remove current Database,
- Create a new system to generate citation/bibliography source lists,
- Add most commonly used citation styles (Zotero has 17 default ones, I would propose having the same),
- Connect to the CSL project repository of citation styles (mentioned on bug 121945)

For UI:
- Better define how to create a Bibliography Manager dialog.

I attached an initial mock-up of how think such a dialog should look and what functionalities it should have.
Comment 8 Heiko Tietze 2018-12-11 14:09:39 UTC
Kind of duplicate to bug 94418 but let's keep it open.
Comment 9 Cor Nouws 2019-02-05 20:22:34 UTC
*** Bug 94418 has been marked as a duplicate of this bug. ***
Comment 10 Pedro 2022-03-27 16:49:08 UTC
There are two reference managers that could be promoted by LibreOffice as reference managers by integrating them. Only one of them is open-source.
I am excluding Jabref because that one mostly focuses on LaTex text editing and not document editing software like Writer and Word.
Most other reference managers are outdated and with no development for a few years.

The two reference managers are Zotero and Mendeley. Mendeley was acquired by Elsevier and is closed source. I would be very against integrating it, since as an academic I very much disapprove Elsevier's business model.

That leaves Zotero. There's also an examplo of Zotero having integrated their extension in word editing software. Namely in Softmaker's Office Textmaker. I believe that for previous compatibility sake they left the old bibliographic reference management system on Textmaker.

However, according to this topic in Zotero forum:
https://forums.zotero.org/discussion/89804/softmaker-office-cannot-modify-citations

The integration of the Zotero addon by default was done exlcusively by Softmaker's devs. I doubt any help will come from Zotero developers if such an option was followed.
Comment 11 Pedro 2022-03-27 16:49:58 UTC
Created attachment 179142 [details]
Zotero extension integrated by default in Softmaker Textmaker

In their Ribbon UI.
Comment 12 Pedro 2022-03-27 16:54:35 UTC
Created attachment 179143 [details]
Zotero addon in the menubar

The addon neatly integrated into the menubar.

This would be the blueprint for integrating the Zotero addon in Writer.
Pros of having it pre-installed like in Textmaker:
1 - Incentivize users to use a properly maintained reference manager,
2 - Incentivize users to use a reference manager with interoperability,
3 - Incentivize users to use a reference manager that will create an external library, or a library that can be easily imported/exported and used in other documents.

Cons of having it pre-installed:
1 - The maintenance of this integration would fall squarely on LibreOffice developers. If anything broke, LibreOffice devs would have to fix it.
Comment 13 Alex Thurgood 2024-02-16 08:19:41 UTC
(In reply to Pedro from comment #12)
> Created attachment 179143 [details]
> Zotero addon in the menubar
> 
> The addon neatly integrated into the menubar.
> 

Out of interest, where is the data for the bibliography stored ? My understanding was that Textmaker used SQLite databases, e.g. for mailmerge, is this the same here with the Zotero binding ? If so, does the data get written to a sqlite db somewhere ?

Also, if the functionality requires built-in SQLite support, then, a whole new consideration needs to be given as to how one would get that working reliably in LO.
Comment 14 V Stuart Foote 2024-02-16 13:22:44 UTC
(In reply to Alex Thurgood from comment #13)
> 
> Out of interest, where is the data for the bibliography stored ? My
> understanding was that Textmaker used SQLite databases, e.g. for mailmerge,
> is this the same here with the Zotero binding ? If so, does the data get
> written to a sqlite db somewhere ?
> 
> Also, if the functionality requires built-in SQLite support, then, a whole
> new consideration needs to be given as to how one would get that working
> reliably in LO.

Zotero and its API uses an embedded SQLite instance to hold its data. That can be fully local (in Zotero's user profile) or linked references or a mix of both.

We'd have to link to that as a data source for Base/Bibliography or use the Zotero API in a plug-in.
Comment 15 Eyal Rozenberg 2025-03-25 20:58:37 UTC
Is the current bibliography database actually built on LO Base capabilities? And - does it have an active/respnsive maintainer?

Also, about Zotero integration:

* Can we possibly arrange that with more "dogfooding"? i.e. for aspects of the bibliography data to be viewed and edited as an LO Base database, or as Calc spreadshet?
* Is Zotero instrumentable for use as a library?
* If so, at what level? What does the API look like?
* Is the Zotero project _interested_ in integration with LibreOffice?
* Given how Zotero is (also) a separate and independent app - would it not make sense to have this feature be optional, either as an extension, or at least a separately-installable package alongside the rest of LO?
Comment 16 Heiko Tietze 2025-03-26 07:34:59 UTC
The bibliography is a database. The module Base can be used to show the content and to create various views. We do not have a dedicated maintainer (for almost every feature) but in case of Base and bibliography also not much development. Zotero integration is blocked by Java that we cannot use in the default configuration. Plus, it is a 3rd party tool which should be kept as such.
Comment 17 V Stuart Foote 2025-03-26 13:08:42 UTC
(In reply to Heiko Tietze from comment #16)
> ... Zotero integration is blocked by Java that we cannot use
> in the default configuration. Plus, it is a 3rd party tool which should be
> kept as such.

No, only the current Zotero plug-in extension (.oxt--build bundled with the desktop application release build) requires Java. 

Otherwise IIUC actual Zotero's API is JavaScript .js based and local content held in an SQLite instance, with the API providing access to WWW personal and global data stores.

So for our needs the Zotero API could be wrapped in Python and complied for packaging with LibreOffice, but needing to reimplement the UI and data formatting features the plug-in already provides.

The greater issue would be handling the local SQLite instance the Zotero API requires. And replicating Zotero table space to HQSQL table space *would* still require Java.

To avoid dependence on Java (which is really just a packaging concern) we probably would need to either replicate the Zotero table space into FirebirdSQL, or somewhat more reasonable to go ahead and implement the SQLite instance needed to host Zotero for our Bibliography DB needs.

IIRC back in the day, a number of the devs had advocated for having SQLite available, not to replace HQSQL (the Firebird remains best non-Java SQL implementation) but for internal use for configuration and plug-in support.
Comment 18 Heiko Tietze 2025-03-27 07:43:49 UTC
We discussed the topic in the design meeting. Zotero and co. are great tools, with likely a lot of development power behind. We should not reinvent wheels and implement another citation manager while existing work well (and just lack of a better integration).

However, fetching references from a free (!) external source could be done on top of our bibliography management. It would simplify the workflow significantly. Such enhancement is welcome.