Bug 101174 - Enable means to change Formula Editor default font from OpenSymbol via Expert Configuration
Summary: Enable means to change Formula Editor default font from OpenSymbol via Expert...
Status: ASSIGNED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Formula Editor (show other bugs)
Version:
(earliest affected)
Inherited From OOo
Hardware: All All
: medium enhancement
Assignee: dante19031999
URL:
Whiteboard:
Keywords:
: 67152 111700 127414 127553 133597 (view as bug list)
Depends on:
Blocks: 32362 Formula-Editor Fonts
  Show dependency treegraph
 
Reported: 2016-07-28 13:17 UTC by V Stuart Foote
Modified: 2020-11-17 15:23 UTC (History)
16 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description V Stuart Foote 2016-07-28 13:17:04 UTC
Current the Math Formula Editor is hard coded to use OpenSymbol [1] and assigned variables FONTNAME_MATH and then as FNTNAME_MATH [2] and used as FNT_MATH [3][4] 

There are a number of high quality font faces that have complete coverage of the Greek and Character Like symbols that the formula editor makes use in its specialized Symbols catalog, e.g. Cambria Math, Asana Math, STIX, etc. Any of which might be preferred by users as functional replacements to the bundled OpenSymbol font.

Suggest an ability in Expert Configuration dialog to change the font assigned to FONTNAME_MATH, or used as FNT_MATH would be useful. It might be grouped with the other org.openoffice.Office.Math StandardFormat stanzas.

This would be as an alternative to the general Tools -> Options -> Fonts: Replacement Table GUI where the replacement is global for all use of a font, OpenSymbol in this case. And if it proves popular, the ability to change the default font could later be added to the GUI settings panel for Math.

=-ref-=
[1] http://opengrok.libreoffice.org/xref/core/starmath/inc/types.hxx#24
[2] http://opengrok.libreoffice.org/xref/core/starmath/inc/format.hxx#32
[3] http://opengrok.libreoffice.org/xref/core/starmath/source/format.cxx#74
[4] http://opengrok.libreoffice.org/search?q=FNT_MATH&project=core
Comment 1 V Stuart Foote 2016-07-28 17:12:22 UTC
Definitions for the other Greek symbols %alpha... %gamma in the Formula Editor symbol catalog look to be defined in:

http://opengrok.libreoffice.org/xref/core/officecfg/registry/data/org/openoffice/Office/Math.xcu

Glyphs refered to by decimal equivs of their Unicode codepoints, but since the .XCU has a hardcoded OpenSymbol font Name attribute set it may impact ability to have a simple attribute change set from Expert Configuration.
Comment 2 Khaled Hosny 2016-11-11 23:34:47 UTC
Just changing the font is not enough, may symbols are hardcoded to use PUA code points from OpenSymbol, so changing the font will give you garbage. This needs to be fixed first.
Comment 3 V Stuart Foote 2016-11-12 00:46:41 UTC
OK, so IIUC use of the font replacement table work because the glyph has already been selected and then the replacement can proceed?

Should the OpenSymbol font be reworked to eliminate any direct use of the PUA defined glyphs?
Comment 4 Mike Kaganski 2016-11-12 07:19:37 UTC
I'm not sure it's possible. To have correct display, glyph's size should be specifically chosen to be usable stretched, stacked etc. Also, we stretch characters where they should be composed of a number of parts to display OK (braces/parens, etc) - now stretching makes glyphs thin etc. Fixing this requires changing current composition of glyphs. And if we go this route, then substitute should have these glyph parts available at the same codepoints as well.
Comment 5 V Stuart Foote 2016-11-12 16:31:48 UTC
Sure, and bug 32362 suggests the formula editor must go this direction.
 but doesn't all of this require we have the U+239B -> U+23B7 glyphs available to use? Not yet the case in OpenSymbol, but common in other symbol fonts.
 
Seems like we should dump any hard coded use of OpenSymbol's PUA glyphs, and use Unicode near term in preparation.

And once we move to using the appropriate Unicode glyps, doing proper OpenType Math table shaping seems advisable as in bug 103680.

Problem of the PUA assignment aside, an effort here to provide UI to be able to change the default font seems reasonable, and would survive any refactoring of the sm math internals.
Comment 6 john 2017-03-15 06:31:43 UTC
My comment would be that it's essential that we be able to change the font of our formula, independent of this immediate issue with the font metrics of OpenSymbol. Agree with Stuart's comments about the need to move away from any/all hard-coded dependencies on a particular font. But in the interim -- switch to a well-behaved font instead!!
Comment 7 V Stuart Foote 2017-05-08 23:50:04 UTC
*** Bug 67152 has been marked as a duplicate of this bug. ***
Comment 8 V Stuart Foote 2017-08-11 19:09:48 UTC
*** Bug 111700 has been marked as a duplicate of this bug. ***
Comment 9 Volga 2017-08-11 19:36:09 UTC
(In reply to V Stuart Foote from comment #5)
> Seems like we should dump any hard coded use of OpenSymbol's PUA glyphs, and
> use Unicode near term in preparation.
I suggest we can replace such hard coded PUA code points to Unicode equivalences firstly, and then create a converter to replace them when LO imported a formulae including such glyphs.
Comment 10 V Stuart Foote 2019-09-07 02:37:30 UTC
*** Bug 127414 has been marked as a duplicate of this bug. ***
Comment 11 V Stuart Foote 2019-09-15 16:17:41 UTC
*** Bug 127553 has been marked as a duplicate of this bug. ***
Comment 13 V Stuart Foote 2020-06-02 15:40:08 UTC
*** Bug 133597 has been marked as a duplicate of this bug. ***
Comment 14 dante19031999 2020-07-21 14:38:21 UTC
I'm working on this matter. Adding utf32 support for starmath in order to be able to solve a lot of requests. Changing opensymbol is hardcore because it'a a #define where the name is written, but should be possible. Anyway, I would say it's important to be able to get rid of it. Symbols witch are not in opensymbol are not correctly rendered on math. This affects for example resolution of tdf#49990 for example. Would try to use STIX2 or other mentioned by V Stuart Foote before falling onto OpenSymbol.
Comment 15 V Stuart Foote 2020-07-24 12:22:41 UTC
(In reply to dante19031999 from comment #14)
> I'm working on this matter. Adding utf32 support for starmath in order to be
> able to solve a lot of requests. 

@Dante, great to have you hacking at this. As Mike K. and Kahaled point out there is a lot of the PUA mappings in StarSymbol/OpenSymbol that is hard coded into the sm module for composing the sm nodes. Those mappings will need to be reworked to use the Unicode (BMP & SMP) directly.

Assigning specific font as here is less of an issue--though would be nice--as os/DE fallback should protect most usage.
Comment 16 Mike Kaganski 2020-07-24 12:31:11 UTC
(In reply to V Stuart Foote from comment #15)
> Assigning specific font as here is less of an issue--though would be
> nice--as os/DE fallback should protect most usage.

If I understand this as "having a fallback is not important" (which I may easily misunderstand, sorry), then that's very wrong. I would love to see this fixed; but for Math, the symbols are the crucial part, and not having a font to provide them on a system would make Math broken. So - any fix to this should make sure that we provide a font (I suggest to keep OpenSymbol for that) with the full repertoire of glyphs for Math. Currently we bundle it in all TDF packages, and have it in a special directory in installdir, where it is taken from if not installed in OS. That should remain. Also, the glyphs that are not in some PUA positions should still be available there in OpenSymbol, and also on proper Unicode positions: to allow users to have old documents that use that font for any different purpose (e.g., they could had inserted symbols as bullets in Writer).
Comment 17 Mike Kaganski 2020-07-24 12:32:40 UTC
(In reply to Mike Kaganski from comment #16)
> Also, the glyphs that are not in some PUA positions should still be available there

Sorry, a thinko. Should have been:

Also, the glyphs that are in some PUA positions should still be available there
Comment 18 V Stuart Foote 2020-07-24 12:55:32 UTC
(In reply to Mike Kaganski from comment #16)
> If I understand this as "having a fallback is not important" (which I may
> easily misunderstand, sorry), then that's very wrong. 

Unfortunately you did (but then I was not clear). What I meant was that a shift to fully Unicode based composition of sm nodes--system provided font fallback should protect against missing coverage in any font selected for formula editor use.

The PUA mappings need to be eliminated.  For example, bug 131160#c3 and bug 108697
Comment 19 Mike Kaganski 2020-07-24 13:06:57 UTC
(In reply to V Stuart Foote from comment #18)
> (In reply to Mike Kaganski from comment #16)
> > If I understand this as "having a fallback is not important" (which I may
> > easily misunderstand, sorry), then that's very wrong. 
> 
> Unfortunately you did (but then I was not clear).

Sorry for that; still I think that comment 16 has important things to consider regardless.

> The PUA mappings need to be eliminated.  For example, bug 131160#c3 and bug
> 108697

And what happens then with existing documents using those codepoints? We must not break them.
Comment 20 V Stuart Foote 2020-07-24 14:45:55 UTC
(In reply to Mike Kaganski from comment #19)
> > The PUA mappings need to be eliminated.  For example, bug 131160#c3 and bug
> > 108697
> 
> And what happens then with existing documents using those codepoints? We
> must not break them.

I guess, and two major ones would be default quick select Bullets:
Current PUA -> Unicode
E00C -> U+25C6 BLACK DIAMOND
E00A -> U+25A0 BLACK SQUARE

But while those could be mapped onto their correct Unicode--they and others in common use could be kept also at their PUA assignment. There already are dozens that are duplicated.

On the other hand, since OpenSymbol *really* needs a major clean up to bring its mappings current with Unicode version 13, the STIX2 Math font could instead be the better font to distribute--if the residual PUA and linkage to StarSymbol/OpenSymbol could be redirected, and the font metrics for STIX2 accommodated in the sm node logic and any other locations in the UI.

On one side we have the legacy--but unsupported OpenSymbol needing font design work. On the other multiple license appropriate font projects offering well maintained symbol coverage fonts.

I think we can cobble together the few glyphs Dante needs to round out OpenSymbol (or even agressively expand its coverage) but are we really best served by that?
Comment 21 Mike Kaganski 2020-07-24 14:52:26 UTC
(In reply to V Stuart Foote from comment #20)
> I think we can cobble together the few glyphs Dante needs to round out
> OpenSymbol (or even agressively expand its coverage) but are we really best
> served by that?

Yes. We have to provide backward compatibility. Which means that even if we decide to use another well-maintained font, we still must provide the old font for use in existing documents (which must definitely *not* change their look after this change, even to something that you, me, or anybody else considers a better-looking alternative). Which in turn means we must make this old font to work with new code that expects to find some things in new places; which means that we have to fix it ... which means that deciding to use another font does not allow us to avoid any burden, only would add (a little (?)) to that burden.
Comment 22 V Stuart Foote 2020-07-24 16:23:56 UTC
(In reply to Mike Kaganski from comment #21)
> Yes. We have to provide backward compatibility. ...

Found some of that ugliness just now looking at bug 47679 and the StarSymbol/OpenSymbol PUA mapping tables [1].

=-ref-=

[1] https://opengrok.libreoffice.org/xref/core/unotools/source/misc/fontcvt.cxx?r=a7f74d5d
Comment 23 dante19031999 2020-11-10 14:24:33 UTC
If we change the font, we will have to deprecate the old one. We will have to change things like the sqrt bar height and much others.
Comment 24 V Stuart Foote 2020-11-10 16:15:06 UTC
(In reply to dante19031999 from comment #23)
> If we change the font, we will have to deprecate the old one. We will have
> to change things like the sqrt bar height and much others.

Is that true? Always seemed composing the nodes would parse the glyph metric and then calculate the correct bar length positioned to close with the glyph, and to extend over the radicand. Don't think it is fixed (and why when font metric gets mishandled we end up with badly scaled nodes).
Comment 25 dante19031999 2020-11-10 16:45:41 UTC
(In reply to V Stuart Foote from comment #24)
> (In reply to dante19031999 from comment #23)
> > If we change the font, we will have to deprecate the old one. We will have
> > to change things like the sqrt bar height and much others.
> 
> Is that true? Always seemed composing the nodes would parse the glyph metric
> and then calculate the correct bar length positioned to close with the
> glyph, and to extend over the radicand. Don't think it is fixed (and why
> when font metric gets mishandled we end up with badly scaled nodes).

Well. On my first apporach to this I tried simpy switching the font by a different one. I got problems with roots (the sqrt / root node has a node with the sqrt / root glyph, wich is scaled and pasted, after you put the bar at the height of the bar of the glyph), text attributes wich can be composed ( vec, hat, tilde, ... ) and others. So changing the font requieres work and completly get rid of opensymbol. Even so I doesn't mind working on it, because actual opensymbol is inconpleet ( see tdf#134214 ), wich blocs implementations of things as tdf#49990, tetradimensional integration, four dots derivate and much more ).

So I do believe we should switch it for a new one, an open free font supported by an organization, wich would be correctly maintained and would have all unicode64 features. About old documents, it should not be a problem, the user never got to choose how it looks.
Comment 26 dante19031999 2020-11-10 16:46:37 UTC
By the way, i did it with STIX.
Comment 27 V Stuart Foote 2020-11-10 17:08:09 UTC
(In reply to dante19031999 from comment #26)
> By the way, i did it with STIX.

OK, though I've noticed STIX2 Math has pretty tall (ascent & descent) font metrics, the U+221A, U+221B, U+221C have too much internal leading.

Libertinus Math might be a better test, Khaled did a nice job with those.
Comment 28 dante19031999 2020-11-10 17:43:40 UTC
(In reply to V Stuart Foote from comment #27)
> (In reply to dante19031999 from comment #26)
> > By the way, i did it with STIX.
> 
> OK, though I've noticed STIX2 Math has pretty tall (ascent & descent) font
> metrics, the U+221A, U+221B, U+221C have too much internal leading.
> 
> Libertinus Math might be a better test, Khaled did a nice job with those.

OK. We can change OpenSymbol for Libertinus Math font.
I don't care wich font it is as long at it is complete.
When I was working in this, I tried to upload to gerrit a ttf ( I think it was this font file extension ) file and couldn't. I'm gonna need someone with the permissions required to upload it on a patch, so I can work on it. So, can you do it? It is a need because we have to guarantee the font is available.

There's a lot of work to do. Will take several patches, and some old document crashes, wich we will have to solve in it's due moment. Once we have the new font working, we have to make nodes font independent, and after that we can think on allowing the font to be changed.
Comment 29 V Stuart Foote 2020-11-10 18:52:55 UTC
(In reply to dante19031999 from comment #28)
>...
> When I was working in this, I tried to upload to gerrit a ttf ( I think it
> was this font file extension ) file and couldn't. I'm gonna need someone
> with the permissions required to upload it on a patch, so I can work on it.
> So, can you do it? It is a need because we have to guarantee the font is
> available.
> 
> ...

Not me. Mike, Heiko?  And Libertinus Math would be a starting point to determine if an alternate to OpenSymbol is even going to be feasible. Expect it needs to be bundled to test that in many locations.

=-ref-=
https://github.com/alerque/libertinus
Comment 30 dante19031999 2020-11-10 21:47:49 UTC
(In reply to V Stuart Foote from comment #29)

> Not me. Mike, Heiko?  And Libertinus Math would be a starting point to
> determine if an alternate to OpenSymbol is even going to be feasible. Expect
> it needs to be bundled to test that in many locations.
> 
> =-ref-=
> https://github.com/alerque/libertinus

I found a Heiko in the CC list. I emailed him. If i get no answer I will assume I've picked another Heiko and will need they're contact info (if it sounds irrespective it's google's fault, if not google's fault too). I think it is feasible. But I'll need a least a year for the total change. However already took a look (switching math fonts). Some operators are a little tiny and scalable commands as widevec or brackets total disaster. Gonna take lots of patches total switch.

I'm adding the email instructions in case of someone falls here and can help:

Hello.
I'm Dante, a voluntary developper for LO. I'm working on tdf#101174. In order to continue working on it we need to add a font to Libreoffice: Libertinus math ( It's what we choose in the chat ), which will replace OpenSymbol in the future. However, we need someone with the permissions and authority to upload the file. Main reasons to do the change are: OpenSymbol is outdated and are characters missing ( #td134214 ) wich halts developpement, like for example setp ( tdf#49990 ), mathbb, fraktur ( listed as TODO in the code ), and maintenance costs. They will decrease if we use a well maintained font.

From this git repo: https://github.com/alerque/libertinus
Add files: core/source/LibertinusMath-Regular.sfd
At: core/extras/source/opentype/LibertinusMathRegular.sfd

There's also a makefile to edit, but I'll handle it once it is uploaded.
Comment 31 Heiko Tietze 2020-11-11 08:58:51 UTC
(In reply to dante19031999 from comment #30)
> I found a Heiko in the CC list.

I do not have permission to directly upload files. You could ask on the IRC channel #libreoffice-dev for help.

It's not just the font to upload but all hard-coded references needs to be changed ("git grep OpenSymbol | wc -l" returns 165). And with another font might we may run into regressions such as users expect pixel-perfect layout for older documents. OpenSymbol causes some trouble with bullets, there are many tickets, that might be solved but with regression costs. Meaning: we need at least a transition plan. Miklos, what's your thoughts here?

Similar request has been made in bug 135788 to "Substitute fonts Linux Libertine and Linux Biolinum with Libertinus". The response was not consistently positive.
Comment 32 dante19031999 2020-11-11 10:13:23 UTC
(In reply to Heiko Tietze from comment #31)

> It's not just the font to upload but all hard-coded references needs to be
> changed ("git grep OpenSymbol | wc -l" returns 165). And with another font
> might we may run into regressions such as users expect pixel-perfect layout
> for older documents. OpenSymbol causes some trouble with bullets, there are
> many tickets, that might be solved but with regression costs. Meaning: we
> need at least a transition plan. Miklos, what's your thoughts here?

I guess not giving the plan is letting you in the dark. I've already been studying the how to. But before a little bit of technical information of starmath:

The math font is defined in inc/types.hxx as "OpenSymbol".
On source/cfgitem.cxx you won't be able to setup the FNT_MATH index of fonts.
If you added a bullet into the code it will have it's own token of type TCHARACTER (TG::NONE).
If you did something like %bullet it will be TSPECIAL (TG::NONE). In theory they remember the character code and it's font. So as long as OpenSymbol remains available it will be OK.

So, here is the plan ( feel free of improving it ):
1. Upload the Libertinus font sfd file.
2. Add to the OpenSymbol make file code to also build Libertinus (fontforge).
3. On cfgitem.cxx make "setupable" the FNT_MATH index of fonts. But setup the default values to match old defined config.
4 Force the root symbol node to always use OpenSymbol font.
5. From now on we are adding experimental features, so end user won't have to deal with the changes until the really end.
6. Inside the node codes, interfere the font and change it for libertinus. But not so fast. We only change it if the token is in TG::UnOper, TG::Product (except TOVERBRACE and TUNDERBRACE), TG::Standalone, TG::Sum, TG::Oper.
7. We take a look and solve any problem. Untill here the easy part.
8. We continue with groups TG::Lbrace and TG::Rbrace. Here is where we have to deal with the scalable brackets problem. However it should be possible since this font is compatible with latex and works fine.
9. We complete TG::Product and continue with TG::Attribute. In this occasion we gonna have to deal with some problems I found with horizontal scale change. However it should be possible since this font is compatible with latex and works fine.
10. Here we stop. Make very hardcore tests and get sure everything works fine.
11. Now we change the "OpenSymbol" for "Libertinus math" on types.hxx and make features no experimental.
12. The roles are now inverted, instead of interfering what we are changing, we are interfering what we are not changing. Meaning, interfering for the use of opensymbol when required. However there is no much more to do. What is remaining is TG::NONE. In there are things like functions ( FNT_FUNCTION, no need ), identities ( TIDENT ) wich in theory does not use OpenSymbol, OpenSymbol doesn't even have A-Z, matrix and other groups of data. No much more. The change is quasi accomplished. There is TCHARACTER. That one will be the difficult one.
13. The final point: the root symbol node, make it universal for fonts, so this original tdf objective can be accomplished.
14. Add the entry on advanced configuration.
Comment 33 dante19031999 2020-11-11 10:15:15 UTC
(In reply to Heiko Tietze from comment #31)

> I do not have permission to directly upload files. You could ask on the IRC
> channel #libreoffice-dev for help.
 
Gonna try to get some support by IRC channel and email.
Comment 34 V Stuart Foote 2020-11-11 14:37:54 UTC
(In reply to dante19031999 from comment #33)
>  
> Gonna try to get some support by IRC channel and email.

OK, but in addition to the sm module use, also investigate that OpenSymbol is the glyph repository of all the PUA & Unicode table mappings onto legacy StarBat, StarMath, Wingdings, Wingdings2, Wingding3, Webdings calls [1].  Meaning no font will be a simple drop in replacement for OpenSymbol--and that more is required.  

Current 'Font Replacement' from Tools -> Options works bcz the mapping of OpenSymbol PUA is retained. Bug 47679 shows how losing the PUA mapping can impact things.

So not just the composition of sm nodes, but the current PUA table mappings will need to be reworked, so either elimination or with reassignment onto defined Unicode points, for this to be viable. 

=-ref-=
[1] ttps://opengrok.libreoffice.org/xref/core/unotools/source/misc/fontcvt.cxx