Bug Hunting Session
Bug 56076 - FONTS SUBSTITUTION: no way to control glyph substitution if glyph is missing from font
Summary: FONTS SUBSTITUTION: no way to control glyph substitution if glyph is missing ...
Status: RESOLVED NOTOURBUG
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Writer (show other bugs)
Version:
(earliest affected)
3.6.3.1 rc
Hardware: Other All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-10-17 09:42 UTC by Yury
Modified: 2014-04-08 06:09 UTC (History)
0 users

See Also:
Crash report or crash signature:


Attachments
minimal document exhibiting the issue (10.35 KB, application/vnd.oasis.opendocument.text)
2012-10-17 09:43 UTC, Yury
Details
the screenshot of the auto ('hidden') glyph substitution in the doc (15.87 KB, image/jpeg)
2012-10-17 09:46 UTC, Yury
Details
same document, with LibO font substitution activated (GTK system font->CMU Serif) (15.72 KB, image/jpeg)
2012-10-17 09:47 UTC, Yury
Details
Screenshot after configuring FontConfig (37.85 KB, image/png)
2013-11-26 08:48 UTC, Khaled Hosny
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Yury 2012-10-17 09:42:25 UTC
If the font being used doesn't contain certain glyph, LibO substitutes one with the appropriate glyph taken from the font which has one.
Unfortunately, there is no way to control the substitution, at the very least on the lines of style used and the GTK system font is always selected as the source of the substitution.
I have 'Lucida Sans' as a GTK system font, so I'm getting a somewhat ugly output, with mixed styles and, possibly, differing sizes, too.
I believe this is related to #34499 (which was closed unaddressed because of NEEDINFO overdue) and to #32419.
Is there any way to control what font serves as a source of substitutions (eventually, the set of such sources)? Fontconfig tuning or some font trick in LibO, perhaps?

I'm attaching the short document exhibiting the problem and the corresponding screenshot.

*** The second screenshot shows what happens if I make a font substitution for the GTK system font in LibO (the 'Tools->Options->View->Use system font for UI' is checked):
1) The UI font gets substituted -- in this case, 'Lucida Sans' is replaced by 'CMU Serif' in menus, toolbars etc. BTW, that is how I may AGAIN have MY choice of the UI font in the recent LibO releases.
2) The text in document, which was styled in 'Lucida Sans' is rendered now in 'CMU Serif', too. 
3) HOWEVER, glyphs which were previously being substituted by glyphs from 'Lucida Sans', continue to be substituted by glyphs from 'Lucida Sans'!
Comment 1 Yury 2012-10-17 09:43:27 UTC
Created attachment 68679 [details]
minimal document exhibiting the issue
Comment 2 Yury 2012-10-17 09:46:39 UTC
Created attachment 68680 [details]
the screenshot of the auto ('hidden') glyph substitution in the doc
Comment 3 Yury 2012-10-17 09:47:42 UTC
Created attachment 68681 [details]
same document, with LibO font substitution activated (GTK system font->CMU Serif)
Comment 4 Yury 2012-12-27 06:04:36 UTC
Regarding the hypothetical controls of such substitution, I would suggest one of the following:

1) Extend the functionality of font families list as entered with ';' separator in the field of 'format->character/paragraph style' dialog (and saved as comma-separated list in the ODF). Currently, that list allows for 'family missing' family fallback(-s). What if it gave hints for the 'glyph missing' family fallback(-s), too?

2) Extend the functionality of 'options->fonts' dialog. That would require adding a third column of checkboxes with meaning 'substitute if glyphs are missing'.

In any case, the result of family substitution MUST be explicated somewhere, on status bar, possibly.
Comment 5 Yury 2012-12-27 06:11:21 UTC
If somebody would kindly point out to me the LibO sets of sources dealing with the: (a) families lists in character/paragraph styles (b) glyph missing situation, I could try to dabble with the problem myself.

Also, please tell me what GTK widget and other "styles" LibO uses (refers to)? I mean "styles" as in ~/.gtkrc-2.0 file. Until anything positive comes up, I'd try to make a kludge from those.

I definitely can't find the info mentioned above by myself.
Comment 6 Yury 2012-12-27 06:14:13 UTC
After much thought, I'm raising the importance to major. Just think of it -- the text gets processed in arbitrary (effectively, random) manner?
Comment 7 Yury 2013-04-05 09:48:47 UTC
By the way, the capital 'Φ' glyph on the screenshots which is missing from 'PT Serif' and 'Latin' is Greek (U+03A6), not Cyrillic.
Comment 8 Urmas 2013-04-05 12:09:31 UTC
There should be no glyph substitution at all: the text processor is not a web browser and should give to user precise control over the fonts and characters used.
Comment 9 Yury 2013-04-05 14:36:50 UTC
In theory, yes, but that would take years to implement. So I would settle for something adequate right now. Environment variable, multiply fonts in paragraph style, anything.
Comment 10 Stefan Knorr (astron) 2013-04-20 15:23:14 UTC
Can reproduce with LibreOffice 4.0.1 on Linux, x86-64.

This bug can be worked around easily (by directly formatting parts of the text with another font), so setting severity to normal again.

Option (1) from your comment 4 seems like the best way to improve behaviour to me.
Comment 11 Stefan Knorr (astron) 2013-04-20 15:36:12 UTC
Also, let me say that I don't agree with comment 8 : Suppose you send a document to someone else who doesn't have the font specified in your document (or an older version of it), you probably want that person to be able to read the document, no?
Comment 12 Yury 2013-04-20 17:19:42 UTC
First thing first, I'd like to thank you, Stefan, for your prompt answer.

Now...

(In reply to comment #10)
> This bug can be worked around easily (by directly formatting parts of the
> text with another font), so setting severity to normal again.
> 
> Option (1) from your comment 4 seems like the best way to improve behaviour
> to me.

(In reply to comment #11)
> Also, let me say that I don't agree with comment 8 : Suppose you send a
...

Comment 8 might be somewhat drastic, but I'd still call this quite an issue.

So I can reformat the offending part with another font. But what if I didn't even /notice/ the offending part? 
What if another font /also/ has these glyphs missing or has /another/ set of glyphs missing?
LibO gives completely no indication of this kind of typesetting problem. What's worse, it sort of lies to the operator. Set a cursor into the such 'silently processed' region. Does it at least /show/ anywhere that the face is different from what's set in style?

Meanwhile, WordPerfect had been allowing the nearly perfect control over the glyphs/codes and faces for about 20 years now (I mean at least since WP for Win 6.0). Hmm?

BTW, with the current default faces' lists, the 'Andale Sans UI' recipe doesn't work anymore and has to be renamed 'Segoe UI' recipe now.
Comment 13 Caolán McNamara 2013-05-14 19:16:19 UTC
Glyph substitution under Linux is entirely a matter for fontconfig, we give fontconfig the name of the font, the missing glyphs, the language of the text (if known), etc. and take what it gives.
Comment 14 Yury 2013-05-14 20:15:43 UTC
Please reconsider, e.g., making the LibO request a sequence of font faces (with appropriate provisions for such multi-selections in paragraph/char styles).

Right now you are just confirming that the software *external* to LibO is bypassing all kinds of user control and dictates how things are being rendered inside the LibO.

That is definitely OURBUG, in a very real sense of the word.
Comment 15 Khaled Hosny 2013-06-22 10:13:59 UTC
(In reply to comment #8)
> There should be no glyph substitution at all: the text processor is not a
> web browser and should give to user precise control over the fonts and
> characters used.

You are confusing word presossors which should just work (i.e. always give the user some readible text) with typesetting systems (some times called DTP), which should give the typsetter the finest control about the final output.
Comment 16 Yury 2013-06-22 13:35:56 UTC
(In reply to comment #15)
> (In reply to comment #8)
> > There should be no glyph substitution at all: the text processor is not a
> > web browser and should give to user precise control over the fonts and
> > characters used.
> 
> You are confusing word presossors which should just work (i.e. always give
> the user some readible text) with typesetting systems (some times called
> DTP), which should give the typsetter the finest control about the final
> output.

There is 'no control' and there is 'the finest control', but I think many folks would settle for a little more control (than there is now) over the text appearance in LibO. The 'faces list' which is acceptable to HAVE in the ODF should have some EFFECT as well.

Why's the bug title substitution not automated to take serifed glyphs for the serifed body font, at least?

Why're the bitmapped fonts excluded from being UI font, BTW?
Comment 17 Khaled Hosny 2013-11-26 07:46:51 UTC
(In reply to comment #16)
> (In reply to comment #15)
> > (In reply to comment #8)
> > > There should be no glyph substitution at all: the text processor is not a
> > > web browser and should give to user precise control over the fonts and
> > > characters used.
> > 
> > You are confusing word presossors which should just work (i.e. always give
> > the user some readible text) with typesetting systems (some times called
> > DTP), which should give the typsetter the finest control about the final
> > output.
> 
> There is 'no control' and there is 'the finest control', but I think many
> folks would settle for a little more control (than there is now) over the
> text appearance in LibO. The 'faces list' which is acceptable to HAVE in the
> ODF should have some EFFECT as well.

That reply wasn’t to you, but anyway, you can control this by properly configuring FontConfig to your liking, LibreOffice shouldn’t reinvent FontConfig (which does its job quite nicely and have versatile configurability).

> Why's the bug title substitution not automated to take serifed glyphs for
> the serifed body font, at least?

Not sure I get what you mean here.

> Why're the bitmapped fonts excluded from being UI font, BTW?

Ditto.
Comment 18 Yury 2013-11-26 08:17:07 UTC
Well, go ahead and consider this solved, but this resolution IS INVALID (see below).

(In reply to comment #17)

> That reply wasn’t to you, but anyway, you can control this by properly
> configuring FontConfig to your liking, LibreOffice shouldn’t reinvent
> FontConfig (which does its job quite nicely and have versatile
> configurability).

LibO plainly makes INCORRECT USE of fontconfig features.

The mentioned sans-serifed glyph substituted for glyph not present in serifed (installed!) font face is quite an example of this.

See screenshots attached. I have strings in serifed faces which get glyph from sans-serifed face substituted. And there ARE serifed faces with that glyph installed on my system.

What further "proper" configuring do I "need" to do? Isn't this advice actually going too far?

***

> > Why're the bitmapped fonts excluded from being UI font, BTW?
> 
> Ditto.

There are no raster fonts in list of choices in the fonts substitution dialog (where one may choose a replacement for Andale Sans UI or Segoe UI faces).

Why so? I understand that rasters may be not a good choice for a content, but why exclude those from UI use?

I don't like begging, and so have earlier looked in the source myself. Unfortunately, the code proved to be too convoluted for me, but I have noticed that raster fonts are somehow excluded from the fonts list (used, e.g., in fonts substitution dialog). I didn't manage to find the fontconfig request in question.
Comment 19 Khaled Hosny 2013-11-26 08:46:59 UTC
(In reply to comment #18)
> Well, go ahead and consider this solved, but this resolution IS INVALID (see
> below).
> 
> (In reply to comment #17)
> 
> > That reply wasn’t to you, but anyway, you can control this by properly
> > configuring FontConfig to your liking, LibreOffice shouldn’t reinvent
> > FontConfig (which does its job quite nicely and have versatile
> > configurability).
> 
> LibO plainly makes INCORRECT USE of fontconfig features.

No, it does not.

> The mentioned sans-serifed glyph substituted for glyph not present in
> serifed (installed!) font face is quite an example of this.
> 
> See screenshots attached. I have strings in serifed faces which get glyph
> from sans-serifed face substituted. And there ARE serifed faces with that
> glyph installed on my system.

Try "fc-match -s 'PT Serif'" You will find that the next listed font that contains the requested character is a Sans font. If you want a different fallback font, you have to configure FontConfig as such.

> What further "proper" configuring do I "need" to do? Isn't this advice
> actually going too far?

Please refer to FontConfig documentation. That is why this bug is resolved as NOTOURBUG.

One way to do this is to add:
<alias>
  <family>PT Serif</family>
  <prefer>
    <family>serif</family>
  </prefer>
</alias>

etc. to your FontConfig configuration. You can even list specific fonts there, please refer to the respective documentation.

> > > Why're the bitmapped fonts excluded from being UI font, BTW?
> > 
> > Ditto.
> 
> There are no raster fonts in list of choices in the fonts substitution
> dialog (where one may choose a replacement for Andale Sans UI or Segoe UI
> faces).

Do you have such fonts in a format that LibreOffice can use (e.g. not in X bitmap format which I think support for it was long dropped). I admit I don’t know much about that dialog and how it works internally, so I’m just speculating here.
Comment 20 Khaled Hosny 2013-11-26 08:48:50 UTC
Created attachment 89820 [details]
Screenshot after configuring FontConfig
Comment 21 Yury 2013-11-26 09:31:48 UTC
I'm still very unconvinced (but have it your own way, of course).

I do NOT believe I ought to have to hand-tune lots of fonts in a config file having obscure and convoluted format in order just to get a sensible font face substitution (is it the same on Windows and Mac, BTW?). If I add a font, do I have to do this again?

I knew what fc-match would show, of course. The point is such use of the fontconfig engine can show only the output of the built-in rules (which, yes, would give fallback to DejaVuSans on almost all linux systems out-of-the-box). While (obviously?) the problem I'm talking should be solved by more accurate use of the fontconfig engine (there IS such API in fontconfig, isn't there?)

I might yet agree to the need for editing fontconfigs in order to get a SPECIFIC substitution (glyph from the specific face), although that's also has too much of an 1980s-style usability.

If only somebody directed me to the code doing the fontconfig match request, I'd just attempt to make this mod for myself.

***

> > > > Why're the bitmapped fonts excluded from being UI font, BTW?
> > > 
> > > Ditto.
> > 
> > There are no raster fonts in list of choices in the fonts substitution
> > dialog (where one may choose a replacement for Andale Sans UI or Segoe UI
> > faces).
> 
> Do you have such fonts in a format that LibreOffice can use (e.g. not in X
> bitmap format which I think support for it was long dropped). I admit I
> don’t know much about that dialog and how it works internally, so I’m just
> speculating here.

I have such fonts installed in format which every other multi-platofrm application on my system is capable of using (there are BDFs and PCFs). Not LibO, though.
Comment 22 Khaled Hosny 2013-11-26 10:50:32 UTC
(In reply to comment #21)
> If only somebody directed me to the code doing the fontconfig match request,
> I'd just attempt to make this mod for myself.

vcl/generic/fontmanager/*