Bug 71603 - Fallback handling of Unicode font replacement on Windows affected by method of launch
Summary: Fallback handling of Unicode font replacement on Windows affected by method o...
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: LibreOffice (show other bugs)
Version:
(earliest affected)
4.1.3.1 rc
Hardware: Other All
: medium normal
Assignee: Khaled Hosny
URL:
Whiteboard: target:5.3.0
Keywords:
Depends on:
Blocks: Font-Rendering Fonts
  Show dependency treegraph
 
Reported: 2013-11-14 10:48 UTC by pierre-yves samyn
Modified: 2019-06-26 13:03 UTC (History)
8 users (show)

See Also:
Crash report or crash signature:


Attachments
clip from current master of document with Segoe UI Symbol glpyhs (57.78 KB, image/png)
2016-01-07 00:35 UTC, V Stuart Foote
Details
zip with screenclips of opening Unicode using fallbacks differ by method of launch -- StartCenter or OS (102.43 KB, application/zip)
2016-01-11 21:00 UTC, V Stuart Foote
Details
sample with both BMP and SMP codepoints that have fallback font problems (16.00 KB, application/vnd.oasis.opendocument.text)
2016-01-14 04:41 UTC, V Stuart Foote
Details
Autocorrect support for Emoji inserts clock face codepoints (10.34 KB, application/vnd.oasis.opendocument.text)
2016-01-17 16:54 UTC, V Stuart Foote
Details
sample document mixed language multiple fonts (15.29 KB, application/vnd.oasis.opendocument.text)
2016-10-31 16:20 UTC, V Stuart Foote
Details

Note You need to log in before you can comment on or make changes to this bug.
Description pierre-yves samyn 2013-11-14 10:48:43 UTC
Hello

May be related to Bug 59184 (linux Issue)
I open this one for Windows

Description : Unicode character are not displayed

Platform 4.1.3.1 and 4.2.0.0.alpha0+ Windows 7

1. Deleting profile (Delete the profile is not necessary to reproduce the problem. This step  is only to exclude a possible cause).
2. Launch LibreOffice
3. File> Open =>  "Test document with unicode caracter" 

attachment 72757 [details]

Unicode characters are not displayed *and* the preview does not work in the font list (see UnicodeFileOpen.png screenshot)

attachment 89147 [details]

Quit LibreOffice

1. Deleting profile
2. Double click in Explorer windows on the document

The document loads, the unicode characters are displayed (and preview
works) (see UnicodeOpenExpl.png screenshot).

attachment 89146 [details]

The problem is also reproduced (qa-fr) with
LO Portable 3.6.6.2, 4.1.3.2 LO, LO 4.2.0.0.alpha0+ on Windows 7 Home
Premium.


Regards
Pierre-Yves
Comment 1 Jacques Guilleron 2013-11-15 22:36:11 UTC
Hello,

I confirm this behaviour for LO Portable 3.6.6.2, 4.1.3.2 LO, LO 4.2.0.0.alpha0+ on Windows 7 Home Premium.

Jacques
Comment 2 QA Administrators 2015-04-19 03:22:26 UTC Comment hidden (obsolete)
Comment 3 pierre-yves samyn 2015-04-19 06:17:21 UTC
Hi

Still reproduced on windows 7/64 & Version: 4.4.3.1
Build ID: b2f347f2ac68821efc00b6f1793cda90af748118
Locale: fr_FR

Regards
Pierre-Yves
Comment 4 V Stuart Foote 2016-01-07 00:33:25 UTC
The issue is with the font set--those glyphs are simply not defined in the Liberation Serif font set. Easily verified with a Unicode block display utility like BableMap.

And, when sample document is edited and paragraph styles modified to use a font that contains glyphs for those Unicode pages they display correctly. Segoe UI Symbol for example provides glyphs for all. Liberation Sans and OpenSymbol do not. 

Suppose it is possible a replacement table had been defined, seems less likely that glyphs had been dropped from the fonts.

Anyhow, this is working correctly in current master and 5.1 and 5.0 releases.

See attached clip.
Comment 5 V Stuart Foote 2016-01-07 00:35:06 UTC
Created attachment 121756 [details]
clip from current master of document with Segoe UI Symbol glpyhs
Comment 6 V Stuart Foote 2016-01-07 01:12:20 UTC
(In reply to V Stuart Foote from comment #4)
> 
> Anyhow, this is working correctly in current master and 5.1 and 5.0 releases.
> 
> See attached clip.

Whoops not so fast. Seems it is *not* correct in 5.0 or 5.1 font faces with defined glyphs are rendering correctly on master (but only checked back to 14 December so not sure when it was picked up).  Back to NEW.

@Chris, Tor noticed you've been poking at salfont and salgdi3 -- had this behavior somehow been a loss in dynamic glyph fallback that has now cleared?  

=-=-=
ref Herbert D.'s work on AOO https://bz.apache.org/ooo/show_bug.cgi?id=101552
Comment 7 Tor Lillqvist 2016-01-07 05:09:18 UTC
V: Chris's latest work is in master only, not in 5.1 and even less in 5.0.
Comment 8 V Stuart Foote 2016-01-07 15:20:13 UTC
We need to adjust the summary and the STR

Fallback handling of Unicode font replacement on Windows affected by method of launch.

On Windows -- since it is apparently working correctly in Linux (bug 59184 with http://cgit.freedesktop.org/libreoffice/core/commit/?id=5a0984fd383e54c130016d15a0b72c230968c61e&h=libreoffice-4-1 ) -- using the OS to launch the document (i.e. Open from Windows Explorer) is fully performing fallback font replacement in LibreOffice.  

But, not the same if LibreOffice is already running. Fallback font replacement is not occurring for Unicode panes.

STR with test document attachment 72757 [details]

A. Correct fallback
1. open Windows Explorer
2. locate downloaded attachment
3. open (with double click, or context menu)

result: Document opens, paragraph styled with Liberation Serif but containing Unicode code points not included in that font face are rendered with alternates (Segoe UI Symbol).

4. close, reopen from Start Center or MRU list.

result: Document opens, fallback again performed, code points rendered with replacement graphemes rendered.

B. Incorrect fallback
1. close LibreOffice
2. Open to StartCenter
3. open (Open, Ctrl+O, File -> Open) the attachment; 

result: Document opens, but paragraphs styled with Liberation Serif and containing Unicode code points not included in Liberation Serif panes are not rendered by fallback to alternate fonts.


Additionally with current builds of master/5.2.0alpha0 > 2016-01-03 the metrics being used seems to have shifted resulting in excessive line spacing.

Note: when launched from CLI with swriter.exe <attachment> builds through 5.1 perform fallback correctly. But are affected by Chris S' current work > 2016-01-03 on master.
Comment 9 Chris Sherlock 2016-01-11 16:50:01 UTC
Drat. Sorry, I honestly didn't notice this bug till now, got lost in a mail filter. I'll look at this tomorrow. I'd say it's been caused by some refactoring I've done, which I'm going to test and then revert anyway.
Comment 10 Chris Sherlock 2016-01-11 17:06:38 UTC
Hmmm... The fact it's working in master (partially) may be a happy accident. 

There are a few things I did, the first was that I streamlined the ImplFontAttributes, ImplDevFontAttributes and ImplFontMetricData classes (which I am now seriously considering to have been a bit of a mistake!) into ImplFontAttributes.

The other thing I've done is put accessors and mutators around ImplFontAttributes (now FontAttributes) because I ultimately want to move away from making PhysicalFontFace and FontSelectPattern inherit from FontAttributes; a font has-a collection of FontAttributes after all, arguably it is not an is-a relationship.

Pointers to the code for my own reference: LogicalFontInstance::GetGallbackForUnicode is the function that finds the fallback glyph if it's not in the existing font instance. 

http://opengrok.libreoffice.org/xref/core/vcl/source/font/fontinstance.cxx#GetFallbackForUnicode

Could it be that initialising the constructor attributes explicitly has worked around some sort of undefined behaviour? Will need to bisect this tomorrow.

I suppose I'm finally going yo have to install lode on my Windows laptop and OS X machine. :-)
Comment 11 Chris Sherlock 2016-01-11 17:20:50 UTC
Or... Fallback was not working due to a bug in the hash map implementation? When I followed the recommended practice of extending the std namespace with a custom hash I have a sneaking suspicion that when I rewrote it to make it clearer to myself it potentially got the hashcode working....

http://cgit.freedesktop.org/libreoffice/core/commit/vcl/source/font?id=50261ea4b7242e365717c77a9370afd1044a97e3
Comment 12 Chris Sherlock 2016-01-11 17:25:32 UTC
Better cgit reference to commit:

http://cgit.freedesktop.org/libreoffice/core/commit/?id=50261ea4b7242e365717c77a9370afd1044a97e3
Comment 13 Chris Sherlock 2016-01-11 17:44:22 UTC
Ah. Sadly, I cannot take any glory for fixing this bug. I suspect this was fixed in bug 92505, I believe there was a bug in the font cache (actually, if it was this then I'm concerned that initialising FontSelectPattern with a copy constructor is blasting a member variable...).

Commit by Miklos Vajna:

http://cgit.freedesktop.org/libreoffice/core/commit/vcl/source/outdev/font.cxx?id=e047caf4bb461fb168d9fe08228c049938ca1698
Comment 14 Chris Sherlock 2016-01-11 17:48:35 UTC
Sorry for bug spam. Test this out tomorrow.
Comment 15 V Stuart Foote 2016-01-11 21:00:28 UTC
Created attachment 121863 [details]
zip with screenclips of opening Unicode using fallbacks differ by method of launch -- StartCenter or OS

Chris, *

(In reply to Chris Sherlock from comment #13)
> Ah. Sadly, I cannot take any glory for fixing this bug. I suspect this was
> fixed in bug 92505, I believe there was a bug in the font cache (actually,
> if it was this then I'm concerned that initialising FontSelectPattern with a
> copy constructor is blasting a member variable...).
> 

Still seems there is an issue in selecting the fallback font used for replacement of missing codepoint glyphs, from BMP Unicode as in 92505, and as here from the SMP Unicode plane

Attaching screen clip examples (with 2016-01-06 master and 5.1.0.1--as named in zip) using the testdoc from bug 92505, using only BMP codepoints, but still displays distinction in fallback replacements made depending on if opening from OS or if launching from StartCenter with LO already running.

With the SMP (the 1F300 codepoints of this issue and bug 59184) no fallback replacements occur when LibreOffice is already running on launch from StartCenter. But on initial launch from CLI or OS, a full slate of replacements glyphs are found.

Is choice of fallback fonts in Windows somehow tied to user profile/locale?  Recall browsing through code that excluded some unlikely fonts based on selected language--that was Caolán's work on the Linux fontconfig.cxx (http://cgit.freedesktop.org/libreoffice/core/commit/?id=5a0984fd383e54c130016d15a0b72c230968c61e&h=libreoffice-4-1)
Comment 16 Chris Sherlock 2016-01-14 01:36:13 UTC
I'll have to dig into the code... 

Sorry if this a silly question, but those latest screenshots - is the original document attached to this bug? I want to load it up and see what happens on an OS X and Linux system.
Comment 17 V Stuart Foote 2016-01-14 04:41:39 UTC
Created attachment 121921 [details]
sample with both BMP and SMP codepoints that have fallback font  problems

@Chris, *
(In reply to Chris Sherlock from comment #16)
> I'll have to dig into the code... 
> 
> Sorry if this a silly question, but those latest screenshots - is the
> original document attached to this bug? I want to load it up and see what
> happens on an OS X and Linux system.

They were all launches of attachment 117007 [details] (from bug 92505)

But, I've attached a new combined sample document combining issues of BMP codepoints (from 0380, 2100, 2700, 2900) and SEP codepoints (from 1F300). Problems with fallback font replacement in both Unicode planes.
Comment 18 V Stuart Foote 2016-01-17 16:54:26 UTC
Created attachment 122044 [details]
Autocorrect support for Emoji inserts clock face codepoints

As in bug 97191 -- the clock face Emoji codepoints 1F559, 1F55A, 1F55B are another issue with fallback font handling. The autocorrect replacement is done, but without glyph available in the grapheme for current font--the space is rendered empty in the document.

User thinks there is a problem and clobbers the codepoint rather than a Ctrl-Z to revert the autocorrect.
Comment 19 V Stuart Foote 2016-03-16 17:02:20 UTC
On Windows 10 Pro 64-bit en-US with
Version: 5.2.0.0.alpha0+ (x64)
Build ID: 042f16a19e3d5f884759dae71264433b988df0e6
CPU Threads: 8; OS Version: Windows 6.19; UI Render: GL; 
TinderBox: Win-x86_64@62-TDF, Branch:MASTER, Time: 2016-03-16_10:03:38
Locale: en-US (en_US)

SimpleWinLayout rendering with the DirectWrite implementation for OpenGL rendering has not corrected the fallback font mechanism. I did not expect it to, but kind of hoped so.

The method of launch is still influencing if font fallback replacement occurs when opening attachment 121921 [details]

Continues to affect both OpenGL rendering and default rendering.
Comment 20 Tor Lillqvist 2016-06-01 06:14:57 UTC
Ah, we already have a bug report for this phenomenon, good. And all back from 2013, even. I have recently noticed this while investigating several other bugs.
Comment 21 V Stuart Foote 2016-08-05 16:04:15 UTC
STR as in comment 8 continue valid on Windows 10 Pro 64-bit en-US with

installed
Version: 5.2.0.4 (x64) 
Build ID: 066b007f5ebcc236395c7d282ba488bca6720265
CPU Threads: 8; OS Version: Windows 6.19; UI Render: GL; 
Locale: en-US (en_US)

and with a current master

/a parallel install (Windows Explorer openwith and navigate to soffice.exe)
Version: 5.3.0.0.alpha0+
Build ID: 626a1aa960ed36f5c3370e2aab1e72d41df875cf
CPU Threads: 8; OS Version: Windows 6.19; UI Render: GL; 
TinderBox: Win-x86@42, Branch:master, Time: 2016-08-04_01:38:13
Locale: en-US (en_US); Calc: CL
Comment 22 Xisco Faulí 2016-09-10 15:56:16 UTC
Hi Chris,
I'm setting this ticket back to NEW as it has been inactive for more than 3
months.
Feel free to assign it back to you if you're still working on this.
Regards
Comment 23 V Stuart Foote 2016-10-31 16:20:52 UTC
Created attachment 128383 [details]
sample document mixed language multiple fonts

Looking at experimental HarfBuzz based common layout (available on master from 2016-10-21, and in 5.3.0alpha1) and find fallback font use on Windows builds are still affected by the means of launching Writer module to edit a file.

Attaching a new sample document to better challenge HarfBuzz layout.

It uses a common paragraph rendered in Symbola (ttf-ancient-fonts), DejaVu Sans, Arial Unicode, and Noto Emoji.

With a SAL_USE_COMMON_LAYOUT variable set when launched from Start Center receive one layout. And when launched from a command windows with swriter.exe tdf71603_fallback_testing_All_BMP.odt you get quite different layout result.
Comment 24 V Stuart Foote 2016-10-31 16:24:54 UTC
@Khaled, *

Not sure it is reasonable to expect the new layout engine to rectify this old Windows font glitch--but it is certainly affected by it.

Opening the new test document from the command line or in a new session from the Start Center should show the issue well.
Comment 25 Khaled Hosny 2016-11-01 03:40:32 UTC
This is a font fallback issue which is orthogonal to the work done in bug 89870.
Comment 26 Commit Notification 2016-11-01 21:48:13 UTC
Khaled Hosny committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=03bff1b6b953e4b7a54d2fb7bbf366bea7e959d9

tdf#71603: Improve font fallback on Windows a bit

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 27 Commit Notification 2016-11-01 23:43:53 UTC
Khaled Hosny committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=5d39c2013374727b1c8f147b8b99d54402a7ff02

tdf#71603: Create a new DC for the font fallback

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 28 Khaled Hosny 2016-11-01 23:45:16 UTC
Took a bit of debugging, but should be fixed now (see details in the above commit message).
Comment 29 V Stuart Foote 2016-11-02 15:08:35 UTC
On Windows 10 Pro 64-bit en-US with

Version: 5.3.0.0.alpha1+
Build ID: 5d39c2013374727b1c8f147b8b99d54402a7ff02
CPU Threads: 8; OS Version: Windows 6.2; UI Render: GL; Layout Engine: new; 
TinderBox: Win-x86@62-merge-TDF, Branch:MASTER, Time: 2016-11-02_01:01:09
Locale: en-US (en_US); Calc: CL

Version: 5.3.0.0.alpha1+
Build ID: 5d39c2013374727b1c8f147b8b99d54402a7ff02
CPU Threads: 8; OS Version: Windows 6.2; UI Render: GL; Layout Engine: old; 
TinderBox: Win-x86@62-merge-TDF, Branch:MASTER, Time: 2016-11-02_01:01:09
Locale: en-US (en_US); Calc: CL

Documents open the same now if launched from Start Center/Open, and if launched form command line wint the new HarfBuzz and old layout engine. Also with the default GDI SimpleWinLayout.

There is different fallback selection occurring between old layout and new HarfBuzz based layout--but it is now consistent by method of launch.

A nice bonus of the patch is that the AutoCorrect Options dialog now shows the replacement glyph for all emoji and special characters fixing bug 92940