Bug 103514 - MS bitmap fonts (Courier, MS Sans Serif, MS Serif) not displayed with OpenGL rendering
Summary: MS bitmap fonts (Courier, MS Sans Serif, MS Serif) not displayed with OpenGL ...
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: graphics stack (show other bugs)
Version:
(earliest affected)
5.1.0.3 release
Hardware: All Windows (All)
: medium normal
Assignee: Khaled Hosny
QA Contact:
URL:
Whiteboard: target:5.3.0 target:5.2.4 target:5.4....
Keywords: bibisected
Depends on: 105015
Blocks: VCL-OpenGL Font-Substitution HarfBuzz
  Show dependency treegraph
 
Reported: 2016-10-26 09:03 UTC by V Stuart Foote
Modified: 2017-11-04 11:14 UTC (History)
9 users (show)

See Also:
Crash report or crash signature:


Attachments
ODT test file with strings of MS bitmap fonts (10.22 KB, application/vnd.oasis.opendocument.text)
2016-10-26 14:07 UTC, V Stuart Foote
Details
zip of four screen clips of rendering MS bitmap fonts (256.26 KB, application/zip)
2016-10-26 15:46 UTC, V Stuart Foote
Details
ODT test document with the six fonts to ignore (10.92 KB, application/vnd.oasis.opendocument.text)
2016-10-28 02:15 UTC, V Stuart Foote
Details
ODT test document with the six fonts to ignore (FODT) (27.03 KB, application/vnd.oasis.opendocument.text)
2016-10-28 16:59 UTC, V Stuart Foote
Details
ODT test document with the six fonts to ignore (10.69 KB, application/vnd.oasis.opendocument.text)
2016-10-28 17:00 UTC, V Stuart Foote
Details

Note You need to log in before you can comment on or make changes to this bug.
Description V Stuart Foote 2016-10-26 09:03:07 UTC
Description:
Was testing OpenGL with HarfBuzz enabled and disabled, and noticed odd fallback behavior for the MS system provided bitmap font (Courier, MS Sans Serif, MS Serif) when OpenGL rendering is enabled.

A bibisect results to this range:
https://cgit.freedesktop.org/libreoffice/core/log/?qt=range&q=6b65a0e83c4798f117be61af91dbaebdc85e94b7..acd3ebefccd0b4570100c426574d83cfa9464f20

So looks that on abandoning Uniscribe shaping and reverting to DirectWrite with GDI for SimpleWinLayout and DirectWrite for OpenGL in WinLayout, that the MS "bitmap" fonts have been rendered blank in the Special Character dialog grid when OpenGL rendering is enabled.

Introduction of HarfBuzz shaping with OpenGL results in this font being completely replaced by fallback. Not sure that is correct--but it was different.

Steps to Reproduce:
Windows builds (only?).

1. enable OpenGL Rendering, restart
2. open Writer
3. open Special Character dialog (default Liberation Serif)
4. navigate the drop list to Courier
5. grid populated?
6. if move to Courier New "ttf" font--its grid populated

affecting builds through

Version: 5.3.0.0.alpha1+
Build ID: f7c61b08d526c79ecd1522dff79386059b6125e0
CPU Threads: 8; OS Version: Windows 6.19; UI Render: GL; 
TinderBox: Win-x86@39, Branch:master, Time: 2016-10-25_00:39:40
Locale: en-US (en_US); Calc: CL

Actual Results:  
font grid is empty

Expected Results:
font grid showing just the 


Reproducible: Always

User Profile Reset: n/a

Additional Info:
But when enabling HarfBuzz SAL common layout -- (set SAL_USE_COMMON_LAYOUT=1) -- and restart. With OpenGL enabled, same STR shows a full grid of a fallback fonts filling the entire BMP!

Worth fixing the "old" OpenGL rendering?

Seems like maybe a better way with the new common layout would be to ignore the MS bitmap fonts? They don't show in the paragraph/character drop list of fonts. And with HarfBuzz we drop PS Type1 support--so dropping the *.fon bitmap fonts from the UI may not be too unreasonable.

=-ref-=
https://support.microsoft.com/en-us/kb/2396756


User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:49.0) Gecko/20100101 Firefox/49.0
Comment 1 V Stuart Foote 2016-10-26 09:13:10 UTC
More than just visibility in the Special Character grid, if text strings with any of these characters are inserted into a document using the Special Character dialog--on enabling OpenGL the text is not rendered visible. 

And when SAL common layout is enabled--the replacement font is not well rendered --or even the wrong glyphs are substituted, not sure.
Comment 2 V Stuart Foote 2016-10-26 14:07:02 UTC
Created attachment 128287 [details]
ODT test file with strings of MS bitmap fonts

A test document, toggle OpenGL rendering on and off--with and without SAL common layout.
Comment 3 Khaled Hosny 2016-10-26 14:22:27 UTC
We should simply disable bitmap-only fonts, as a matter of fact I did this already last night for the new layout. Can you try with a build that has commit 48304cb5b4d8df71463837e2e48aa7c4d9594df9? (I don’t have a Windows box handy to test right now).
Comment 4 Khaled Hosny 2016-10-26 14:24:02 UTC
Can you also attach screenshots for the different setups?
Comment 5 V Stuart Foote 2016-10-26 15:46:12 UTC
Created attachment 128290 [details]
zip of four screen clips of rendering MS bitmap fonts

(In reply to Khaled Hosny from comment #4)
> Can you also attach screenshots for the different setups?

attached in a zip archvie
Comment 6 V Stuart Foote 2016-10-26 15:59:13 UTC
Should add that with OpenGL enabled, it looks like on the Writer canvas the MS Courier is getting a replacement with Courier New TTF. 

That doesn't happen in the Special Character dialog which goes blank with OpenGL and SAL common layout not active. 

With common layout, fall back of the entire BMP occurs in the Special Character dialog both with and without OpenGL.

=-Sidebar-=
Points out a need to add control to the Special Character dialog to offer full BMP/SMP (and other Unicode planes) of a "composite" font -- i.e. what fall-back will be if that font is selected. And a default to show just the glyph coverage of that single font.
Comment 7 kompilainenn 2016-10-26 20:01:47 UTC
confirmed for

Version: 5.2.3.1
Build ID: 01ec8f357e651ca9656837b783cf7e6a32ee4d92
CPU Threads: 4; OS Version: Windows 6.1; UI Render: default; 
Locale: ru-RU (ru_RU); Calc: group
Comment 8 V Stuart Foote 2016-10-26 20:17:53 UTC
So, this applies to the new common layout, but it will do nothing for releases through 5.2--should the old DirectWrite with OpenGL rendering need to be corrected?

(In reply to Khaled Hosny from comment #3)
> We should simply disable bitmap-only fonts, as a matter of fact I did this
> already last night for the new layout. Can you try with a build that has
> commit 48304cb5b4d8df71463837e2e48aa7c4d9594df9? (I don’t have a Windows box
> handy to test right now).

Testing that on Windows 8.1 Ent 64-bit en-US with
Version: 5.3.0.0.alpha1+
Build ID: 9c34797cc98030614b384b6ea0f233d59ae82253
CPU Threads: 8; OS Version: Windows 6.29; UI Render: GL; 
TinderBox: Win-x86@39, Branch:master, Time: 2016-10-26_03:58:19
Locale: en-US (en_US); Calc: CL

Enabling SAL common layout composing the Writer canvas is now ignoring system installed bitmap fonts (on Windows Courier, MS Serif and MS Sans Serif).  The WinLayout fallback seems to be making reasonable font selections--Mono spaced for Courier, a Serif font, and a Sans Serif respectively.

Unfortunately we have no way to determine exactly the font being substituted. But the Tools -> Options -> Fonts: font table substitution can be used to manually select the replacement font to be used. 

The Special Character dialog also is ignoring the system installed fonts. With common layout, the dialog is offering system font Segoe UI as the replacement font as the edit cursor is positioned into on a word with the "not installed" font [1].

=-Note-=
[1]. Adjusting the Special Character dialog to offer the substituted font at the text cursor rather than Segoe system default at the text cursor would be helpful
Comment 9 Khaled Hosny 2016-10-26 23:19:42 UTC
(In reply to V Stuart Foote from comment #8)
> So, this applies to the new common layout, but it will do nothing for
> releases through 5.2--should the old DirectWrite with OpenGL rendering need
> to be corrected?

I guess we can do the same an ignore bitmap-only fonts when OpenGL is active.
Comment 10 Michael Meeks 2016-10-27 09:28:38 UTC
Completely agreed - disabling bitmap fonts, ideally globally is the way to go =) We don't want documents that  look different with different rendering backends. There is no good DirectWrite API to render those anymore. Microsoft dropped them (for good reason) in 2013: https://www.poremsky.com/office/type-1-and-other-older-fonts-not-supported-in-office-2013/ =) If we're not using DirectWrite - we will get no accurate physical font API, and so we will have horrible font-fallback behaviour that is de-coupled from glyph-widths.

Khaled - sounds like you've fixed this - and VCL is no longer picking up and listing these fonts ? - if so, I'm happy =) I'd love to do that globally for all backends.
If so - can we close this ?
Comment 11 V Stuart Foote 2016-10-27 19:40:21 UTC
@Khaled, *

So yes the new HarfBuzz based common layout handles the bitmap/raster fonts nicely with fallback.

But can we really call it fixed at this point?  Still should do something to likewise "ignore" the bitmap/raster fonts with existing DirectWrite and OpenGL rendering.

Otherwise this mishandling will be with us for remainder of 5.2 and likely all of the 5.3 release until we shift entirely to HarfBuzz based shaping at 5.4?
Comment 12 Khaled Hosny 2016-10-27 21:07:45 UTC
(In reply to V Stuart Foote from comment #11)
> @Khaled, *
> 
> So yes the new HarfBuzz based common layout handles the bitmap/raster fonts
> nicely with fallback.
> 
> But can we really call it fixed at this point?  Still should do something to
> likewise "ignore" the bitmap/raster fonts with existing DirectWrite and
> OpenGL rendering.
> 
> Otherwise this mishandling will be with us for remainder of 5.2 and likely
> all of the 5.3 release until we shift entirely to HarfBuzz based shaping at
> 5.4?

https://gerrit.libreoffice.org/#/c/30345/
Comment 13 V Stuart Foote 2016-10-28 02:15:12 UTC
Created attachment 128314 [details]
ODT test document with the six fonts to ignore

A test file composed with the six fonts that will be ignored. 

Composed on Windows 10 with 5.2.2.2
Comment 14 Commit Notification 2016-10-28 08:42:03 UTC
Khaled Hosny committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=8f0f5e0c709d01555a4069f8665889924ed181c7

tdf#103514: Always ignore bitmap fonts on Windows

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 15 Khaled Hosny 2016-10-28 10:37:27 UTC
Do we want to back-port this for 5.2?
Comment 16 V Stuart Foote 2016-10-28 12:02:42 UTC
 A good candidate to backport for 5.2.4
Comment 17 V Stuart Foote 2016-10-28 16:59:13 UTC
Created attachment 128326 [details]
ODT test document with the six fonts to ignore (FODT)

as FODT for convenience.
Comment 18 V Stuart Foote 2016-10-28 17:00:54 UTC
Created attachment 128327 [details]
ODT test document with the six fonts to ignore

And just so its around the ODF archive version as well.
Comment 19 V Stuart Foote 2016-10-30 19:32:52 UTC
Back-port to 5.2 up for review...
https://gerrit.libreoffice.org/#/c/30356
Comment 20 Commit Notification 2016-10-31 09:35:51 UTC
Khaled Hosny committed a patch related to this issue.
It has been pushed to "libreoffice-5-2":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e80621fb04093d5ab950485357f6058dcbaf8aaf&h=libreoffice-5-2

tdf#103514: Always ignore bitmap fonts on Windows

It will be available in 5.2.4.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 21 Commit Notification 2016-12-15 10:13:07 UTC
Khaled Hosny committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=ef4b9032de55e6b1b182e4ead1bbe6e590df296e

tdf#103514: Try harder to ignore non-SFNT fonts

It will be available in 5.4.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 22 Commit Notification 2016-12-15 12:31:58 UTC
Khaled Hosny committed a patch related to this issue.
It has been pushed to "libreoffice-5-3":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=fcbbd6089da7e60bf616041517a2aee3017ae448&h=libreoffice-5-3

tdf#103514: Try harder to ignore non-SFNT fonts

It will be available in 5.3.0.1.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 23 Commit Notification 2017-01-02 14:52:19 UTC
Khaled Hosny committed a patch related to this issue.
It has been pushed to "libreoffice-5-2":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=f420f4c2e84deb182e4f675c1a0d45bede110ff2&h=libreoffice-5-2

Revert "tdf#103514: Always ignore bitmap fonts on Windows"

It will be available in 5.2.5.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 24 Julien Nabet 2017-01-04 22:31:25 UTC
Khaled: just to be sure, is it on purpose that the revert has only be done on 5.2 branch? I mean, are 5.3 and master branches ok without the revert? (unless I missed something)
Comment 25 Khaled Hosny 2017-01-05 00:03:04 UTC
It was reverted on 5.2 only to avoid regression on a stable branch, while we try to find a proper fix for 5.3 (which is not released yet).
Comment 26 Julien Nabet 2017-01-05 11:00:11 UTC
(In reply to Khaled Hosny from comment #25)
> It was reverted on 5.2 only to avoid regression on a stable branch, while we
> try to find a proper fix for 5.3 (which is not released yet).

Ok, thank you for the explanation.
Comment 27 Julien Nabet 2017-01-13 21:47:35 UTC
*** Bug 105315 has been marked as a duplicate of this bug. ***