Bug 97319 - With OpenGL enabled Unicode codepoints beyond BMP not rendered, affecting SMP and plane 2 -- Windows only?
Summary: With OpenGL enabled Unicode codepoints beyond BMP not rendered, affecting SMP...
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: graphics stack (show other bugs)
Version:
(earliest affected)
5.2.0.0.alpha0+
Hardware: All Windows (All)
: high major
Assignee: Not Assigned
URL:
Whiteboard: target:5.2.0 target:5.1.3
Keywords: bisected, regression
: 98709 (view as bug list)
Depends on:
Blocks: VCL-OpenGL
  Show dependency treegraph
 
Reported: 2016-01-22 15:25 UTC by V Stuart Foote
Modified: 2016-10-25 19:02 UTC (History)
11 users (show)

See Also:
Crash report or crash signature:


Attachments
odt with multiple fonts and both BMP and SEP codepage use (13.57 KB, application/vnd.oasis.opendocument.text)
2016-01-24 20:41 UTC, V Stuart Foote
Details
glyph rendering affected by OpenGL (85.22 KB, image/jpeg)
2016-01-30 19:14 UTC, V Stuart Foote
Details

Note You need to log in before you can comment on or make changes to this bug.
Description V Stuart Foote 2016-01-22 15:25:43 UTC
On Windows 10 Pro 64-bit en-US with
Version: 5.2.0.0.alpha0+
Build ID: c81eddbb20c84280aa64c712e34c829380b24527
CPU Threads: 8; OS Version: Windows 6.19; UI Render: GL; 
TinderBox: Win-x86@39, Branch:master, Time: 2016-01-22_04:19:03
Locale: en-US (en_US)

with or without OpenGL enabled.

Have lost the SEP Unicode codepoints above FFFF both in documents and in the special character dialog.
Comment 1 V Stuart Foote 2016-01-24 20:40:53 UTC
For now Windows only, but can someone check Linux and OS X.

Attaching test document authored on Windows with both Symbola and Code 2001 fonts installed to a Windows 10 Pro 64-bit en-US (10586 build) system.

http://cgit.freedesktop.org/libreoffice/core/log/?id=fefd1221be844a033e409a18e05e8c6e98f6d1a7&qt=range&q=6b65a0e83c4798f117be61af91dbaebdc85e94b7..c81eddbb20c84280aa64c712e34c829380b24527

Unfortunately several pieces in movement in this range... sorry.

Michael S.'s work on i18n Unicode support for StarMath
Tor's work reintroducing SimpleWinLayout (bug 96420)
Chris' work on FontFamily

Also, somewhere in the mix of this and handling fallback font we have lost ability to handle the SEP and higher plane codepages in the Special Symbol dialog.

=-biset results-=
OK 22e5170af74c635cf55d089f97946b6dc86f82ad - 2016-01-06 813a319fe836d1ed1c967928bc044643d0b4c07d - 2016-01-12 c71b5b4d2ec76c0a204f9515dece1e0e0689ce3c - 2016-01-13 70ea14baf7d43c00f73807bce13629ca25320558 - 2016-01-14 f0841c6c86c8c8403eb1d78a1bd43a8adac75e3a - 2016-01-15 0174562fa9e49bf989a571c6ccd51e558109b561 - 2016-01-16 49b5eed56c470975927bb7b0328337ab8a76a910 - 2016-01-17 000df1832b54ba8f48c7f1c4c1cd92b70f6402da - 2016-01-18 447c313586e9b36acff393feae15f5e1b63861ae - 2016-01-19 029ce852c2f67e06d60e0ce50fff936c8e2ce9f4 - 2016-01-20 6b65a0e83c4798f117be61af91dbaebdc85e94b7 - 2016-01-21

Bad c81eddbb20c84280aa64c712e34c829380b24527 - 2016-01-22 3fc292f7b32f30b98dad208eb03e086b927d38a2 - 2016-01-23 49b5eed56c470975927bb7b0328337ab8a76a910 - 2016-01-24
Comment 2 V Stuart Foote 2016-01-24 20:41:52 UTC
Created attachment 122194 [details]
odt with multiple fonts and both BMP and SEP codepage use
Comment 3 V Stuart Foote 2016-01-30 19:08:07 UTC
On Windows 10 Pro 64-bit en-US with
Version: 5.2.0.0.alpha0+
Build ID: 9784ff3d878eaa21491fbd779e57d7d4710f5449
CPU Threads: 8; OS Version: Windows 6.19; UI Render: default; 
TinderBox: Win-x86@39, Branch:master, Time: 2016-01-30_01:31:57
Locale: en-US (en_US)

@Tor, I know Chris has been working on this, and fallback fonts for bug 71603, but with this latest build seem to be getting different renderings of the Unicode BMP and SEP depending on OpenGL rendering.

With OpenGL enabled, the font metrics are garbled, OpenSymbol is especially bad--but also, glyphs from the SEP codepoints are not being rendered in their respective fonts. See attached.

As to bug 71603, no improvement to the fallback font handling for fonts missing BMP or SEP codepoints.  But weird in that autocorrect entering emoji in paragraph with Symbola, e.g. :kimono: works, but for Segoe UI Emoji paragraph it does not and I get an undefined glyph.
Comment 4 V Stuart Foote 2016-01-30 19:14:53 UTC
Created attachment 122290 [details]
glyph rendering affected by OpenGL
Comment 5 V Stuart Foote 2016-01-30 19:30:49 UTC
The Special Characters dialog is also affected. 

With OpenGL disabled, viewing Symbola font for codepoints in the SEP, glyphs are rendered. But again weird in that Segoe UI Symbol or Emoji do not show codepages beyond the BMP (display ends at FFFF).

And a seeming OpenGL glitch, when enabled the SEP codepages for Symbola all end up with a pair of place holder glyphs.
Comment 6 V Stuart Foote 2016-02-13 17:48:48 UTC
On Windows 10 Pro 64-bit (build 10586) en-US with
Version: 5.2.0.0.alpha0+
Build ID: 2b60321b21ff9ada64576f5711950b616b8a25ba
CPU Threads: 8; OS Version: Windows 6.19; UI Render: GL; 
TinderBox: Win-x86@42, Branch:master, Time: 2016-02-12_23:49:18
Locale: en-US (en_US)

and

Version: 5.1.1.1
Build ID: c43cb650e9c145b181321ea547d38296db70f36e
CPU Threads: 8; OS Version: Windows 6.2; UI Render: GL; 
Locale: en-US (en_US)

Work around the issue of bug 71603 using CLI or OS to launch swriter directly, opening the above attachment 122194 [details] or attachment 108223 [details] (from bug 85109) 
 
With OpenGL disabled the SEP, and also Plane 2 (from CJK Unified Ideographs Extension B block in Sim-sum) glyphs are rendered and fallback occurs.

With OpenGL rendering enabled SEP and Plane 2 glyphs are not rendered--fall back does not occur.

So this really looks to be an OpenGL only font glitch, but just Windows??? Some other confirmation would be nice.

Adding to bug 93529

=-aside-=
filed bug 97839 for the Special Character dialog...

Although with OpenGL disabled, when SEP or higher planes are exposed by OS or CLI launch, the Special character dialog is having trouble with characters in the paste preview bar, not composing the multi-byte codepoints.
Comment 7 How can I remove my account? 2016-03-14 14:46:06 UTC
What's SEP?
Comment 8 V Stuart Foote 2016-03-14 16:28:10 UTC
(In reply to Tor Lillqvist from comment #7)
> What's SEP?
/me embarrassed...

Make that Supplementary Multilingual Plane (SMP), aka Unicode Plane 1

And where we now draw many autocorrect substitution entered Emoji glyphs from, so pretty much all LO users affected.
Comment 9 Commit Notification 2016-03-15 08:24:41 UTC
Tor Lillqvist committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=73443fd278811837650160482c34c15e8830f0d3

tdf#97319: Handle surrogate pairs in glyph caching for SimpleWinLayout

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 10 How can I remove my account? 2016-03-15 10:05:49 UTC
The above commit makes the glyphs show up, but they are still rendered with incorrect widths / at incorrect locations (all on top of each others).
Comment 11 How can I remove my account? 2016-03-15 11:22:13 UTC
Remaining problem could be related to glyph fallback. Did we lose glyph fallback completely now for SimpleWinLayout?
Comment 12 V Stuart Foote 2016-03-15 13:00:54 UTC
(In reply to Tor Lillqvist from comment #11)
> Remaining problem could be related to glyph fallback. Did we lose glyph
> fallback completely now for SimpleWinLayout?

The later issues of bug 71603?  Chris's cmnts 10 - 13 ref Miklos's work on bug 92505

Unfortunately, I'm waiting for a current Windows TB to roll to be able to test this as well as the DiretWrite patch for bug 97171

Thanks though, I know dropping Uniscribe and now having to fix SimpleWinLayout has been a pain.
Comment 13 Commit Notification 2016-03-16 17:35:09 UTC
Tor Lillqvist committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e3d0e8069e3bd82831b0070f70052f2202180192

tdf#97319: Give up on caching non-BMP glyphs

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 14 Commit Notification 2016-03-17 07:57:54 UTC
Tor Lillqvist committed a patch related to this issue.
It has been pushed to "libreoffice-5-1":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=c54310c6b0f14a649ab3d290c012fff2fab86d85&h=libreoffice-5-1

tdf#97319: Give up on attempting to cache non-BMP glyphs for SimpleWinLayout

It will be available in 5.1.3.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 15 V Stuart Foote 2016-03-29 22:29:27 UTC
*** Bug 98709 has been marked as a duplicate of this bug. ***
Comment 16 Michael Meeks 2016-04-08 15:21:38 UTC
Any input here ? can we close it ? =) ultimately glyph fallback is a bit of a horror here I think ...
Comment 17 V Stuart Foote 2016-04-08 17:30:30 UTC
On Windows 10 Pro 64-bit en-US with
Version: 5.2.0.0.alpha0+
Build ID: 157469896ef56720f33676222b95e81c04ab5c72
CPU Threads: 8; OS Version: Windows 6.2; UI Render: default; 
TinderBox: Win-x86@62-merge-TDF, Branch:MASTER, Time: 2016-04-06_20:10:15
Locale: en-US (en_US)

We now get glyph rendering now for SMP with OpenGL rendering via DirectWrite, as well for SMP with default rendering in SimpleWinLayout.

So, probably OK to close this issue.

There remains a noticible difference in rendering the SMP glyps remains between OpenGL DirectWrite and default rendering.  Almost as if with OpenGL and DirectWrite the block holding the glyph is off-center and the glyph is clipped.

Strangely, with OpenGL the top edge and left edge of glyps with SMP codepoints are clipped, the BMP codepoints seem unaffected.

With default rendering it looks like positioning of both BMP and SMP glyphs are good.

Also, the issues of bug 71603 continue to impact this. When launched via StartCenter no fall-back font assignment occurs. Must launch directly via command line "swriter.exe <path\filename.odt>" to get viable font substitution. So, assume there some difference in method of opening with a locale established, or not, impacts the available font ranges.