Bug 66715 - EDITING: Special spaces aren’t displayed correctly
Summary: EDITING: Special spaces aren’t displayed correctly
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: graphics stack (show other bugs)
Version:
(earliest affected)
4.1.0.2 rc
Hardware: Other Linux (All)
: medium normal
Assignee: Khaled Hosny (inactive)
URL:
Whiteboard: BSA target:4.2.0 target:4.1.2
Keywords: regression
Depends on:
Blocks:
 
Reported: 2013-07-08 22:25 UTC by Aelfinn
Modified: 2013-09-07 08:58 UTC (History)
5 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Aelfinn 2013-07-08 22:25:13 UTC
Problem description:

Special spaces such as thin space (u2006) and hair space (u200a) aren’t displayd correctly but as normal spaces. 

Steps to reproduce:
1. Insert e.g. u2006 in your text by pressing Ctrl-Shift+u+2006

Current behavior: A normal-width space is inserted

Expected behavior: A thin space should be inserted

              
Operating System: Linux (Other)
Version: 4.1.0.2 rc
Last worked in: 4.0.4.1 rc
Comment 1 Aelfinn 2013-07-08 22:27:03 UTC
Last working version was 4.1.0rc1. Sorry for misinformation.
Comment 2 Aelfinn 2013-07-19 14:35:28 UTC
This bug is still there in 4.1.0-rc3 – did a fresh install, new profile, but the problem persists.
Comment 3 Michael Stahl (CIB) 2013-07-30 14:39:22 UTC
apparently using the wrong font, problem happens with 
/usr/share/fonts/liberation/LiberationSerif-Regular.ttf
but not the bundled one.

weirdly it also only happens with an installed LO but not a dev-install.

fixed on master, now get about 30 more duplicate fonts discarded.
Comment 4 Commit Notification 2013-07-30 14:47:38 UTC
Michael Stahl committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=1c48e4efa2369e5708798bdefb46b74a86415d00

fdo#66715: fontconfig: try harder to ignore duplicate fonts



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 5 Commit Notification 2013-07-31 15:51:48 UTC
Michael Stahl committed a patch related to this issue.
It has been pushed to "libreoffice-4-1":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=ff8a6b661dc304cb3a858facbd0dbda66ff01641&h=libreoffice-4-1

fdo#66715: fontconfig: try harder to ignore duplicate fonts


It will be available in LibreOffice 4.1.1.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 6 Aelfinn 2013-08-20 15:32:03 UTC
This bug is *not* resolved in 4.1.1rc1: special spaces work again for only some fonts, for most fonts installed on my system, though, they still don’t.
Comment 7 Michael Stahl (CIB) 2013-08-20 21:13:22 UTC
indeed, problem still happens with system fonts, for example
"Bitstream Charter" on Fedora 19, which worked with LO 4.0.4.2.

bibisect says:
3def5194ddaf9c4d766b71527874bd1a973b43e5..9a7603187eb5cc580d33212ee147f9ac89de55f4

which contains this:

commit bff8fa97e16f0f06fddc5545ea36c8bd2b18a580
Author:     Khaled Hosny <khaledhosny@eglug.org>
AuthorDate: Mon May 6 11:08:29 2013 +0200

    Enable HarfBuzz by default
    
    HarfBuzz integration should be functional now, so to give it more wider
    testing it is made now a required dependency on non-Windows non-Mac OSs.
    By default text layout is now done by HarfBuzz but ICU LayoutEngine is
    kept as a fallback and can be enabled with SAL_USE_ICULE env variable.


... and runninng current libreoffice-4-1 branch exhibits the problem
(thin spaces have full width, presumably because the font does not
contain glyphs for them (Bitstream Charter does not according to fontforge)),
but if one sets the SAL_USE_ICULE variable the thin spaces work,
so likely the harfbuzz layout is not perfect yet and missing some fallback.

Khaled, do you have some time to look at this?
Comment 8 Khaled Hosny (inactive) 2013-08-20 23:07:23 UTC
This is a HarfBuzz “feature”, when a character is not supported by the font it tries to use Unicode normalization to find an equivalent character that is supported by the font. It seems all those special spaces decompose to the ASCII space, U+0020, so HarfBuzz ends up using it and we end up thinking the font supports that character and do not trigger our font fallback mechanisms. We can query the font’s “cmap” table to find what characters are supported before passing the text to HarfBuzz, but then we lose HarfBuzz’s normalization fallback completely which is quite a useful feature in many other cases.

Behdad, is there a way to handle the spaces differently on HarfBuzz’s side?
Comment 9 Behdad Esfahbod 2013-08-20 23:11:09 UTC
(In reply to comment #8)
> This is a HarfBuzz “feature”, when a character is not supported by the font
> it tries to use Unicode normalization to find an equivalent character that
> is supported by the font. It seems all those special spaces decompose to the
> ASCII space, U+0020, so HarfBuzz ends up using it and we end up thinking the
> font supports that character and do not trigger our font fallback
> mechanisms. We can query the font’s “cmap” table to find what characters are
> supported before passing the text to HarfBuzz, but then we lose HarfBuzz’s
> normalization fallback completely which is quite a useful feature in many
> other cases.

So, how is your font fallback triggered?  Do you check for notdef glyphs in harfbuzz output or what?  Because harfbuzz doesn't choose the font, harfbuzz client does.

> Behdad, is there a way to handle the spaces differently on HarfBuzz’s side?

Of course, just use a custom unicode-funcs that has whatever you want for the decompose_compatibility().
Comment 10 Khaled Hosny (inactive) 2013-08-20 23:23:58 UTC
(In reply to comment #9)
> (In reply to comment #8)
> > This is a HarfBuzz “feature”, when a character is not supported by the font
> > it tries to use Unicode normalization to find an equivalent character that
> > is supported by the font. It seems all those special spaces decompose to the
> > ASCII space, U+0020, so HarfBuzz ends up using it and we end up thinking the
> > font supports that character and do not trigger our font fallback
> > mechanisms. We can query the font’s “cmap” table to find what characters are
> > supported before passing the text to HarfBuzz, but then we lose HarfBuzz’s
> > normalization fallback completely which is quite a useful feature in many
> > other cases.
> 
> So, how is your font fallback triggered?  Do you check for notdef glyphs in
> harfbuzz output or what?

We just check for glyph index == zero and trigger fallback in this case.

> > Behdad, is there a way to handle the spaces differently on HarfBuzz’s side?
> 
> Of course, just use a custom unicode-funcs that has whatever you want for
> the decompose_compatibility().

I should have thought of that :) We currently use the built-in ICU unicode-funcs, not sure if one can change just decompose_compatibility(), but I’ll give it a try
Comment 11 Behdad Esfahbod 2013-08-21 04:20:07 UTC
(In reply to comment #10)
> (In reply to comment #9)
> > (In reply to comment #8)
> > > This is a HarfBuzz “feature”, when a character is not supported by the font
> > > it tries to use Unicode normalization to find an equivalent character that
> > > is supported by the font. It seems all those special spaces decompose to the
> > > ASCII space, U+0020, so HarfBuzz ends up using it and we end up thinking the
> > > font supports that character and do not trigger our font fallback
> > > mechanisms. We can query the font’s “cmap” table to find what characters are
> > > supported before passing the text to HarfBuzz, but then we lose HarfBuzz’s
> > > normalization fallback completely which is quite a useful feature in many
> > > other cases.
> > 
> > So, how is your font fallback triggered?  Do you check for notdef glyphs in
> > harfbuzz output or what?
> 
> We just check for glyph index == zero and trigger fallback in this case.

Good to know.  Thanks.

> > > Behdad, is there a way to handle the spaces differently on HarfBuzz’s side?
> > 
> > Of course, just use a custom unicode-funcs that has whatever you want for
> > the decompose_compatibility().
> 
> I should have thought of that :) We currently use the built-in ICU
> unicode-funcs, not sure if one can change just decompose_compatibility(),
> but I’ll give it a try

Sure you can.  That's what the parent arg in hb_unicode_functs_create() is for.  And you can chain to the parent, check the result to see if it's a sole space character, and return false.
Comment 12 Khaled Hosny (inactive) 2013-08-22 19:34:19 UTC
Given the nature of compatibility decomposition, I think we are better off avoiding it entirely and allowing for font fallback instead, so I just did that.
Comment 13 Commit Notification 2013-08-22 19:39:39 UTC
Khaled Hosny committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=1f9d1e15634b1ef1dea37dc00c7bacb4f97996d6

fdo#66715: Avoid compatibility decomposition with HarfBuzz



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 14 Behdad Esfahbod 2013-08-22 20:04:12 UTC
You can use a static ufuncs.  No need to create one every time.
Comment 15 Khaled Hosny (inactive) 2013-08-22 20:52:58 UTC
(In reply to comment #14)
> You can use a static ufuncs.  No need to create one every time.

Makes sense, I should do the same for font functions as well.
Comment 16 Khaled Hosny (inactive) 2013-08-23 23:05:46 UTC
(In reply to comment #15)
> (In reply to comment #14)
> > You can use a static ufuncs.  No need to create one every time.
> 
> Makes sense, I should do the same for font functions as well.

http://cgit.freedesktop.org/libreoffice/core/commit?id=94a742fbc092eb75dbcffcbce0ac96679c90add4
Comment 17 Commit Notification 2013-08-24 04:46:14 UTC
Khaled Hosny committed a patch related to this issue.
It has been pushed to "libreoffice-4-1":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=fbaedf8bd45d3cc4409c92a4b798d9d9788567ed&h=libreoffice-4-1

fdo#66715: Avoid compatibility decomposition with HarfBuzz


It will be available in LibreOffice 4.1.2.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.