Bug 36703 - Regressions with the new graphite2
Summary: Regressions with the new graphite2
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: UI (show other bugs)
Version:
(earliest affected)
3.4.0 Beta3
Hardware: x86 (IA32) All
: medium critical
Assignee: martin_hosken
QA Contact: martin_hosken
URL:
Whiteboard:
Keywords:
: 36510 (view as bug list)
Depends on:
Blocks: 34420 mab3.5
  Show dependency treegraph
 
Reported: 2011-04-30 00:23 UTC by László Németh
Modified: 2013-11-15 11:11 UTC (History)
9 users (show)

See Also:
Crash report or crash signature:


Attachments
Valgrind memcheck log (18.54 KB, text/plain)
2011-04-30 00:28 UTC, László Németh
Details
Test document for discretionary hyphenation (9.72 KB, application/vnd.oasis.opendocument.text)
2011-04-30 00:32 UTC, László Németh
Details
Screenshots for the previous test document (35.55 KB, image/png)
2011-04-30 00:38 UTC, László Németh
Details
test document for font features (40.27 KB, application/vnd.oasis.opendocument.text)
2011-05-05 06:15 UTC, László Németh
Details
PDF output with regressions (LibO 3.4) (197.14 KB, application/pdf)
2011-05-05 06:16 UTC, László Németh
Details
PDF output without regressions (LibO 3.3.2) (216.95 KB, application/pdf)
2011-05-05 06:17 UTC, László Németh
Details
ODT test file for hyphenation (11.52 KB, application/vnd.oasis.opendocument.text)
2011-09-01 09:00 UTC, László Németh
Details
PDF output of LibreOffice 3.4 with missing hyphenation (39.10 KB, application/pdf)
2011-09-01 09:02 UTC, László Németh
Details
PDF output of LibreOffice 3.3 with correct hyphenation (39.41 KB, application/pdf)
2011-09-01 09:03 UTC, László Németh
Details
Halt by name feature of Linux Libertine G (8.29 KB, application/vnd.oasis.opendocument.text)
2012-01-10 01:13 UTC, László Németh
Details
Halt by zooming dotted tabulation formatted with Linux Libertine G (8.68 KB, application/vnd.oasis.opendocument.text)
2012-01-10 01:24 UTC, László Németh
Details

Note You need to log in before you can comment on or make changes to this bug.
Description László Németh 2011-04-30 00:23:46 UTC
Typing "Qu" halts LibreOffice 3.4 (using Linux Libertine G). In the previous versions, "Qu" changed to a ligature. Valgrind log is attached.

Other differences:

a) bad kerning, eg. kerning is missing from the first position in "AVAVAVA".

b) With the experimental version of Linux Libertine G, positioning of diacritical marks with GDL feature "attach" doesn't work with the new engine.

c) Discretionary hyphenation (screenshot: http://openoffice.org/bugzilla/attachment.cgi?id=33622) is missing. See http://openoffice.org/bugzilla/show_bug.cgi?id=58558 for test data. I attach a new Hungarian test data, too (it needs only the default Hungarian language data of LibreOffice).

A positive difference, that the bad kerning at hyphenation within ligatures is fixed.
Comment 1 László Németh 2011-04-30 00:28:45 UTC
Created attachment 46182 [details]
Valgrind memcheck log

Some ligature replacements, like Qu halt LibreOffice. Valgrind detects some problems.
Comment 2 László Németh 2011-04-30 00:32:47 UTC
Created attachment 46183 [details]
Test document for discretionary hyphenation
Comment 3 László Németh 2011-04-30 00:38:16 UTC
Created attachment 46184 [details]
Screenshots for the previous test document

Left: LibreOffice 3.3.2 Right: LibreOffice 3.4 (missing discretionary hyphenation)
Comment 4 László Németh 2011-05-02 14:02:37 UTC
Confirmed in the Bug 36510, also on Windows.
Comment 5 László Németh 2011-05-05 06:12:19 UTC
It seems, I can fix the program halt with the simplification of the GDL conditions, but there are a lot of other regressions in hyphenation, kerning and font features. I tested them with the last Graphite compiler (version 4), too. For the font features I attach some new test files.
Comment 6 László Németh 2011-05-05 06:15:07 UTC
Created attachment 46358 [details]
test document for font features
Comment 7 László Németh 2011-05-05 06:16:38 UTC
Created attachment 46359 [details]
PDF output with regressions (LibO 3.4)
Comment 8 László Németh 2011-05-05 06:17:24 UTC
Created attachment 46360 [details]
PDF output without regressions (LibO 3.3.2)
Comment 9 Petr Mladek 2011-05-12 07:24:19 UTC
I have reproduced it with LO-3.4.0-beta5 build on SLED11-SP1.

The Libertine font is not default => it should not block the 3.4.0 release => lovering severity a bit.

Though, it might hit quite many people => adding to the list of most annoying bugs.

Note that the root of the problem might be the same as for the bug #36510.
Comment 10 martin_hosken 2011-05-14 01:37:41 UTC
I have been unable to rebuild the LinLibertineG font for testing. If someone could send me the .gdl and .ttf from which the font is built that would help. I only need one font, say LinLibertineG_Re

Anyway, from what I could build, I would recommend that dealing with the some 4000+ warnings generated by the compiler would help matters a lot. The only warning you can safely ignore is type 3521 (overlaps), and we are working on making it that all warnings *should* be headed. Note that U+0028 for example is not a valid glyph reference, I think you mean unicode().

Thus this font only causes problems when ligatures are created, which only really come into play with some of the more obscure replacements (for example texm=1) since they do not have the correct associations set up on the ligature replacements. The new engine is not as forgiving in this area as the previous was (in the desire for speed). I'll try harder to make it more robust here (given the requested files above), but may not get it done in time for release.

There is new code in the grcompiler code (which can easily be built for linux  (and probably anything) btw, no need to rely on wine) that should automatically try to insert associations for you. Also pass constraints work with the new engine (only) which will probably help speed this font up. This font is going to be relatively slow with all the feature checking that is in there at the moment. And remember that a failed feature test takes just as long as a successful one.
Comment 11 martin_hosken 2011-05-14 01:40:50 UTC
Oops. Forgot to add that I would be happy to help get the GDL on this font to not cause the warnings (given said ttf and gdl ;)
Comment 12 László Németh 2011-05-14 17:06:21 UTC
(In reply to comment #11)
> Oops. Forgot to add that I would be happy to help get the GDL on this font to
> not cause the warnings (given said ttf and gdl ;)

Martin, thanks in advance. Here is the requested input files (development version, I have already removed the old U+ unicode notations, also fixed several warnings):

http://www.numbertext.org/tmp/LinLibertine_Re-4.7.5.gdl
http://www.numbertext.org/tmp/LinLibertine_Re-4.7.5.ttf

Output, with strange behaviour in LibreOffice 3.4:

http://www.numbertext.org/tmp/LinLibertine_Re-4.7.5_G.ttf

Eg. see the feature hlig: the s t -> s_t replacement is occured in starting positions without setting hlig.
Comment 13 martin_hosken 2011-05-16 01:00:27 UTC
Wow! You are really hammering the limits with this font. The main fun is that there is a 128 rule limit on any state, and LinLibertineG as it stands goes waay over that. Fixes to the engine have been added to mitigate this, but at the expense of a possibly expected rule not firing. But at least the engine doesn't crash.

The test document now renders correctly, except for standard ligatures on Qu and some other edge cases, due to the horrendous number of rules involving poor Qu in one pass.

The way around this is to split up some of these feature substitutions into separate passes, bearing mind that the previous substitutions will have happened. It may also help address some multi-feature interactions that I haven't tested, but could imagine getting somewhat interesting.
Comment 14 Michael Meeks 2011-05-17 22:32:04 UTC
*** Bug 36510 has been marked as a duplicate of this bug. ***
Comment 15 László Németh 2011-05-18 06:17:19 UTC
(In reply to comment #13)
I'm glad of the fixes. Thanks for your help. I will try split the complex rules related to the Qu and its feature-dependent capitalization (by the way, the previously attached gdl has already had simpler conditions, fixing the program halting in OOo 3.4 beta). 

Is it possible that the hyphenation problem described in my first comment hasn't been solved yet? If yes, maybe comparising the Graphite integration with the old one could help to fix this important feature.

Also I tried to fix the cursor movement based on your letter, but the suggested rule forbids the substitution:

unicode(0x005C) p("a") p("l") p("p") p("h") p("a") > _ _ _ _ _ unicode(0x03B1):(1 6) // \alpha -> α
Comment 16 László Németh 2011-09-01 08:58:40 UTC
Most of the reported font features work with Graphite 2, but

(1) feature "name" for number to number name conversion halts LibreOffice 3.4,

(2) hyphenation is much worse with the Graphite 2 integration (simply not enough good for the minimal typographical requirements). See the attached documents (grhyph.pdf, grhyph.odt, grhyph_in_LibreOffice_3.3.pdf).

(3) Upcoming version of the Linux Libertine G font handles combining diacritics with correct positioning plus kerning (SIL Graphite fonts have excellent support for combining diacritics, but not kerning) with Graphite 1 engine. Unfortunatelly, Graphite 2 in LibreOffice 3.4.x hasn't supported these fonts, yet (bug report: https://sourceforge.net/tracker/?func=detail&aid=3402499&group_id=66144&atid=513479).
Comment 17 László Németh 2011-09-01 09:00:55 UTC
Created attachment 50818 [details]
ODT test file for hyphenation
Comment 18 László Németh 2011-09-01 09:02:06 UTC
Created attachment 50819 [details]
PDF output of LibreOffice 3.4 with missing hyphenation
Comment 19 László Németh 2011-09-01 09:03:29 UTC
Created attachment 50820 [details]
PDF output of LibreOffice 3.3 with correct hyphenation
Comment 20 Michael Meeks 2011-09-05 07:51:48 UTC
Martin - is this fixed with the update to graphite2-1.0.2 as merged to libreoffice-3-4 and expected in 3.4.4 ? Laszlo - have you tested vs. master - or a recent snapshot build of the -3-4 branch ?
Comment 21 László Németh 2011-09-06 04:56:30 UTC
According to Martin (see SF.net issue), positioning has already been fixed for 3.5/3.4.4. I will check it.
Comment 22 László Németh 2011-09-09 08:00:39 UTC
Positioning works fine (test result: http://www.numbertext.org/tmp/combiningtest.pdf)

I have found two new bugs testing Italic correction in LibreOffice 3.4:

Bad condition handling (https://sourceforge.net/tracker/?func=detail&aid=3406802&group_id=66144&atid=513479)

LibreOffice 3.4 doesn't handle the rules with space (https://sourceforge.net/tracker/?func=detail&aid=3406723&group_id=66144&atid=513479)
Comment 23 László Németh 2011-10-19 01:45:14 UTC
I have made a new SF issue for the hyphenation bug with more examples (including the possible fix by direct character formatting, like in the old, but solved bugs of the previous Graphite integration):

http://sourceforge.net/tracker/index.php?func=detail&aid=3425291&group_id=66144&atid=513479
Comment 24 László Németh 2012-01-10 01:06:11 UTC
Graphite 2 has been instable with Linux Libertine G, yet. Next test files halt LibreOffice. I believe, the first one (haltwithname.odt) can be solved by fixing this non-intended backward incompatibility with Graphite rules: http://sourceforge.net/tracker/?func=detail&aid=3409703&group_id=66144&atid=513479. For the second one (haltwithzoomdotted) I will modify the resource intensive dots-to-ellipsis rules temporarily, but the strange bug (LibreOffice halts by zooming the document) suggests other kind of problems, too.
Comment 25 László Németh 2012-01-10 01:07:39 UTC
Graphite 2 has been instable with Linux Libertine G, yet. Next test files halt LibreOffice. I believe, the first one (haltwithname.odt) can be solved by fixing this non-intended backward incompatibility with Graphite rules: http://sourceforge.net/tracker/?func=detail&aid=3409703&group_id=66144&atid=513479. For the second one (haltwithzoomdotted) I will modify the resource intensive dots-to-ellipsis rules temporarily, but the strange bug (LibreOffice halts by zooming the document) suggests other kind of problems, too.
Comment 26 László Németh 2012-01-10 01:13:00 UTC
Created attachment 55369 [details]
Halt by name feature of Linux Libertine G

During converting longer numbers to text Linux Libertine G uses multiple back references to the same slot, but it seems, this is a serious problem for Graphite 2.
Comment 27 László Németh 2012-01-10 01:24:29 UTC
Created attachment 55370 [details]
Halt by zooming dotted tabulation formatted with Linux Libertine G

The halt is strange, because the (maybe) related dots-to-ellipsis substitution depends from leading letters.
Comment 28 martin_hosken 2012-01-10 02:35:51 UTC
Does your font compile with no warnings?
Comment 29 László Németh 2012-01-10 03:45:00 UTC
I have got a lot of warning messages, so I will try to reduce the reported problems. I have found this warning-free test case for a similar (maybe related) LibreOffice/Graphite 2 halt: http://sourceforge.net/tracker/?func=detail&aid=3409703&group_id=66144&atid=513479.
Comment 30 László Németh 2012-01-11 03:05:23 UTC
The second problem (halt by dotted tabulation) caused by some conflict between the old conditional dot-to-ellipsis substitution and the converted (unconditional) one from the new version of the OpenType version of Linux Libertine font families, so I will fix it soon.
Comment 31 László Németh 2012-01-16 03:17:13 UTC
Crash by dotted tabulation has been successfully fixed with the new release of the Linux Libertine/Biolinum G fonts (under integration, thanks to András Tímár). Related Graphite bug report: https://sourceforge.net/tracker/?func=detail&aid=3474371&group_id=66144&atid=513479
Comment 32 Rainer Bielefeld Retired 2012-04-05 09:00:08 UTC
3.4 lifecycle is terminated, so shift to “Bug 37361 LibreOffice 3.5 most annoying bugs”.

I can't contribute anything concerning bug problem.
Comment 33 martin_hosken 2012-04-05 19:20:56 UTC
Please retest under 3.5 since changes have been made. If still a problem, will look more deeply. Which issues are still outstanding, which are resolved? It would help if someone could post an authoritative font (preferably both final .ttf and source GDL) to test against. TIA
Comment 34 Axel Niedenhoff 2012-05-03 22:26:17 UTC
It seems to me that automatic hyphenation is still a problem. I tried with LO 3.5.3 (Windows XP). I did not try with the Hungarian documents, but checked with German. Here is what I did:

* First I wanted to establish that automatic hyphenation itself works. So I worked with a non-Graphite font (DejaVu Serif). I used the German word “ihren”, which should hyphenate as “ih-ren”. I typed the word, filled the space before it with random text until the test word did not fit on the line any more. I found that LO properly hyphenated the word.

* I tried the same with Linux Libertine G. In this case, LO did not hyphenate the word. Inserting a manual hyphenation at the proper place worked, though.

* I tried the same with OpenOffice.org 3.3. There, automatic hyphenation works with both fonts.

So, to conclude: In a case where automatic hyphenation can be shown to work with a non-Graphite font, it does not work with a Graphite font.

By the way, the Linux Libertine G I used in my tests was the one dated 2012-01-16.
Comment 35 Michael Meeks 2012-05-04 02:43:25 UTC
wow this is a frankenstein bug with lots of different issues; it'd be nice to split them out and turn it into a tracker bug I guess.
Comment 36 László Németh 2012-05-04 06:36:34 UTC
(In reply to comment #35)
> wow this is a frankenstein bug with lots of different issues; it'd be nice to
> split them out and turn it into a tracker bug I guess.

I have made one for the most annoying hyphenation problem: Bug 49486
Comment 37 László Németh 2012-07-06 07:12:24 UTC
I closed this issue, because my crash with name feature of Linux Libertine G has been solved with removing an unintentional Graphite cache size limitation by the environmental variable SAL_GRAPHITE_CACHE_SIZE.

I will make a new issue for the space-related Graphite regressions.

Many thanks for the hyphenation patch and its integration (unfortunatelly, I couldn't test it, yet).