Bug 99402 - FORMATTING_Black question mark for Mathtype symbols (Brackets)
Summary: FORMATTING_Black question mark for Mathtype symbols (Brackets)
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: filters and storage (show other bugs)
Version:
(earliest affected)
5.0.0.5 release
Hardware: All All
: medium normal
Assignee: Mike Kaganski
URL:
Whiteboard: target:5.3.0
Keywords: bibisected, bisected, regression
: 95902 100629 100651 (view as bug list)
Depends on:
Blocks: 101889
  Show dependency treegraph
 
Reported: 2016-04-19 17:30 UTC by terateck3
Modified: 2016-09-04 06:21 UTC (History)
6 users (show)

See Also:
Crash report or crash signature:


Attachments
Since I can not add more than one attachment, I place the image file in the Powerpoint document. (73.94 KB, application/wps-office.pptx)
2016-04-19 17:30 UTC, terateck3
Details
LO 3.5, linux (72.71 KB, image/png)
2016-04-20 17:04 UTC, raal
Details
LO 5.3.0.0 daily build (08-31), Linux (146.26 KB, image/png)
2016-08-31 23:51 UTC, Aron Budea
Details

Note You need to log in before you can comment on or make changes to this bug.
Description terateck3 2016-04-19 17:30:21 UTC
Created attachment 124514 [details]
Since I can not add more than one attachment, I place the image file in the Powerpoint document.

I have provided a sample Powerpoint 2013 slide for testing on anyone's Libreoffice versions.

I can can confirm that it works o.k with the very same version (Libreoffice 5.1.1.3) of the Portable App version on Windows 7. The image slide shows this. However, there seems to be an OLE image overlayed on the symbols.

I have installed various fonts to try and solve the problem, but to no avail.

I am currently running a current version of Manjaro Linux.
Comment 1 raal 2016-04-20 17:04:12 UTC
I can confirm with Version: 5.2.0.0.alpha0+
Build ID: 0f27cc992a99568e46ffe807ef9dbb5ba0bc601f
CPU Threads: 4; OS Version: Linux 4.2; UI Render: default; 
TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2016-04-12_23:40:16

in LO 3.5 it looks better, regression
Comment 2 raal 2016-04-20 17:04:31 UTC
Created attachment 124537 [details]
LO 3.5, linux
Comment 3 Aron Budea 2016-06-18 16:09:00 UTC
Looks fine in 4.4.0.3 as well.
Doesn't look fine in 5.0.5.2.
Comment 4 Aron Budea 2016-06-19 01:45:42 UTC
Bug 95902 might be the same issue.
Comment 5 Aron Budea 2016-08-21 16:21:20 UTC
git bisect start 'latest' 'oldest'
# good: [0c30a2c797b249d0cd804cb71554946e2276b557] source-hash-45aaec8206182c16025cbcb20651ddbdf558b95d
git bisect good 0c30a2c797b249d0cd804cb71554946e2276b557
# good: [2ce02b2ce56f12b9fcb9efbd380596975a3a5686] source-hash-17d714eef491bda2512ba8012e5b3067ca19a5be
git bisect good 2ce02b2ce56f12b9fcb9efbd380596975a3a5686
# good: [40875247f0002056effdf6d2fbe43627691cd86c] source-hash-93f0b14458a618ad575cd446680e5c4aa7d87bdc
git bisect good 40875247f0002056effdf6d2fbe43627691cd86c
# skip: [61f66b1a251477193d796411ca95f50d606ead45] source-hash-3fd5f8919ec2256c70ff26c14cb9f8065c5cb2f1
git bisect skip 61f66b1a251477193d796411ca95f50d606ead45
# good: [e7374cd735af2344dae55be40946d96633d2f6ee] source-hash-8a91528a3e03fe6e2923c33327b687ecf57adb0b
git bisect good e7374cd735af2344dae55be40946d96633d2f6ee
# bad: [541837707e7b0c5f5335180de535043c43e78e8d] source-hash-0811de12ee6727bbb9d4265217833ba02301eed8
git bisect bad 541837707e7b0c5f5335180de535043c43e78e8d
# good: [2faf2a5126e3ccf78be3d6619b571358bb7af742] source-hash-2523972f6a066488c649ab97dcba4f458126f18b
git bisect good 2faf2a5126e3ccf78be3d6619b571358bb7af742
# bad: [036709cec55938932b487542cdbace379c2651e2] source-hash-d8eee8e4d1a303044bf34b28c2e95bd6da23fd79
git bisect bad 036709cec55938932b487542cdbace379c2651e2
# bad: [85e4f6947f419de71f7079926bcb198f7071402f] source-hash-ceb6f473837261f2a6e43e028ce9da3daccc2f6c
git bisect bad 85e4f6947f419de71f7079926bcb198f7071402f
# bad: [39abb47f0db81fb201197bc56d3c27dfc0ef43cf] source-hash-ffaa8e48b5a98bb7dd1891da09bf796467cf849f
git bisect bad 39abb47f0db81fb201197bc56d3c27dfc0ef43cf
# bad: [54a2e10b5b5ef17bdbceb9c2a922fb2eed30a6d7] source-hash-3d9edac679a6846bdb21540a42c04e73dd56ea40
git bisect bad 54a2e10b5b5ef17bdbceb9c2a922fb2eed30a6d7
# bad: [e3f04edf3662fd513aa45ec28f92f3f280e5ca61] source-hash-399fa0ee267a1927c14ce7ca7c19881761034154
git bisect bad e3f04edf3662fd513aa45ec28f92f3f280e5ca61
# good: [4d0fdc1bc9bd53620cecd5bb08a9399817a7cfd9] source-hash-4f864949a9484bbf21911859398743bfe2b1430f
git bisect good 4d0fdc1bc9bd53620cecd5bb08a9399817a7cfd9
# bad: [356a690e35b48b585827c8b66ab8ac6a76d8000a] source-hash-cbf8a5c8b208c0a6b87a19f799dc5d5613e61e5f
git bisect bad 356a690e35b48b585827c8b66ab8ac6a76d8000a
# bad: [8756a2db22ac4c3252c8c0295ac8fb0ee6864606] source-hash-c6bc9b33d5cac1ea40a829754004fde6ae16d8b1
git bisect bad 8756a2db22ac4c3252c8c0295ac8fb0ee6864606
# first bad commit: [8756a2db22ac4c3252c8c0295ac8fb0ee6864606] source-hash-c6bc9b33d5cac1ea40a829754004fde6ae16d8b1
Comment 6 Aron Budea 2016-08-21 16:24:09 UTC
Adding Mike Kaganski to CC.

Mike, can you please take a look?
 8756a2db22ac4c3252c8c0295ac8fb0ee6864606 is the first bad commit
commit 8756a2db22ac4c3252c8c0295ac8fb0ee6864606
Author: Matthew Francis <mjay.francis@gmail.com>
Date:   Wed May 27 22:54:28 2015 +0800

    source-hash-c6bc9b33d5cac1ea40a829754004fde6ae16d8b1
    
    commit c6bc9b33d5cac1ea40a829754004fde6ae16d8b1
    Author:     Mike Kaganski <mikekaganski@hotmail.com>
    AuthorDate: Wed May 6 19:10:05 2015 +1000
    Commit:     Caolán McNamara <caolanm@redhat.com>
    CommitDate: Sun May 10 20:02:49 2015 +0000
    
        tdf#74301: WMF: use LibreOffice locale on OEM_CHARSET/DEFAULT_CHARSET
    
        [MS-WMF] 2.1.1.5 CharacterSet Enumeration defines DEFAULT_CHARSET
        "Specifies a character set based on the current system locale;
        for example, when the system locale is United States English,
        the default character set is ANSI_CHARSET"
    
        OEM_CHARSET is defined as follows: "Specifies a mapping to one of
        the OEM code pages, according to the current system locale setting"
    
        Currently, LO WMF import treats these values as synonim for
        RTL_TEXTENCODING_MS_1252. This patch fixes this to use LibreOffice
        locale setting instead.
    
        Strictly speaking, this is not quite correct for OEM. E.g., for
        Russian, DEFAULT_CHARSET is equal to codepage 1251, while OEM_CHARSET
        is codepage 866. But I don't know how to distinguish these two.
    
        Change-Id: Id5eaf070cde040bd6eb1db28f1a498d647ba227e
        Reviewed-on: https://gerrit.libreoffice.org/15641
        Tested-by: Jenkins <ci@libreoffice.org>
        Reviewed-by: Caolán McNamara <caolanm@redhat.com>
        Tested-by: Caolán McNamara <caolanm@redhat.com>
Comment 7 Aron Budea 2016-08-21 16:33:50 UTC
*** Bug 95902 has been marked as a duplicate of this bug. ***
Comment 8 Aron Budea 2016-08-21 16:55:16 UTC
Occurs in Windows as well, as described in [1] in bug 95902.

[1] https://bugs.documentfoundation.org/show_bug.cgi?id=95902#c9
Comment 9 Aron Budea 2016-08-21 17:01:16 UTC
*** Bug 100629 has been marked as a duplicate of this bug. ***
Comment 10 Mike Kaganski 2016-08-22 14:39:09 UTC
Oh, couldn't imagine that my fix to Bug 74301 will let all those bugs out... Bugs in WMF-generation software :)

Instead of correctly writing the LOGFONT structures for "Symbol" font with CharSet = RTL_TEXTENCODING_SYMBOL, they put there CharSet = OEM_CHARSET.

So, I need to enable special handling for that special-case font.
A patch is submitted to gerrit: https://gerrit.libreoffice.org/28322
Comment 11 Mike Kaganski 2016-08-22 14:41:16 UTC
*** Bug 100651 has been marked as a duplicate of this bug. ***
Comment 12 Urmas 2016-08-23 17:52:31 UTC
DEFAULT_CHARSET is not a valid value. Its only use is to be passed in a CreateFont function, it has absolutely no sense otherwise, and it on the Russian system it is ANSI_CHARSET which corresponds to the codepage 1251.
Comment 13 Mike Kaganski 2016-08-23 21:46:06 UTC
(In reply to Urmas from comment #12)
1. It's not true. MS documentation lists it as a normal value **in on-disk data**.
2. Even if it would be so, tell that to those who create files.
Comment 14 Commit Notification 2016-08-30 05:41:31 UTC
Mike Kaganski committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=20f6a6b159c69771dc0e087f63b6c701908e32e2

tdf#99402: fix Metafile Font handling

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 15 Mike Kaganski 2016-08-31 22:53:22 UTC
Let's call this fixed.
If your testing shows that this bug isn't resolved, please reopen the issue.
Comment 16 Aron Budea 2016-08-31 23:51:12 UTC
Created attachment 127106 [details]
LO 5.3.0.0 daily build (08-31), Linux

Tested this and duplicate reports (except the one that required Japanese setting) with daily build in Windows 7 and Ubuntu 16.04 (see build details at the end).
All look as expected in Windows 7. (great!)
Brackets are generally replaced with squares in Linux. (not great)

See attached image for how the slide looks in Linux.
In 4.4 only the top brackets were rectangles (sometimes the middle ones changed to rectangles during zooming), though the best would be to get rid of all rectangles.

Bug 95902's https://bugs.documentfoundation.org/attachment.cgi?id=120616 :
Shows rectangles instead of brackets, in 4.4 they were brackets.

Bug 100651's https://bugs.documentfoundation.org/attachment.cgi?id=125956 :
Shows rectangles instead of both normal and square brackets, in 4.4 they were brackets.

Bug 100629's https://bugs.documentfoundation.org/attachment.cgi?id=125931 :
Shows sum symbol properly both in daily build and 4.4.


Version: 5.3.0.0.alpha0+
Build ID: 4a63c145dcce8411c5707f6b99877cc87a4f6c5d
CPU Threads: 1; OS Version: Linux 4.4; UI Render: default; 
TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2016-08-31_19:54:16
Locale: en-US (en_US.UTF-8); Calc: group

Version: 5.3.0.0.alpha0+
Build ID: 34dced99c33a97dab86c4538fa267ad4ad4fb41f
CPU Threads: 4; OS Version: Windows 6.1; UI Render: default; 
TinderBox: Win-x86@42, Branch:master, Time: 2016-08-31_00:00:01
Locale: hu-HU (hu_HU); Calc: CL
Comment 17 Mike Kaganski 2016-09-01 00:48:05 UTC
(In reply to Aron Budea from comment #16)

Hm. It seems that under Linux, the Symbol font uses different encoding?
Maybe I'll need to use RTL_TEXTENCODING_MS_1252 for Symbol unconditionally, instead of proper RTL_TEXTENCODING_SYMBOL?

But you say that for some cases, 4.x had recctangles, too? Personally I'm not a fan of making another kludge for some quirks in improper Linux font...
Comment 18 Mike Kaganski 2016-09-03 00:18:39 UTC
1. Please see comment 1 and attachment 124537 [details] from comment 2.
You may see that Linux has always had problems displaying Symbol glyphs.
This issue had been filed against specific issue: a regression introduced in 5.0. This issue had been resolved. I feel that reopening it in comment 16 is wrong. The issues that Aron Budea points to, should be tracked in another bug, that will be "inherited from OOo".

2. Under Linux, after the regression is fixed, the attacment 124514 from comment 0 shows two problems: one with brackets, and another one with "h" with stroke in second line in h^2/2m. Both problems are caused by absent fonts: brackets - from Symbol; stroked h - from MT Extra. Symbol font is mapped to OpenSymbol under Linux. (I only mention MT Extra here for completeness/reference; it's out of scope here.)

The Symbol font that is shipped with Windows (and is used in WMFs) has Symbol charset (see https://www.microsoft.com/typography/fonts/font.aspx?FMID=1650). OpenSymbol, on the other side, only has 1252 Latin-1 charset in it.

Fixing the original Linux problem by forcing Win1252 codepage everywhere seems to work, even under Windows (at least my Win10 shows it well; seems that Windows somehow makes internal recoding here). But I'm not sure if that's a correct solution. IMO, more correct solution would be adding the Symbol charset to OpenSymbol font itself.

If we choose not to fix the OpenSymbol font, the required code change is starting from line 163 of winmtf.cpp.
I close this again as resolved fixed; please find the relevant bug (or file a new one if required) and cc me there; I'll prepare the patch for it, and if reviewed positively, the Symbol problem could be fixed.