Bug 93302 - Greek letter phi and varphi in formulas are swapped when saving in.doc, .docx or .rtf
Summary: Greek letter phi and varphi in formulas are swapped when saving in.doc, .docx...
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Writer (show other bugs)
Version:
(earliest affected)
4.3.7.2 release
Hardware: All All
: high major
Assignee: Not Assigned
URL:
Whiteboard: target:5.1.0 target:5.0.3
Keywords:
Depends on:
Blocks:
 
Reported: 2015-08-09 15:06 UTC by jc_denton_2052
Modified: 2016-10-25 19:21 UTC (History)
6 users (show)

See Also:
Crash report or crash signature:


Attachments
Document ocntaining the greek characters that wil be swapped if saved in .doc, .docx or .rtf (12.65 KB, application/vnd.oasis.opendocument.text)
2015-08-09 15:41 UTC, jc_denton_2052
Details
.docx file containing the swapped phi and varphi (4.86 KB, application/vnd.openxmlformats-officedocument.wordprocessingml.document)
2015-08-09 15:43 UTC, jc_denton_2052
Details
Same code points but different fonts (11.99 KB, application/vnd.oasis.opendocument.text)
2015-08-17 02:30 UTC, Regina Henschel
Details

Note You need to log in before you can comment on or make changes to this bug.
Description jc_denton_2052 2015-08-09 15:06:46 UTC
I'm working on a text file with formulas.
I noticed that if I insert the letter φ (%phi) in a formula and then save in .doc, .dox or .rtf, if I close and reopen the file, all the φ are turned in ϕ.
Vice versa, if I insert %varphi (ϕ) in a formula, save in that formats, close and reopen, all the ϕ are turned in φ.
Comment 1 Joel Madero 2015-08-09 15:29:54 UTC
Please attach a sample file. Setting to NEEDINFO - once you attach a simple test file (odt and then the corresponding doc, docx, and rtf) please set to UNCONFIRMED.

In the future please remember the following when reporting:
1) Reproducible steps in enumerated form is always best:
a) Open writer;
b) Open attached odt;
c) Save file as doc;
d) close LibreOffice;
e) Open the newly created doc file


2) Always attach relevant simple test cases.


Thanks!

If you have time to help out - our QA team is looking for volunteers, feel free to come say hello: http://webchat.freenode.net/?channels=libreoffice-qa
Comment 2 jc_denton_2052 2015-08-09 15:41:07 UTC
Created attachment 117788 [details]
Document ocntaining the greek characters that wil be swapped if saved in .doc, .docx or .rtf
Comment 3 jc_denton_2052 2015-08-09 15:43:06 UTC
Created attachment 117789 [details]
.docx file containing the swapped phi and varphi

Thank you for your suggestions Joel, I uploaded the two sample files as requested.
Comment 4 Joel Madero 2015-08-09 15:47:27 UTC
Interesting bug - confirmed:

Ubuntu 15.04 x64
LibreOffice Version: 5.0.0.4 (don't upgrade version as version is oldest version not latest confirmed)

Setting to:

New;
Major - loss of data;
High - default for major bugs
Comment 5 Julien Nabet 2015-08-09 16:55:13 UTC
I submitted a patch to review:
https://gerrit.libreoffice.org/#/c/17618/1
Comment 6 Regina Henschel 2015-08-17 02:30:45 UTC
Created attachment 117956 [details]
Same code points but different fonts

I think, the problem is not only in the Catalog, but in the font OpenSymbol.
The font "Cambria Math", which is used in OOXML formulas in Word, has the glyphs as it is shown in the current Unicode charts http://www.unicode.org/charts/PDF/U0370.pdf, but OpenSymbol has the glyphs the other way round.

Currently %varphi is set to U+03C6 in the Catalog. ODF specifies to use MathML for ODF. In  http://www.w3.org/2003/entities/2007doc/#phi the MathML entity "varphi" is set to code point U+03D5 (= ϕ).

There has been some confusion about phi and varphi in the standards, read discussion and links in https://bugzilla.mozilla.org/show_bug.cgi?id=321438. And the meaning of \varphi in LaTeX is different from the entity 'varphi' in MathML. But I think, we should try to move StarMath closer to the current version of MathML, because MathML is normative for ODF.

So my suggestion would be, not only to change the code point in the Catalog, but exchange the glyphs in OpenSymbol too. That way LibreOffice comes nearer to the standards and still keeps the old assignment of %varphi to the straight glyph variant of phi.
Comment 7 Julien Nabet 2015-08-17 05:37:10 UTC
Thank you Regina for your feedback.

Following Khaled's comments, I abandonned the patch because I don't know how to fix this (see https://gerrit.libreoffice.org/#/c/17618/).
I mean, changing .sfd isn't as easy as changing source code.
Comment 8 Khaled Hosny (inactive) 2015-08-18 01:02:54 UTC
https://gerrit.libreoffice.org/#/c/17822/

The glyphs for %phi and %varphi symbols were swapped in commit 81001f2c89e5932a8bfde26aacb9277b59146dff (back in 2009), as part of https://bz.apache.org/ooo/show_bug.cgi?id=105084 (see the document attached there), but no justification was given. Given the shape of the symbols after that change contradicts the ones in the Unicode code charts and other popular math fonts, I simply swapped them back.
Comment 9 Khaled Hosny (inactive) 2015-08-18 01:09:42 UTC
(In reply to Regina Henschel from comment #6)
> Currently %varphi is set to U+03C6 in the Catalog. ODF specifies to use
> MathML for ODF. In  http://www.w3.org/2003/entities/2007doc/#phi the MathML
> entity "varphi" is set to code point U+03D5 (= ϕ).

This document seems to be obsolete (see the “W3C Editor‘s Draft” on the margin), this should be the latest one http://www.w3.org/TR/MathML2/isogrk3.html and it has phi mapped to the straight symbol while varphi to the open one.
Comment 10 Regina Henschel 2015-08-18 16:39:57 UTC
The current definition is in http://www.w3.org/TR/xml-entity-names/#phi, referenced from http://www.w3.org/TR/MathML3/chapter7.html#chars.entities.

Notice, that there is a difference between the entity varphi in MathML and the command \varphi of TeX, e.g. in Wikipedia https://de.wikipedia.org/wiki/Hilfe:TeX#Griechische_Buchstaben.

Therefore there exists no unambiguous assignment between "varphi" and glyph. You can consider
(1) aligned with MathML: %varphi = ϕ = U+03D5 (straight glyph)
(2) aligned with TeX: %varphi = \varphi = U+03C6 (curly glyph)
We have to look which kind of assignment fits best to %varphi. 

Currently %varphi is written as UTF-8 CF86 (= Ux03C6) into MathML. That results in a curly-phi glyph when rendering the MathML in a browser, or when importing the MathML into equitation editor in Word, or when opening an .odt-file in Word.

Now look at it from a user point of view. The user had used %varphi because he wanted a straight phi (not knowing, that the MathML in the file format gives a curly phi). When you now swap the glyphs in OpenSymbol font, such documents will show a curly phi and the old document is broken.

Therefore my suggestion is to swap the glyphs in OpenOffice _and_ the code point assignment in the Catalog. That way, the old documents still show a straight phi, after the corrected OpenSymbol font is used, because the StarMath command from the annotation is used and not the MathML, when opening old documents, and the predefined assignments are not embedded into the object. Such way older documents do not break. It would mean, that from now on the interpretation (1) would be used.

From an interoperability point of view, the effective result has to be so, that the glyph the user sees in a Math-OLE object in LO must be the same as if the user would render the MathML of the object in a browser.

I personally like interpretation (1) anyway, because MathML is the normative part in the file format, and I think we should try to align StarMath to MathML when ever possible. My wish is to get rid of StarMath in the long term.
Comment 11 Khaled Hosny (inactive) 2015-08-18 17:00:21 UTC
I think the swapping of the glyph that was done in 2009 is the bug here, it simply means that documents created before this date was rendered differently. Also other ODF implementations was/are likely rendering it differently already. So that is where the compatability was broken and we are fixing it now, though a bit too late.

There is no direct relationship between entity names and StarMath symbol names, so that name of the entity shouldn’t be a concern here.

I think we shouldn’t be changing the mapping/meaning of StarMath symbol names, it should be immutable IMO.
Comment 12 Commit Notification 2015-08-18 19:45:58 UTC
Khaled Hosny committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=75a8b1cf84ac718aa1c8d671cd27b0b10307f50a

tdf#93302: Fix OpenSymbol %phi and %varphi glyphs

It will be available in 5.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 13 Caolán McNamara 2015-08-18 19:47:30 UTC
do any of the tables in unotools/source/misc/fontcvt.cxx need to be updated now for this ?
Comment 14 Khaled Hosny (inactive) 2015-08-18 20:02:08 UTC
(In reply to Caolán McNamara from comment #13)
> do any of the tables in unotools/source/misc/fontcvt.cxx need to be updated
> now for this ?

I don’t think so, there weren’t any changes to the table corresponding to the font change in 2009.
Comment 15 Julien Nabet 2015-09-20 07:35:56 UTC
Any plan to backport the patch on 5.0 branch?
Comment 16 Khaled Hosny (inactive) 2015-09-20 21:37:04 UTC
(In reply to Julien Nabet from comment #15)
> Any plan to backport the patch on 5.0 branch?

Not by me, I just tried to chery-pick it to libreoffice-5-0 locally and got a merge conflict and I have no time to check it.
Comment 17 Julien Nabet 2015-09-29 20:13:51 UTC
(In reply to Khaled Hosny from comment #16)
> (In reply to Julien Nabet from comment #15)
> > Any plan to backport the patch on 5.0 branch?
> 
> Not by me, I just tried to chery-pick it to libreoffice-5-0 locally and got
> a merge conflict and I have no time to check it.

I gave a try to backport libreoffice-5-0 branch with this one: https://gerrit.libreoffice.org/#/c/19000/
I didn't have any conflicts, perhaps I missed something?
Comment 18 Commit Notification 2015-09-30 10:12:29 UTC
Khaled Hosny committed a patch related to this issue.
It has been pushed to "libreoffice-5-0":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=bf6e219ee57bb17fdb4f176bc4888a540cbeb290&h=libreoffice-5-0

tdf#93302: Fix OpenSymbol %phi and %varphi glyphs

It will be available in 5.0.3.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 19 Julien Nabet 2015-09-30 10:16:18 UTC
Now we can put this one to FIXED.