Bug 132688 - Display of diacritics added to existing files is broken in lines with punctuation or footnotes/endnotes
Summary: Display of diacritics added to existing files is broken in lines with punctua...
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Writer (show other bugs)
Version:
(earliest affected)
6.4.0.2 rc
Hardware: All Linux (All)
: medium normal
Assignee: Not Assigned
URL:
Whiteboard: target:7.1.0 target:7.0.2 target:6.4.7
Keywords: bibisected, bisected, regression, text:rtl
Depends on:
Blocks: Hebrew
  Show dependency treegraph
 
Reported: 2020-05-04 16:35 UTC by Yotam Benshalom
Modified: 2023-06-22 03:14 UTC (History)
4 users (show)

See Also:
Crash report or crash signature:


Attachments
First line is fine, second line demonstrates problem (see description) (13.00 KB, application/msword)
2020-05-04 16:35 UTC, Yotam Benshalom
Details
Same file saved as .odt. same behaviour (11.84 KB, application/vnd.oasis.opendocument.text)
2020-05-10 16:43 UTC, Yotam Benshalom
Details
Adding diacritics places them to the left of their intended place (4.43 KB, application/vnd.openxmlformats-officedocument.wordprocessingml.document)
2021-02-21 19:10 UTC, Yotam Benshalom
Details
Another file, same old bug: if you add vowel marks, they will be misplaced (7.82 KB, application/vnd.openxmlformats-officedocument.wordprocessingml.document)
2021-06-21 15:35 UTC, Yotam Benshalom
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Yotam Benshalom 2020-05-04 16:35:37 UTC
Created attachment 160353 [details]
First line is fine, second line demonstrates problem (see description)

When adding diacritic marks to words in an existing .doc file created in word, the diacritic marks are misplaced if the line contains any pointer to a footnote or endnote. The do not appear in the letter, but rather a bit after it.
A workaround is to delete the letter, type it again and add the diacritic mark.

To reproduce:
1. Open the attached document.
2. Place the caret after the first letter in the first line and add the Hebrew diacritic mark Kamatz. Result: the kamatz is added correctly under the letter.
3. Place the caret after the first letter in the second line and add the Hebrew diacritic mark Kamatz. Result: the kamatz is added incorrectly, slightly after the letter.

This seems small, but it makes the work of adding diacritic marks to long texts impossible.
Comment 1 Yotam Benshalom 2020-05-06 21:07:37 UTC
Update: this problem occurs whenever the line contains a punctuation mark, like a coma, a question mark and so on. The line must contain only letters for the diacritics to be displayed correctly upon adding them.
Comment 2 Yotam Benshalom 2020-05-06 21:19:54 UTC
I discovered more about this bug:
1. It happens with native .odt files as well as with .doc files.
2. It is a bug in display only. When I added the diacritics to the file, they are shown in the wrong place, but if I save it, closes and re-opens it the diacritics are are all in their correct position.

Could this be a problem with the late implementation of the harfbuzz engine?
Comment 3 Eyal Rozenberg 2020-05-10 16:33:22 UTC
(In reply to Yotam Benshalom from comment #2)


So, if it happens with native ODT files - can you give new reproduction instructions using only LibreOffice only?

Also, when opening the file you attached - the Qamatz is misplaced on _both_ lines - it's too far to the left. It is indeed rectified when saving, closing and reloading. I checked with 

Version: 6.4.0.3
Build ID: b0a288ab3d2d4774cb44b62f04d5d28733ac6df8
CPU threads: 4; OS: Linux 5.2; UI render: default; VCL: gtk3; 
Locale: he-IL (en_IL); UI-Language: en-US
Comment 4 Yotam Benshalom 2020-05-10 16:43:45 UTC
Created attachment 160611 [details]
Same file saved as .odt. same behaviour
Comment 5 Yotam Benshalom 2020-05-10 16:46:37 UTC
I can't reproduce this with an .odt file made from scratch, only with a .doc saved as .odt, so this may be related to the font not existing on our system (maybe it uses one kind of hinting for the letters and another for the diacritics?). However, LibreOffice should allow me to work in this situation, which is very common.
Comment 6 Buovjaga 2020-08-28 13:37:58 UTC
(In reply to Yotam Benshalom from comment #5)
> I can't reproduce this with an .odt file made from scratch, only with a .doc
> saved as .odt, so this may be related to the font not existing on our system
> (maybe it uses one kind of hinting for the letters and another for the
> diacritics?). However, LibreOffice should allow me to work in this
> situation, which is very common.

If it is only with .doc files, it might somehow be related to compatibility options. Some of these are in Tools - Options - Writer - Compatibility, but not all of them are available in the UI. See for example bug 76005. Thanks to Mike Kaganski for the tip.
Comment 7 Yotam Benshalom 2020-08-28 14:11:19 UTC
(In reply to Buovjaga from comment #6)
> (In reply to Yotam Benshalom from comment #5)
> > I can't reproduce this with an .odt file made from scratch, only with a .doc
> > saved as .odt, so this may be related to the font not existing on our system
> > (maybe it uses one kind of hinting for the letters and another for the
> > diacritics?). However, LibreOffice should allow me to work in this
> > situation, which is very common.
> 
> If it is only with .doc files, it might somehow be related to compatibility
> options. Some of these are in Tools - Options - Writer - Compatibility, but
> not all of them are available in the UI. See for example bug 76005. Thanks
> to Mike Kaganski for the tip.

Changing all of the values in Tools - Options - Writer - Compatibility does not resolve the problem.

This is an issue introduced in latest versions of LO, so I don't think it is an anomaly in the source file - it seems more like a regression.

I found the settings.xml file within the .odt, but how should I know if something there is relevant? I have no idea what the values there mean.
Comment 8 Buovjaga 2020-08-28 14:25:19 UTC
(In reply to Yotam Benshalom from comment #0)
> To reproduce:
> 1. Open the attached document.
> 2. Place the caret after the first letter in the first line and add the
> Hebrew diacritic mark Kamatz. Result: the kamatz is added correctly under
> the letter.

How should I add it? I pasting first, but the result was incorrect on the first line as well. Then I added Hebrew keyboard in KDE and tried all keys, but I could not find Kamatz.

You can also bibisect the problem yourself, if you want: https://wiki.documentfoundation.org/QA/Bibisect
Comment 9 Yotam Benshalom 2020-08-28 14:44:42 UTC
> How should I add it? I pasting first, but the result was incorrect on the
> first line as well. Then I added Hebrew keyboard in KDE and tried all keys,
> but I could not find Kamatz.


Use Hebrew (lyx) keyboard. Change the input language to Hebrew (lyx), and then press Shift+E.
I know that for some, the diacritics are placed wrong on the firs line as well.
Comment 10 Buovjaga 2020-08-28 15:06:53 UTC
(In reply to Yotam Benshalom from comment #9)
> > How should I add it? I pasting first, but the result was incorrect on the
> > first line as well. Then I added Hebrew keyboard in KDE and tried all keys,
> > but I could not find Kamatz.
> 
> 
> Use Hebrew (lyx) keyboard. Change the input language to Hebrew (lyx), and
> then press Shift+E.
> I know that for some, the diacritics are placed wrong on the firs line as
> well.

Yeah, I can't do it with my Finnish/Swedish keyboard.

So I encourage you to bibisect the issue.

If you need a gentle start, you can try this tutorial I wrote: https://wiki.documentfoundation.org/QA/Bibisect/Bibisecting_tutorial

Btw. I noticed that the text changed from RTL to LTR since this commit https://git.libreoffice.org/core/commit/7b9878671a74dc9043d0cdc93e019fdd7d622e8c but I guess it is the expected result.
Comment 11 Yotam Benshalom 2020-08-28 15:18:07 UTC
I'll try learning the art of git and bidisection, but adding a keyboard and removing it is just a matter of a few clicks in Gnome language definitions :-)
Comment 12 Buovjaga 2020-08-28 15:29:15 UTC
(In reply to Yotam Benshalom from comment #11)
> I'll try learning the art of git and bidisection, but adding a keyboard and
> removing it is just a matter of a few clicks in Gnome language definitions
> :-)

Argh, I had overlooked the "Modification" dropdown in KDE settings. Now I could test with Shift-E, but for me, all the Kamatzes land in the correct place. So you have to learn bibisecting anyway :)
Comment 13 Yotam Benshalom 2020-08-29 14:17:16 UTC
OK, I did it! Thanks for encouraging me.

Here is the faulty commit:

 source 4b2d4f3c4a68361a6bc03c9ab110ce9376b14b20  

This looks weird, because the commit seems to deal with bulleted lists, but I checked using git cherckout HEAD^1 and this is definitely the culprit.

I added the regression keyword.

P.S. The new automatic forced LTR seems to create a serious problem - it took my perfectly good RTL doc file and changed it into LTR. Should I file this as a new bug, or comment in tdf#99197?
Comment 14 Buovjaga 2020-08-29 15:39:05 UTC
(In reply to Yotam Benshalom from comment #13)
> OK, I did it! Thanks for encouraging me.
> 
> Here is the faulty commit:
> 
>  source 4b2d4f3c4a68361a6bc03c9ab110ce9376b14b20  
> 
> This looks weird, because the commit seems to deal with bulleted lists, but
> I checked using git cherckout HEAD^1 and this is definitely the culprit.
> 
> I added the regression keyword.
> 
> P.S. The new automatic forced LTR seems to create a serious problem - it
> took my perfectly good RTL doc file and changed it into LTR. Should I file
> this as a new bug, or comment in tdf#99197?

Congrats! The result does seem plausible after reading the code a bit.

https://git.libreoffice.org/core/+/4b2d4f3c4a68361a6bc03c9ab110ce9376b14b20%5E%21

I see you commented in bug 99197. Justin is subscribed to it, so he will get it in his inbox.
Comment 15 Yotam Benshalom 2020-09-01 20:16:34 UTC
Added noelgrandin@gmail.com to CC list.
Comment 16 Commit Notification 2020-09-03 13:42:07 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/18e4367c33f327cf09985105bde583cdcc7b2a46

tdf#132688 diacritics broken in lines with punctuation

It will be available in 7.1.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 17 Yotam Benshalom 2020-09-03 13:45:03 UTC
Thanks! Open source is so satisfying.
Comment 18 Buovjaga 2020-09-03 13:59:45 UTC
Yotam: tomorrow you could try a daily build. For Linux, you may find them on https://libreoffice.soluzioniopen.com/

If you don't see it on the appimage page, you can use this method to automatically download the build and generate an appimage yourself: https://wiki.documentfoundation.org/Installing_in_parallel/Linux#Automated_installation

You can see the latest builds here: https://dev-builds.libreoffice.org/daily/master/current.html
Comment 19 Yotam Benshalom 2020-09-05 23:18:37 UTC
The issue is fixed in the latest build :-)
Comment 20 Xisco Faulí 2020-09-06 08:28:13 UTC
(In reply to Yotam Benshalom from comment #19)
> The issue is fixed in the latest build :-)

Thanks for checking, I've backported it to libreoffice-7-0 in https://gerrit.libreoffice.org/c/core/+/101972
Comment 21 Commit Notification 2020-09-06 11:29:49 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "libreoffice-7-0":

https://git.libreoffice.org/core/commit/9ec49c6c2dd58eb60ca0ac5e99edee9ee098302a

tdf#132688 diacritics broken in lines with punctuation

It will be available in 7.0.2.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 22 Yotam Benshalom 2020-09-20 10:59:31 UTC
Issue is fixed for me with version 7.0.2.1.
Comment 23 Commit Notification 2020-09-29 15:29:58 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "libreoffice-6-4":

https://git.libreoffice.org/core/commit/48cf4c412cc4ddeccf1d21a81b3b4859b6a79459

tdf#132688 diacritics broken in lines with punctuation

It will be available in 6.4.8.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 24 Commit Notification 2020-09-30 18:28:53 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "libreoffice-6-4-7":

https://git.libreoffice.org/core/commit/04692e1898448d2344bab67f9e7b2f32198d4411

tdf#132688 diacritics broken in lines with punctuation

It will be available in 6.4.7.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 25 Yotam Benshalom 2021-02-21 19:10:59 UTC
Created attachment 169947 [details]
Adding diacritics places them to the left of their intended place

This is not completely fixed.

I attach a file which demonstrates the same bugged behaviour on LibreOffice 7.1. Please help - this makes my day job significantly harder :-)
Comment 26 Eyal Rozenberg 2021-02-26 19:25:45 UTC
(In reply to Yotam Benshalom from comment #25)
> This is not completely fixed.

In a previous comment you said it was fixed.

> I attach a file which demonstrates the same bugged behaviour on LibreOffice 7.1. 

I assume the reproduction instruction is: "Add a QAMATS" after the first letter"?
Comment 27 Yotam Benshalom 2021-02-27 17:37:46 UTC
Apparently it was not fixed for all cases. The attached file demonstrates the same problem on Version 7.1.1.2.
The instructions are for adding Qamatz, yes. But I eventually started a new bug for this: bug #140622
Comment 28 Yotam Benshalom 2021-06-21 15:34:11 UTC
Here is another file which demonstrates the same bug.
Comment 29 Yotam Benshalom 2021-06-21 15:35:04 UTC
Created attachment 173061 [details]
Another file, same old bug: if you add vowel marks, they will be misplaced
Comment 30 QA Administrators 2023-06-22 03:14:42 UTC
Dear Yotam Benshalom,

To make sure we're focusing on the bugs that affect our users today, LibreOffice QA is asking bug reporters and confirmers to retest open, confirmed bugs which have not been touched for over a year.

There have been thousands of bug fixes and commits since anyone checked on this bug report. During that time, it's possible that the bug has been fixed, or the details of the problem have changed. We'd really appreciate your help in getting confirmation that the bug is still present.

If you have time, please do the following:

Test to see if the bug is still present with the latest version of LibreOffice from https://www.libreoffice.org/download/

If the bug is present, please leave a comment that includes the information from Help - About LibreOffice.
 
If the bug is NOT present, please set the bug's Status field to RESOLVED-WORKSFORME and leave a comment that includes the information from Help - About LibreOffice.

Please DO NOT

Update the version field
Reply via email (please reply directly on the bug tracker)
Set the bug's Status field to RESOLVED - FIXED (this status has a particular meaning that is not 
appropriate in this case)


If you want to do more to help you can test to see if your issue is a REGRESSION. To do so:
1. Download and install oldest version of LibreOffice (usually 3.3 unless your bug pertains to a feature added after 3.3) from https://downloadarchive.documentfoundation.org/libreoffice/old/

2. Test your bug
3. Leave a comment with your results.
4a. If the bug was present with 3.3 - set version to 'inherited from OOo';
4b. If the bug was not present in 3.3 - add 'regression' to keyword


Feel free to come ask questions or to say hello in our QA chat: https://web.libera.chat/?settings=#libreoffice-qa

Thank you for helping us make LibreOffice even better for everyone!

Warm Regards,
QA Team

MassPing-UntouchedBug