Bug 70633 - Writer EDITING: Unable to use AltGr to produce supplementary plane chars
Summary: Writer EDITING: Unable to use AltGr to produce supplementary plane chars
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Writer (show other bugs)
(earliest affected)
3.3.1 release
Hardware: x86-64 (AMD64) Windows (All)
: medium normal
Assignee: Not Assigned
Keywords: bibisectRequest, regression
Depends on:
Blocks: Shortcuts-AltGR
  Show dependency treegraph
Reported: 2013-10-18 21:23 UTC by nrs
Modified: 2019-05-15 12:09 UTC (History)
5 users (show)

See Also:
Crash report or crash signature:

Ahmao Unicode keyboard + layout (335.01 KB, application/zip)
2013-10-18 21:27 UTC, nrs
Tengwar test font, keyboard, & layout (465.88 KB, application/zip)
2014-02-26 06:35 UTC, nrs
picking SMP glyphs via Special Character dialog (58.36 KB, image/png)
2019-05-14 23:18 UTC, V Stuart Foote

Note You need to log in before you can comment on or make changes to this bug.
Description nrs 2013-10-18 21:23:47 UTC
It is not possible to type into Writer any *supplementary plane* chars mapped to the AltGr state.  These chars simply do not appear on screen, on export, or on printing.  By contrast, AltGr-mapped *BMP* chars are OK, as are *supplementary plane* chars mapped to normal or shift states.  There is also no problem typing AltGr-mapped *supplementary plane* chars into Calc & Impress.  These chars can also be readily produced with AltGr in other apps like Word & Excel.

This problem is attested in the following builds & OS:
- LibO on 64-bit English Windows 7 Ultimate SP1
- LibO on 64-bit English Windows 7 Ultimate SP1

Steps to reproduce bug:
1. download & install Miao Unicode font at https://github.com/phjamr/MiaoUnicode/blob/master/MiaoUnicode-Regular.ttf?raw=true
2. unzip attached archive & install Ahmao keyboard (run setup.exe)
3. run Writer & set all Basic Fonts (CTL) to Miao Unicode (in Tools|Options|LibreOffice Writer)
4. select Miao Unicode font
5. on Windows language bar, select MR
6. refer to the 3 .jpg files in attached archive for keys mapped in each state
7. type any mapped keys in normal state: Miao chars (mostly big letters) appear in document -- expected
8. type any mapped keys in shift state: Miao chars (mostly small letters) appear in document -- expected
9. type any mapped keys in AltGr state: *nothing appears* -- BUG!!!
10. select 'Export to pdf' or 'Print' from the File menu: no sign of any chars typed in #9 -- BUG!!!

Control experiments with other apps:
11. open Calc/Impress/Word/Excel/PowerPoint: blank document appears
12. repeat #4-8: expected results
13. type any mapped keys in AltGr state: Miao chars (big letters with wart) appear in document -- expected

Further observations: Writer may be cutting off the 2nd half of the surrogate pair that represents a supplementary char, as suggested by the following test:
14. in Writer, repeat #7-8 in any order multiple times, keeping all chars on 1 line
15. repeat #9 multiple times, continuing on the same line as in #14
16. copy the whole line & paste into Word: a line of blanks appears with trailing 'checked' boxes
17. in Word, place cursor after any 'checked' boxes
18. press Ctrl-X: the string D81B appears, which is the Unicode code point of the 1st half of any surrogate pair representing chars in this (Miao) block

Further control experiment with BMP chars (Hebrew):
19. download & install SBL Hebrew font at http://www.sbl-site.org/Fonts/SBL_Hbrw.ttf
20. download & install SBL Hebrew Tiro keyboard at http://www.sbl-site.org/Fonts/BiblicalHebrewTiro.zip
21. download & refer to SBL Hebrew Tiro keyboard layout at http://www.sbl-site.org/Fonts/BiblicalHebrewTiroManual.pdf
22. in Writer, select SBL Hebrew font
23. on Windows language bar, select HE
24. type any mapped keys in any state: Hebrew chars appear as expected
Comment 1 nrs 2013-10-18 21:27:11 UTC
Created attachment 87837 [details]
Ahmao Unicode keyboard + layout
Comment 2 nrs 2013-10-18 21:32:05 UTC
BTW, the Miao script is used by multiple language groups with as many as 6 million people.  It would be most appreciated if you could fix this ASAP.  Thank you!
Comment 3 nrs 2013-12-19 21:54:54 UTC
This bug is also attested in the following LibO versions, all on 64-bit English Windows 7 Ultimate SP1:

Symptoms (in step 9) are not exactly the same but similar in nature:

LibO a square appears & then changes to crossed boxes on typing further chars mapped to AltGr
LibO nothing appears
LibO similar to plus random line splitting or glyph change on screen which will revert to original on further typing
- if a char mapped to AltGr starts the line, a square appears & subsequent consecutive AltGr-mapped chars on the same line are not rendered at all
- otherwise, chars mapped to AltGr are rendered as crossed boxes
- these symbols (square/crossed box/nothing) will randomly switch among themselves so that sometimes part of the line just disappears & comes back again on typing further chars

Version field set to 3.3.1 release to reflect earliest occurrence attested.
Comment 4 nrs 2014-02-26 06:33:22 UTC
Another case of Writer blocking output of SMP chars mapped to AltGr & shift-AltGr states:

Steps to reproduce bug with Tengwar (re-encoded at U+1CC00..1CC7F for testing):
1. unzip tengmod.zip attached
2. install Tengwar font (right-click tengmod.ttf & select Install)
3. install Tengwar keyboard (run setup.exe)
4. run Writer
5. select Tengwar Telcontar Mod font
6. on Windows language bar, select IS
7. refer to the 4 .jpg files in attached archive for keys mapped in each state
8. type any mapped keys in AltGr or shift-AltGr state: squares & nothing appear -- BUG!!!

According to dev comments re a parallel bug in AOO, this appears to be caused by processing Unicode codepoints via the operator ::rtl::OUString::operator sal_Unicode*, which is defined as only 16-bit unsigned int, effectively dropping all codepoints beyond BMP (thus the surrogate pair cut-off phenomenon).  Pls. kindly revise the code base in concern to use a 32-bit type & fix this defect ASAP.
Comment 5 nrs 2014-02-26 06:35:12 UTC
Created attachment 94751 [details]
Tengwar test font, keyboard, & layout
Comment 6 nrs 2014-02-26 07:24:19 UTC
Attested also in Writer for both Miao & Tengwar.
Comment 7 Björn Michaelsen 2014-03-06 16:54:04 UTC
Enhancement request as per comment 4. Please also add the mentioned bug in 'see also:'
Comment 8 Urmas 2014-03-06 17:48:09 UTC
The parallel bug for OO has no comments.

Also, the proper keyboard handling is anything but enhancement.
Comment 9 nrs 2014-03-06 18:12:51 UTC
Added link to parallel OO bug #124312.  Note that OO dev added a link in the 'Blocks' field to the possible root cause (per comment #4).  That 2nd link is also included in the 'See Also' field here.  For the sake of the several million Miao script users, pls. kindly look into this issue at your earliest convenience.  Thanks!
Comment 10 QA Administrators 2014-10-05 23:05:36 UTC Comment hidden (obsolete)
Comment 11 QA Administrators 2014-11-02 16:43:03 UTC Comment hidden (obsolete)
Comment 12 nrs 2014-11-20 07:51:04 UTC
We already provided the requested info 8.5 months ago (see comment 9 added on 2014-03-06 18:12:51 UTC, 24 mins. after comment 8 & 78 mins after comment 7).  We don't know what more info you need.  Pls. specify.

We just tested with the latest release on 32-bit English Windows Vista Business & the bug is still there:
- No chars in the AltGR & shift-AltGR states can be produced.
- For Ahmao, nothing appeared at all (rf. Description, step #9).
- For Tengwar, spaces appeared instead (cf. Comment 4, step #8).
- When the 'empty' string is copied into Word 2003 & Alt-X pressed,
  - D81B appeared for Ahmao (rf. Description, step #18; NB: 'Ctrl-X' should read 'Alt-X').
  - D833 appeared for Tengwar.

Apparently the surrogate cut-off problem is still not resolved.  Pls. kindly fix it asap.  Thanks!
Comment 13 nrs 2015-11-20 03:59:32 UTC
It's been a year already but nothing has happened apart from defect confirmation.  Pls. kindly follow up asap.  Thanks!
Comment 14 V Stuart Foote 2016-09-05 21:05:27 UTC
With a current build of LO master/5.3.0alpha0+ (Build ID: 696e83b663d4f3e00f23947613f9f3916a4dd14d) on install of Miao unicode font linked in comment 0 -- the Special Character dialog displays and selects glyphs and allows paste of SMP codepoints into the Writer canvas.  The <alt>+X unicode toggle also seems to work correctly for the SMP codepoints.

Could you check if your preferred AltGr based text IME remains balky in Writer for entering SMP codepoints for the font.
Comment 15 Xisco Faulí 2017-09-29 08:51:05 UTC Comment hidden (obsolete, spam)
Comment 16 nrs 2019-05-14 22:57:10 UTC
This issue was confirmed to be resolved in the following build & OS:
- Version (x64) on 64-bit English Windows 7 Ultimate SP1

*HOWEVER*, it is still attested in the following NEWER builds & OS:
- Version (x64) on 64-bit English Windows 7 Ultimate SP1
- Version (x64) on 64-bit English Windows 10 Pro
- Version (x64) on 64-bit English Windows 10 Pro

A minor difference is instead of squares, black diamonds with a white question mark inside are produced.  On pressing Alt-X, the same surrogate values show up as before.

This indicates other fixes in the code base for 6.x has broken the fix for this issue in 5.4 => regression!
Comment 17 V Stuart Foote 2019-05-14 23:18:51 UTC
Created attachment 151415 [details]
picking SMP glyphs via Special Character dialog

(In reply to nrs from comment #16)
On Windows 10 Ent 64-bit en-US (1807) with
Version: (x64)
Build ID: 170a9c04e0ad25cd937fc7a913bb06bf8c75c11d
CPU threads: 8; OS: Windows 10.0; UI render: GL; VCL: win; 
Locale: en-US (en_US); UI-Language: en-US
Calc: CL

Can not confirm mishandling of SMP Unicode glyphs. The IME may be wrong, but handling to document canvas with Special Character dialog, or <Alt>+X toggle correctly assigns the SMP Unicode glyphs.
Comment 18 nrs 2019-05-15 01:32:34 UTC
Thank you for your quick response!  I can also get the desired char via the Special Char dialog, but pls. kindly reread comment #1: This issue is about *typing*, not about pointing & clicking.  And the various control experiments with other apps, including Calc, clearly confirm that the keyboards are working.

The only conclusion possible is that:
- either something in the Writer 6.x code base broke the fix that was in place in 5.4, or
- the fix available in 5.4 is not incorporated into Writer 6.x at all.

Pls. kindly double-check.  Thanks!