Created attachment 71693 [details]
nonbreaking characters under LO

You can see nonbreaking characters under LO displayed with a gray background, as if they were highlighted.

Created attachment 71694 [details]
nonbreaking characters under MS Office

The same text under MS Office Word is displayed as regular characters, which is visually easier to read.

(In reply to comment #3)
> I'm sorry just not following this bug report:
> 
> What might help:
> 
> Provide reproducible steps
> An attachment of an actual document showing these gray areas
> 
> I don't see gray areas at all in Linux with or without non printing
> characters visible.
> 
> Marking as NEEDINFO - with additional information you can mark the bug as
> UNCONFIRMED but please provide very explicit steps so it's easier to follow
> what's going on. 
> 
> 
> Thanks!

I'm under Linux, using latest stable LO available and I can still reproduce it every time. I also use LO daily on Windows. In both cases, I can see what I reported and explained. I can also see it on all 5 computers at work. Were you typing nonbreaking characters as explained in the screenshots?

I'll be attaching an example right now.

Created attachment 89725 [details]
Example of nonbreaking characters with grey background

All the nonbreaking characters have a grey background as if they were highlighted in grey. They are between the brackets []. They can be seen with or without activating the display of special characters.

(In reply to comment #5)
> Created attachment 89725 [details]
> Example of nonbreaking characters with grey background
> 
> All the nonbreaking characters have a grey background as if they were
> highlighted in grey. They are between the brackets []. They can be seen with
> or without activating the display of special characters.

replace "special characters" by "printing characters" in the sentence.

Thank you for reporting this enhancement request! I can confirm that this is a valid enhancement request on:
Version: 4.2.0.0.alpha0+Build ID: 09a4c4d176ff97ab8ff4027af83a549991667baf
Date:   Sat Nov 16 20:32:43 2013 +0100 
Platform: Ubuntu Linux 13.04 x64
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 
As I've been able to confirm the enhancement request I am marking as:

New (confirmed)
Enhancement
Medium


Whiteboard Status - ProposedEasyHack

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 
LibreOffice is powered by a team of volunteers, every bug is confirmed (triaged) by human beings who mostly give their time for free. We invite you to join our triaging by checking out this link:
https://wiki.documentfoundation.org/QA/BugTriage

There are also other ways to get involved including with marketing, UX, documentation, and of course developing -  http://www.libreoffice.org/get-help/mailing-lists/. 

Lastly, good bug reports help tremendously in making the process go smoother, please always provide reproducible steps (even if it seems easy) and attach any and all relevant material

The grey background is actually field shading, which can be toggled on and off separately (View > Field Shadings). But I agree that it would be more intuitive if the highlighting of these formatting characters was linked to the visibility of non-printing characters.

Besides no-break space (U+00A0) and non-breaking hyphen (U+2011), there are other characters that have field shading, including at least soft hyphen (U+00AD), zero width space (U+200B), and word joiner (U+2060). On the other hand, narrow no-break space (U+202F) and figure space (U+2007), for example, do not have field shading, although they are also exceptional as regards line breaking.

*** Bug 63538 has been marked as a duplicate of this bug. ***

*** Bug 52526 has been marked as a duplicate of this bug. ***

With the grey being visible and invisible depending on View > Field Shadings when its not a field like page number or date, this is a bug in my opinion and not an enhancement.

I agree with Jay. The grey shading (or whatever colour) should appear only when the nonbreaking chars are made visible.

I also agree with Jay that using the field shading colour for formatting marks is a bug. Hence I have changed importance to 'medium normal'.

But what is the correct function? I think this has not been properly defined yet. And so this is the enhancement part of this bug report.

I also see the necessity to visualize this kind of characters. To display them with a shading, if the nonprinting characters are displayed, may be a solution. But in this case two things have to be taken into account: 
(1) The term 'nonprinting character' is no longer correct, because some of the formatting marks are printable. It may be better to have an additional item in the View menu in order to display formatting marks.
(2) It may be possible that background colours are used and these colours may be equal or similar to the shading colour. For this case an algorithm should be implemented in order to ensure, that there is a sufficient contrast between shading and background colour.

Like Harald suggested, i agree that the display of formatting marks should be settable in the view menu and in that way it is independent from regular and non-printable character view modes like it is right now. It should be turned off by default, as it is not needed by most regular users, similar to non-printable characters is also turned off by default.

It should be noted that there is no visual difference between a number of these kinds of characters when the gray background is shown. (e.g. non-breaking hyphen and soft hyphen)

Yup, reasonable enhancement to isolate these special characters (as listed by Simo in comment 8, any others?) from current UI handling of toggling as if field data. That UI choice was probably wrong--but does function as designed.  So really not a bug.

Simplest enhancement would be to associate with non-printing character handling of the paragraph breaks. But the better UX might come with implementing a new UI motif and unique color for these elements.

Including in non-printing would be OK for me.
But we may expect people to start complaining about that?
Whatever change is chosen: I'm a fan of easy toggling the visibility on and of.

Created attachment 112753 [details]
More detailed information about these kind of special characters

Meanwhile I found the following sentence in the help: “Field Shadings: Shows or hides field shadings in your document, including non-breaking spaces, custom hyphens, indexes, and footnotes.” If the function will be changed also this text has to be changed.

So it seems that it was intended to use the field shading also for formatting marks and hence it would not be a bug in a narrower sense. But to my opinion it is still rather mistakable. So I would appreciate if formatting marks would be highlighted in a different way. 

In order to get an impression how the different non-printing characters and formatting marks are treated, I examined the current behaviour and created a list with the results. You can find this list inside a document which I will attach to this bug report. My impression is, that there is no clear direction how these special characters are treated. I think the list is not complete, because there are special characters which are used in languages which I do not know. In the list there are also some question marks where I did not know or examine the behaviour. Feel free if you like to complete or enhance the list.

Hence the situation is a bit complicated, a more general approach should be considered. Therefore some questions should be answered. In the attached document I added a list with, what I think are the relevant questions together with some more aspects.

Created attachment 112798 [details]
Use of special characters 2.0

Thanks for the comparison table, which certainly helps in seeing the big picture. However, there seems to be confusion about some Unicode characters, so I have reordered the table and added background colors in cells as an attempt to make it a little clearer overall. I have also omitted some of the name variants (including the German names) and instead added new columns for line-break behavior, samples and notes. The character list is still not complete, since Unicode has, for example, many more space characters in different widths, but most of these allow line breaks in the same way as the normal space, so I don't regard them as particularly problematic here.

I think the primary concern is to be able to show and identify any character that behaves differently from what the user is likely to expect. As far as I can see, exceptional line breaks are the most common case of this. 

(Also take notice of the comments included in the original table by Harald. These have been omitted from my version.)

Lets see if we can get some evaluation by a dev with some code pointers.

(In reply to Harald Koester from comment #17)
> Created attachment 112753 [details]
> More detailed information about these kind of special characters

thanks a lot! Nice :)

(Side note: I expect Field: Hidden text and Field: Hidden paragraph to be shown with Ctr+F9)

> Meanwhile I found the following sentence in the help: “Field Shadings: Shows
> or hides field shadings in your document, including non-breaking spaces,
> custom hyphens, indexes, and footnotes.” If the function will be changed
> also this text has to be changed.

Indeed. Good catch!

> Hence the situation is a bit complicated, a more general approach should be
> considered. Therefore some questions should be answered. In the attached
> document I added a list with, what I think are the relevant questions
> together with some more aspects.

Yes... but in your document I see no blocking for the common opinion in this issue that toggling the visibility with non printing characters makes more sense.

(I expect that the current behavior origins from the code, that they work more like fields.)

I would say: +2 and go..

So this issue should also cover other non-printing characters that have gray backgrounds like soft hyphen, non-width optional break, and non-width no break.

sill in Version: 6.0.0.0.alpha0+
Build ID: 892c719fffa06de4c7aeab497326cad7bae9e5c6
CPU threads: 4; OS: Linux 4.10; UI render: default; VCL: gtk2; 
TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2017-09-27_03:02:09
Locale: nl-NL (nl_NL.UTF-8); Calc: group

One vote is for simplicity but the majority wants a separation of text formatting from fields. Removing UX, once it is implemented we can talk about the right visualization.

See also the discussion around URLs that are better not be treated as field. Meaning, we could check what items is a field at all.

(In reply to Yousuf Philips (jay) (retired) from comment #14)
> Like Harald suggested, i agree that the display of formatting marks should
> be settable in the view menu and in that way it is independent from regular
> and non-printable character view modes like it is right now. It should be
> turned off by default, as it is not needed by most regular users, similar to
> non-printable characters is also turned off by default.

(In reply to V Stuart Foote from comment #15)
> Yup, reasonable enhancement to isolate these special characters (as listed
> by Simo in comment 8, any others?) from current UI handling of toggling as
> if field data. [...]
> Simplest enhancement would be to associate with non-printing character
> handling of the paragraph breaks. But the better UX might come with
> implementing a new UI motif and unique color for these elements.
I vehemently support the idea of a new setting, separated from fields and non-printable characters.

This should also be considered for other components, not only Writer.

Could be an easyhack

*** Bug 121256 has been marked as a duplicate of this bug. ***

No objection to set bug 121256 as a duplicate to this one, but don't forget to add U+200E LEFT-TO-RIGHT MARK, U+200E RIGHT-TO-LEFT MARK and U+202C POP DIRECTIONAL FORMATTING to the list of formatting marks (only the first one is present in attachment 112798 [details] and POP is missing in attachment 112753 [details])

*** Bug 128411 has been marked as a duplicate of this bug. ***

*** Bug 132394 has been marked as a duplicate of this bug. ***

*** Bug 63538 has been marked as a duplicate of this bug. ***

*** Bug 134538 has been marked as a duplicate of this bug. ***

Since my proposal bug 134538 has been marked as duplicate. I want to mention the additional things I wrote about the ZWJ and ZWNJ:
Writer should also ignore them in search, prevent multiple of them next to each other, since that is unwanted and offer them in insert>formatting characters.

*** Bug 102348 has been marked as a duplicate of this bug. ***

*** Bug 147727 has been marked as a duplicate of this bug. ***

You can control the visibility of special characters via Tools > Options > Writer > Formatting Aids. I changed the default for Soft Hyphen and Non-breaking Spaces to off in https://gerrit.libreoffice.org/c/core/+/154847

U+200B ZERO WIDTH SPACE and U+2060 WORD JOINER are a different topic. Both should be treated as spaces to become disabled with this option too. Better handle it as an extra patch.

Heiko Tietze committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/56ce1aa9239637a10f747c00ba93e96bb2e613ab

Resolves tdf#58434 - No field shading for soft hyphen and non-breaking spaces

It will be available in 24.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.

Heiko Tietze committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/4d7a98b582dc70bbffc78e6622969e218f108433

Related tdf#58434 - Treat ZWSP and WJ as hard space

It will be available in 24.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.

@Heiko, your patches for 24.2 don't shift the visibility toggle from Fields to NPC, issue is not resolved. For NPC or zerowidth characters the <Ctrl>+F10 toggle rather than the field shading of <Ctrl>+F8 is still needed.

(In reply to V Stuart Foote from comment #41)

Reopening because the issue of summary has not been addressed--to use the NPC toggle <Ctrl>+F10 rather than the <Ctrl>+F8 field shading toggle to expose formatting marks.

The  Tools -> Options -> LO Writer -> Formatting Aids panel has check boxes in its 'Display Formatting' section, but the NPC/ZW/Hyphenation has not been shifted from handling as Field Shading.  Just for some, like the U+00A0 NO-BREAK SPACE, a display checkbox on the Formatting Aids panel has been added, but the grey field shading remains to muddle the layout.

We still need handling distinct from Field Shading for the Unicode glyphs and object marking as noted above for RTL/LTR and POP, and on the chart in attachment 112798 [details]

*** Bug 160195 has been marked as a duplicate of this bug. ***

*** Bug 160548 has been marked as a duplicate of this bug. ***

(In reply to V Stuart Foote from comment #42)
> We still need handling distinct from Field Shading for the Unicode glyphs
> and object marking as noted above...
Should we keep toggling both with ctrl+f8 with an AND combination for control characters (tools > options > writer > formatting aids) or introduce an extra command like shift+ctrl+f8? 

On/off not only toggles the highlighting but also the character itself for soft hyphen (-) and zero-width space (,). How should this be handled?

(In reply to Heiko Tietze from comment #45)
> Should we keep toggling both with ctrl+f8 with an AND combination for
> control characters (tools > options > writer > formatting aids) or introduce
> an extra command like shift+ctrl+f8? 
> 
> On/off not only toggles the highlighting but also the character itself for
> soft hyphen (-) and zero-width space (,). How should this be handled?

No need for additional distinction between NPC and formatting marks, just stop handling elements as fields.

Goal would be a distinction between "field" elements--what is handled via the <Ctrl>+F2 'Fields...' dialog, the <Ctrl>+F9 "Field Names" toggle, and the <Ctrl>+F8 "Field Shadings"--and *all other* "Formatting Marks" to be handled/toggled visible via the <Ctrl>+F10 toggle.

Simplest is removing all non-field elements (as described above) from the <Ctrl>+F8 toggle of applied gray field shading. 

Then consolidating them *all* under the simple <Ctrl>+F10 toggle (with the blue highlighting on document canvas).

The Formatting aids check boxes can be expanded if needed (to be able to suppress some of the NPC marks, and if you have the cycles) but that is secondary to getting all the non-field elements out from the Ctrl+F8 Field shadings toggle.

(In reply to V Stuart Foote from comment #46)
> Simplest is removing all non-field elements (as described above) from the
> <Ctrl>+F8 toggle of applied gray field shading. 
> 
> Then consolidating them *all* under the simple <Ctrl>+F10 toggle (with the
> blue highlighting on document canvas).
Non-breaking and soft hyphen as well as word joiner don't have special characters shown per ctrl+f10. The zero-width space does but hides it with ctrl+f8 on/off. For example, how to distinguish between a ordinary, breaking hyphen and a soft-hyphen if we go this route?

(In reply to Heiko Tietze from comment #47)
> For example, how to distinguish between a ordinary hyphen and a soft-hyphen...
Likely by keeping the grayish background but toggling it with ctrl+f10.

Heiko Tietze committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/b79e0dd7fc8bc620420ed65edbc105eec3648b34

Resolves tdf#58434 - Show formatting marks independently from fields

It will be available in 24.8.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.

The patch makes all control character toggle via the non-printing character option (ctrl+f10). Fields and co remain at ctrl+f8. The patch also fixes the issue that ctrl+f8 switched on settings under formatting aids and in order to make the whole thing appealing it might be necessary to disable paragraph break and space.

The patch does not change the control characters itself, ie. narrow no-break space. This should be done in a follow-up patch by a more experienced developer. I suggest to file a new ticket.

Heiko asked people on the RTL channel to comment on this bug...

so, I'll first say that I'm having trouble following what changes were affected to fix this. I've also seen the table by Simo and was a bit overwhelmed.

A bit of information:

* With the standard Hebrew keyboard layout (SI-1452), you can insert LRM with RightAlt+NumRow9 , and RLM with RightAlt+NumRow0 . I use this occasionally, especially in textual email.

My thoughts/opinions:

* Characters with control semantics are not fields and should not be controlled by the two field options (shading, show name).

* ... at the same time, I _would_ like to be able to toggle both things, for control characters as well, i.e. are they shaded or not, and do I see the intended glyph (or lack thereof), or rather some text indicating which control character this is, e.g. the text "LRM" or "RLM" instead of ‎ and ‏ respectively. So, I would like controls for that.

* I am not sure whether the shading for fields and for control characters should be the same.

* I am not sure whether I want the shading to be off by default, but whether it is or it isn't, I want it to be easy for newbie users, who are encountering documents with such marks for the first time, to understand what they're seeing - which today is not so much the case I believe. Perhaps something like the hover behavior of these marks? Or an information bar when a document is opened which contains any such marks, which notifies the user of this fact, suggests playing the two toggles and/or instructs the user how to opt out of seeing that notification bar next time.