Bug 74284 - [Accessibility] Extraneous WhiteSpace accessibles cause extensive speech clutter
Summary: [Accessibility] Extraneous WhiteSpace accessibles cause extensive speech clutter
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: UI (show other bugs)
Version:
(earliest affected)
4.2.0.4 release
Hardware: Other Windows (All)
: medium normal
Assignee: Not Assigned
URL:
Whiteboard: target:4.3.0 target:4.2.2
Keywords:
Depends on:
Blocks: a11y-Windows
  Show dependency treegraph
 
Reported: 2014-01-31 16:38 UTC by Marco Zehe
Modified: 2014-02-21 10:41 UTC (History)
11 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Marco Zehe 2014-01-31 16:38:52 UTC
STR:
1. With NVDA running, start LibreOffice, either 4.2.0 or 4.3.0 nightly.
2. You'll hear NVDA repeatedly say "White Space", followed by the description of the "Option Panel" spoken initially.

Exploring the object hierarchy, several direct children and grand children of the Option Pane are WhiteSpace accessibles. The information grabbed with NVDA+F1 for one of these looks as follows:

INFO - globalCommands.GlobalCommands.script_navigatorObject_devInfo (17:31:59):
Developer info for navigator object:
name: u''
role: ROLE_WHITESPACE
states: 
isFocusable: False
hasFocus: False
Python object: <NVDAObjects.IAccessible.IAccessible object at 0x05BFF2B0>
Python class mro: (<class 'NVDAObjects.IAccessible.IAccessible'>, <class 'NVDAObjects.window.Window'>, <class 'NVDAObjects.NVDAObject'>, <class 'baseObject.ScriptableObject'>, <class 'baseObject.AutoPropertyObject'>, <type 'object'>)
description: u'Click an icon to open a new document or a file dialog.'
location: (129, 895, 164, 42)
value: None
appModule: <'soffice' (appName u'soffice', process ID 4648) at address 5bff490>
appModule.productName: u'LibreOffice'
appModule.productVersion: u'4.2.0.4'
TextInfo: <class 'NVDAObjects.NVDAObjectTextInfo'>
windowHandle: 2098126L
windowClassName: u'SALFRAME'
windowControlID: 0
windowStyle: 349110272
windowThreadID: 2112
windowText: u'LibreOffice'
displayText: u''
IAccessibleObject: <POINTER(IAccessible2) ptr=0x594124 at 5c12670>
IAccessibleChildID: 0
IAccessible event parameters: windowHandle=2098126L, objectID=-4, childID=-77
IAccessible accName: u''
IAccessible accRole: ROLE_SYSTEM_WHITESPACE
IAccessible accState:  (0)
IAccessible accDescription: u'Click an icon to open a new document or a file dialog.'
IAccessible accValue: None
IAccessible2 windowHandle: 2098126
IAccessible2 uniqueID: -77
IAccessible2 role: ROLE_SYSTEM_WHITESPACE
IAccessible2 states:  (0)
IAccessible2 attributes: exception: (-2147467259, 'Unspecified error', (None, None, None, 0, None))
Comment 1 V Stuart Foote 2014-01-31 19:19:10 UTC
@Marco, thanks for posting!

I believe this nuisance issue results from the implementation of nested GtkBox or GtkGrid UI Widgets for layout of the StartCenter. Also noted in the Sidebar and it probably exists elsewhere where UI has been implemented.

The JAB Java runtime based bridge announces the structures as 'filler', while IAccessible2 announces as 'white space'.

Not sure if they are handled similarly in AX or ATK.

Not sure if these GTK+ layout elements can be suppressed, otherwise maybe a better accessible role can be assigned.

Is this something that could (or should?) be suppressed in the Assistive Technology?

Ideas?
Comment 2 Marco Zehe 2014-01-31 19:37:55 UTC
(In reply to comment #1)
> I believe this nuisance issue results from the implementation of nested
> GtkBox or GtkGrid UI Widgets for layout of the StartCenter. Also noted in
> the Sidebar and it probably exists elsewhere where UI has been implemented.

Interesting, since such purely presentational items should ideally not clutter the accessible tree at all. If they do not serve any really useful purpose, would it be an option to just not expose them, but make the buttons direct IAccessible children of the panel?
Comment 3 Niklas Johansson 2014-01-31 19:44:32 UTC
LibreOffice API specifies the Accessible role filler as:

Filler role.
An object that fills up space in a user interface. It is often used in interfaces to tweak the spacing between components, but serves no other purpose.

So to me it seems that it should not be announced by screen readers. To my knowledge neither AX nor ATK announce objects with the accessible role filler.
Comment 4 Michael Meeks 2014-02-01 17:32:16 UTC
Intersting - there should be no conceptual problem with the hierarchical helpers - surely; we use them for ATK without issues with orcus, but presumably getting the type right is important. Caolan was just doing an a11y talk so might have an idea =)
Comment 5 James Teh 2014-02-03 02:19:50 UTC
We probably should make NVDA ignore whitespace objects. That said:
* Whitespace does seem like an inappropriate role here, as does filler. Whitespace/filler suggest that they fill up space *between* elements, not that they *contain* elements. I'd think a role of pane or similar would be better here.
* Even if ATs do ignore them, exposing them if they really serve no purpose at all is not ideal in terms of performance, as ATs will hit each one if they have to walk up the hierarchy.
Comment 6 James Teh 2014-02-03 02:26:15 UTC
NVDA ticket: http://community.nvda-project.org/ticket/3839
Comment 7 Marco Zehe 2014-02-03 05:16:46 UTC
I agree with jamie that these layout-containers should not be exposed. In Mozilla, we use the Whitespace role for <br /> elements. And only for those, see https://developer.mozilla.org/en-US/docs/Web/Accessibility/AT-APIs/Gecko/Roles/ROLE_WHITESPACE#content
Comment 8 V Stuart Foote 2014-02-03 06:19:59 UTC
Issue seems to arise from the way the GTK+ Glade UI widgets get assigned their accessible roles--I think with GAIL but I'm not sure of the plumbing to associate to UAA and from there out to JAB (for 4.2 and earlier) and IA2, AX and ATK. The nested boxes or grids are used only for layout, but as GTK+ objects they get parsed and assigned an accessibility role. 

I don't know if that can be adjusted to suppress when used solely for layout. But I beleive Caolán, Kendy and Cédric have the best handle on how the accessible roles for the GTK+ UI widgets are cobbled together and perhaps can comment on a solution.

Meanwhile, I find I need a score card to keep track of the API's--the W3C HTML to Platform Accessibility APIs Implementation Guide fits the bill nicely.

http://rawgithub.com/w3c/html-api-map/master/index.html

Worth keeping handy...
Comment 9 Niklas Johansson 2014-02-03 08:51:58 UTC
Glade does use the role FILLER to layout objects, not only to add space in between objects. And so I believe that the role filler is correct. It seems that Mozilla and W3C maps filler to ROLE_SYSTEM_WHITESPACE so we should probably do the same.

If I understand the code correctly on Mac LibreOffice ignore the role FILLER in the accessibility hierarchy, it is also mapped as AXUnknown
http://opengrok.libreoffice.org/xref/core/vcl/osx/a11ywrapper.mm#712

Would it be possible/make sense to do something similar on Windows?
Comment 10 Caolán McNamara 2014-02-11 21:26:31 UTC
What happens when NDVA meets a "true" gtk application on windows ? e.g something like gimp (or maybe better, assuming it works, gladewin32 itself which presumably contains glade-previewer which might be suitable for testing simple standalone box/grid using testcases.
Comment 11 V Stuart Foote 2014-02-11 22:15:34 UTC
Caolán

(In reply to comment #10)
> What happens when NDVA meets a "true" gtk application on windows ? e.g
> something like gimp (or maybe better, assuming it works, gladewin32 itself
> which presumably contains glade-previewer which might be suitable for
> testing simple standalone box/grid using testcases.

Nothing =(  It seems that the GIMP on Windows offers no Java Accessiblity API access via the JAB, nor UIA nor IAccessible2 interface--just minimal MSAA responses. So seems to be mute with NVDA.

Anyone know if there is a version of GIMP for Windows built with support for Assistive Technology tools?
Comment 12 James Teh 2014-02-11 22:38:11 UTC
(In reply to comment #11)
> Anyone know if there is a version of GIMP for Windows built with support for
> Assistive Technology tools?
As far as I know, GTK isn't accessible on Windows at all (unless of course the app has its own complete accessibility implementation).
Comment 13 Michael Meeks 2014-02-12 11:30:43 UTC
It may well be worth looking at and/or screenshotting some Java apps under Windows to see what 'pane' type peers they use to represent their layout containers [ assuming that they do ] - there are plenty of GridBagLayoutOfDoom type things in Java =) I assume their MSAA bridge has some good tips there.

Input appreciated  ...
Comment 14 James Teh 2014-02-12 23:12:30 UTC
(In reply to comment #13)
> It may well be worth looking at and/or screenshotting some Java apps under
> Windows to see what 'pane' type peers they use to represent their layout
> containers [ assuming that they do ] - there are plenty of
> GridBagLayoutOfDoom type things in Java =) I assume their MSAA bridge has
> some good tips there.
Java doesn't expose anything via MSAA. The Java Access Bridge exposes a completely separate API in Windows which is entirely unrelated to MSAA, so it has its own roles, states, etc.
Comment 15 Michael Meeks 2014-02-13 11:09:30 UTC
Mozilla has some great API documentation here:

http://lxr.mozilla.org/seamonkey/source/accessible/accessible-docs.html

Ominously their drawing of what gets a peer seems to show them exporting a different tree to their internal representation - which (I guess) militates against exposing the layout peers: even though they may well provide useful semantic information to ATs.

I guess I didn't read what we're doing there yet; does the new VCL layout stuff have windows as real children of other windows ? or is the layout hierarchy tacked on the side ? ...
Comment 16 Marco Zehe 2014-02-13 18:13:21 UTC
(In reply to comment #15)
> Mozilla has some great API documentation here:
> 
> http://lxr.mozilla.org/seamonkey/source/accessible/accessible-docs.html
> 
> Ominously their drawing of what gets a peer seems to show them exporting a
> different tree to their internal representation - which (I guess) militates
> against exposing the layout peers: even though they may well provide useful
> semantic information to ATs.

Mozilla has only only one "real" window that gets exposed to assistive technologies, meaning one that has a window handle, control ID and Window Class. All the rest that is exposed is a hierarchy built from the XUL of the Chrome, and the HTML of the web content, if any. So the parent/child relationships are based on the DOM trees of both the browser chrome and content. The accessibility hierarchy is a subset of the DOM hierarchy. In XUL, vbox, hbox and other layout elements are left out, for example. They are for layout purposes only, and the important thing for ATs to know is the location and width and height of actual elements like labels, controls etc. How they got there, and how many boxes are around them, is nothing assistive tecnologies care about too much. :)

> I guess I didn't read what we're doing there yet; does the new VCL layout
> stuff have windows as real children of other windows ? or is the layout
> hierarchy tacked on the side ? ...

It looks very similar to what Mozilla is doing, exposing an accessibility element hierarchy to assistive technologies that resembles the hierarchy of VCL and document elements. And in here, these filler boxes that are used for layout are exposed as whitespace. In Mozilla, the whitespace accessibles are exclusively created for HTML:br elements to indicate to screen readers that particular text doesn't run together.

Hope this helped!
Comment 17 Caolán McNamara 2014-02-17 17:03:53 UTC
hmm, so I assigned "VclBox" containers AccessibleRole::FILLER because that's what gtk (IIRC) assigned "GtkBox" (which I did find a bit odd). While "VclGrids" are assigned AccessibleRole::PANEL, again because that's what (IIRC) gtk did for their GtkGrids. The containers are "real windows" as opposed to sticking something on the side. 

I'm guessing that there isn't a problem with "VclGrids", so lets first experiment with assigning PANEL "A generic container that is often used to group objects" for VclBoxes under windows rather than eliding them from the a11y view, which might have somewhat complicated side-effects.
Comment 18 Commit Notification 2014-02-17 17:05:43 UTC
Caolan McNamara committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=685ec1899435037205d98a102a32ca8b6a4836d0

Related: fdo#74284 try AccessibleRole::PANEL for VclBoxes under windows



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 19 James Teh 2014-02-17 23:16:12 UTC
(In reply to comment #5)
> We probably should make NVDA ignore whitespace objects.
FWIW, I've already committed a change to NVDA to ignore whitespace accessibles as layout objects. It's currently incubating on our "next" branch. Barring any issues, it'll get merged to master in a couple of weeks. That said, I still think there's a more appropriate role.

(In reply to comment #18)
> Related: fdo#74284 try AccessibleRole::PANEL for VclBoxes under windows
What does that role get mapped to in MSAA/IA2? (There's no panel role in either, but there are several pane roles.)
Comment 20 Marco Zehe 2014-02-18 09:02:24 UTC
I can confirm that this build:

Version: 4.3.0.0.alpha0+
Build ID: d20ffc370bc1b946366d4cb1073b815beda706a2
TinderBox: Win-x86@39, Branch:master, Time: 2014-02-18_03:05:33

eliminates the extraneous whitespace accessibles.
Comment 21 Caolán McNamara 2014-02-18 11:48:40 UTC
re "What does that role get mapped to in MSAA/IA2? (There's no panel role in either, but there are several pane roles.)" digging into winaccessibility/source/service/AccObject.cxx I get "IA2_ROLE_OPTION_PANE" (Under gtk's atk it maps to ATK_ROLE_PANEL).
Comment 22 Michael Meeks 2014-02-18 11:55:10 UTC
Marco, James - thanks for the great support in NVDA ! that's great; if you're happy with the IA2_ROLE_OPTION_PANE change would it be good to back-port that to 4.2.<next> ?
Comment 23 Caolán McNamara 2014-02-18 12:00:08 UTC
Given Macro says that "I can confirm that ... d20ffc370bc1b946366d4cb1073b815beda706a2 ... eliminates the extraneous whitespace accessibles." and is the original reporter I'll call this fixed.

I'll go ahead and submit the tweak for 4-2
Comment 24 Marco Zehe 2014-02-18 12:03:08 UTC
(In reply to comment #21)
> re "What does that role get mapped to in MSAA/IA2? (There's no panel role in
> either, but there are several pane roles.)" digging into
> winaccessibility/source/service/AccObject.cxx I get "IA2_ROLE_OPTION_PANE"
> (Under gtk's atk it maps to ATK_ROLE_PANEL).

That's interesting, because in the 4.3-dev build, I now don't see any option panes on that start screen any more. However, since bug 75151 might also play a part here, they may simply not be exposed at all any more since there is neither a name nor an acc description to expose. I am not using a build of NVDA that has the ignoring of whitespace accessibles in yet, so this is definitely something different.
Comment 25 Marco Zehe 2014-02-18 12:04:08 UTC
(In reply to comment #22)
> Marco, James - thanks for the great support in NVDA ! that's great; if
> you're happy with the IA2_ROLE_OPTION_PANE change would it be good to
> back-port that to 4.2.<next> ?

Yup! :)
Comment 26 V Stuart Foote 2014-02-18 15:21:28 UTC
(In reply to comment #24)

> That's interesting, because in the 4.3-dev build, I now don't see any option
> panes on that start screen any more. However, since bug 75151 might also
> play a part here, they may simply not be exposed at all any more since there
> is neither a name nor an acc description to expose. I am not using a build
> of NVDA that has the ignoring of whitespace accessibles in yet, so this is
> definitely something different.

With a build of master -- prior to Caolán's 2014-02-17 patch--
Version: 4.3.0.0.alpha0+
Build ID: 6364a3ec6f92fc1f4ac3c7a4e2bdce2a2140c736
TinderBox: Win-x86@39, Branch:master, Time: 2014-02-13_01:44:32

With NVDA 2013.2 active I was getting the multiple 'White Space' filler when passing between the GTK boxes.  And as noted on bug 75151, as built the master builds don't receive the more annoying description translations that really stick out.

With today's patched build the filler role has been suppressed. 
Version: 4.3.0.0.alpha0+
Build ID: d20ffc370bc1b946366d4cb1073b815beda706a2
TinderBox: Win-x86@39, Branch:master, Time: 2014-02-18_03:05:33
Comment 27 Marco Zehe 2014-02-18 15:23:37 UTC
Thanks Stuart, your explanation from bug 75151 makes perfect sense, and NVDA may just suppress all of them now. I just took a look with accProbe, and am seeing them there, with no name and no description. All good! :)
Comment 28 Commit Notification 2014-02-18 15:50:15 UTC
Caolan McNamara committed a patch related to this issue.
It has been pushed to "libreoffice-4-2":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=a9a2b6d63231b514754f16c029ca35bb7383e139&h=libreoffice-4-2

Resolves: fdo#74284 use AccessibleRole::PANEL for VclBoxes under windows


It will be available in LibreOffice 4.2.3.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 29 James Teh 2014-02-19 01:10:11 UTC
(In reply to comment #24)
> That's interesting, because in the 4.3-dev build, I now don't see any option
> panes on that start screen any more. However, since bug 75151 might also
> play a part here, they may simply not be exposed at all any more since there
> is neither a name nor an acc description to expose.
If neither name nor description is exposed on an option pane, NVDA will treat it as a layout object and thus ignore it. You'll be able to see it if you disable simple review and use object navigation. It's important to note that if these VclBoxes ever get names or descriptions, NVDA will stop ignoring them.
Comment 30 Commit Notification 2014-02-21 00:57:57 UTC
Caolan McNamara committed a patch related to this issue.
It has been pushed to "libreoffice-4-2-1":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=47ff82608b065c3c0d98557583bb822b20fb4081&h=libreoffice-4-2-1

Resolves: fdo#74284 use AccessibleRole::PANEL for VclBoxes under windows


It will be available already in LibreOffice 4.2.1.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 31 Commit Notification 2014-02-21 09:00:38 UTC
Caolan McNamara committed a patch related to this issue.
It has been pushed to "libreoffice-4-2-2":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=5754e66f54fbf0a3e2eb96fa7afdf963a230d670&h=libreoffice-4-2-2

Resolves: fdo#74284 use AccessibleRole::PANEL for VclBoxes under windows


It will be available already in LibreOffice 4.2.2.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 32 Michael Stahl (allotropia) 2014-02-21 10:41:45 UTC
correcting target, the script is confused...