Bug 140659 - Formula editor not operable with screenreaders NVDA or Orca
Summary: Formula editor not operable with screenreaders NVDA or Orca
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Formula Editor (show other bugs)
Version:
(earliest affected)
7.1.0.3 release
Hardware: All All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard: target:7.2.0 target:7.1.4
Keywords: accessibility
Depends on:
Blocks: Formula-Editor a11y Elements-Pane
  Show dependency treegraph
 
Reported: 2021-02-25 09:48 UTC by juergenkohler23
Modified: 2021-09-07 07:26 UTC (History)
5 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description juergenkohler23 2021-02-25 09:48:57 UTC
Description:
- open LibreOffice
- Insert -> Object -> Formula
- In the window that then opens, you are "trapped": nothing is read out, nor can you navigate through it with the tab key

Steps to Reproduce:
1. open LibreOffice
2. Insert -> Object -> Formula

Actual Results:
In the window that then opens, you are "trapped": nothing is read out, nor can you navigate through it with the tab key.

Expected Results:
The formula editor should be navigable with the Tab key. All buttons should be read aloud.


Reproducible: Always


User Profile Reset: No



Additional Info:
Tested with

NVDA 2020.4

Windows
Edition	Windows 		10 Home
Version				1909
Installed on			17.02.2020
Operating system build		18363.1016

LibreOffice
Version: 	7.1.0.3 (x64)
Build ID: 	8349ace3c3162073abd90d81fd06dcfb6b36b994
CPU threads: 	4; OS: Windows 10.0 Build 18363; UI render: Skia/Vulkan; VCL: win
Locale: 		de-DE (de_DE); UI: de-DE
Calc: 		threaded
Comment 1 Peter Franke 2021-02-25 10:21:20 UTC
I also have this bug every time; the formula editor is virtually unusable for screen reader users.
It would be very important that this bug is fixed, because LibreOffice is also used more often in schools at the moment (mathematics lessons or the like).
Comment 2 V Stuart Foote 2021-02-25 16:31:34 UTC
Some of this is a documentation/user-training issue.

Normal LibreOffice F10 & F6 keyboard movements in Forumula Editor seem correct--but there is one F6 stop that sounds nothing.

But, the <Tab> & Cursor <U,D,L,R>  keyboard navigation in the elements panel is inconsistent. 

Focus/accessible events in the Elements panes seem to get out of sync--the accessible event/focus is off by one from the cursor movement selected button. And of course, no meaningful event sounding for the example formulas.

Likewise no meaningful review of the rendered formula, just the StarMath strings of multi-line editable paragraph held in the Command panel.
Comment 3 dante19031999 2021-02-25 18:19:43 UTC
Tried it out in Fedora (linux) with Orca but failed.

The only thing strange found is when you click into something into the element docking window and click's on something says something like "command standle". But it's not the bug described here or not understanding well.

> Focus/accessible events in the Elements panes seem to get out of sync--the accessible event/focus is off by one from the cursor movement selected button. 
Maybe is what I found?

That's why I'm changing it to windows OS only. If I'm wrong change it back.
Would also appreciate a Windows user to check if all OLE objects are affected.
That information could be useful for the fixer if the answer is yes.

> Some of this is a documentation/user-training issue.
Except if it is this case.
Comment 4 V Stuart Foote 2021-02-25 19:31:21 UTC
(In reply to dante19031999 from comment #3)
> Tried it out in Fedora (linux) with Orca but failed.
> 
> The only thing strange found is when you click into something into the
> element docking window and click's on something says something like "command
> standle". But it's not the bug described here or not understanding well.
> 

Dante, when working through accessibility issues it is helpful to frame their behavior as keyboard only navigation. Don't use the mouse. Meaning F10, F6 and some mix of TAB and Cursor.

Done that way it exposes weaknesses in the underlaying UI that mouse pointer/click navigation obscures.

For accessibility, the focus action--by keyboard movement onto a control, or mouse (down or up)--triggers an "accessible event" whose labeling the Assistive Technology is able to parse.

Jan-Marek provided [1] many of the accessible events (as focusable buttons) for the elements panel for see also bug 65587, but for some reason, on Windows at least, the "accessible event" exposed is not matched to the control with UI focus. Off by one so seems to be a logic error of some ilk.

I checked a 7.0.3 build on Fedora 33, none of the buttons in the Element panes sounded an event in Orca, so seems bigger issues there compared to Windows builds.

=-ref-=
[1] https://gerrit.libreoffice.org/73077
Comment 5 V Stuart Foote 2021-02-25 20:09:36 UTC
(In reply to V Stuart Foote from comment #4)
> F10, F6 and some mix of TAB and Cursor.
> 
> Done that way it exposes weaknesses in the underlaying UI that mouse
> pointer/click navigation obscures.
> 

With Orca (Fedora 33 and LO 7.0.4) the Elements pane listbox can not be reached by F6 keyboard navigation, it seems to cycle but does not get focus. 

In addition to none of the Elements control buttons sounding after entering the Elements pane. But, when I was able to get the buttons to start sounding in Orca, but like in Windows the accessible event is off by one from the element button with focus.

@Caolán, you might be interested about the GTK3 issue here.

Version: 7.0.4.2
Build ID: 00(Build:2)
CPU threads: 1; OS: Linux 5.10; UI render: default; VCL: gtk3
Locale: en-US (en_US.UTF-8); UI: en-US
Calc: threaded
Comment 6 Commit Notification 2021-04-21 18:44:43 UTC
Caolán McNamara committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/12ea51ba8bfbc1d22feb2e4d498360eec015859e

Related: tdf#140659 off by one indexes because scrollbar is now outside

It will be available in 7.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 7 Commit Notification 2021-04-21 18:45:55 UTC
Caolán McNamara committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/30b36f290aa0b59fd7d2bd41c8bdfca304d859af

Related: tdf#140659 on control get/lose focus call update custom a11y

It will be available in 7.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 8 Commit Notification 2021-04-22 08:28:42 UTC
Caolán McNamara committed a patch related to this issue.
It has been pushed to "libreoffice-7-1":

https://git.libreoffice.org/core/commit/b45629418d91c7452245ae4c6894fa3c1becf237

Related: tdf#140659 off by one indexes because scrollbar is now outside

It will be available in 7.1.4.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 9 Commit Notification 2021-04-22 08:28:53 UTC
Caolán McNamara committed a patch related to this issue.
It has been pushed to "libreoffice-7-1":

https://git.libreoffice.org/core/commit/6b4d475b9377dba504a08c4317210bf3f8bc1859

Related: tdf#140659 on control get/lose focus call update custom a11y

It will be available in 7.1.4.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 10 Caolán McNamara 2021-04-23 10:11:35 UTC
The above commits should solve the out-by-one in the elements panel and additionally announce focus gained on entering by tab into the elements panel
Comment 11 Commit Notification 2021-04-23 12:28:41 UTC
Caolán McNamara committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/55e2c6c648ed94664903c0c6f53922d03e76b7c1

Related: tdf#140659 designate which widget to get default focus

It will be available in 7.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 12 Commit Notification 2021-04-23 14:37:00 UTC
Caolán McNamara committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/57138e6cfa1dec2c44d7cca55171dca40d45aaa3

Related: tdf#140659 improve F6 task pane switching

It will be available in 7.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 13 Caolán McNamara 2021-04-23 15:41:33 UTC
The second set of commits above should solve the gtk-specific problem in comment #5.

Not sure if any of that helps with the initial comment #1 though where focus is set in the command editor and that starts off blank (where orca reads "paragraph zero" for it) and pressing tab enters a tab into the command editor. If I type into the area and press kp_enter it does read out its contents correctly.

FWIW codewise the a11y code for this widget in 7.2 is svx/source/dialog/weldeditview.cxx where f.e. getAccessibleRole is AccessibleRole::TEXT_FRAME (which for gtk is mapped to ATK_ROLE_PANEL in vcl/unx/gtk3/a11y/gtk3atkwrapper.cxx)
Comment 14 juergenkohler23 2021-08-23 07:38:05 UTC
Switching between the input line and the operator commands is not possible with the keyboard.
In the operator list, I cannot select mathematical symbols with the arrow keys.
If I want to select a symbol in the list (for example by pressing the Enter key), my cursor still remains in the command field and the targeted symbol is not selected.


Windows
Edition Windows 10 Home
Version	20H2
Installed on ‎05.‎02.‎2021
Operating system build	19042.985

LibreOffice
Version: 7.2.0.4 (x64) / LibreOffice Community
Build ID: 9a9c6381e3f7a62afc1329bd359cc48accb6435b
CPU threads: 4; OS: Windows 10.0 Build 19043; UI render: Skia/Raster; VCL: win
Locale: de-DE (de_DE); UI: de-DE
Calc: threaded

NVDA
2021.1
Comment 15 Caolán McNamara 2021-08-24 09:08:35 UTC
"Switching between the input line and the operator commands is not possible with the keyboard."

Here, with focus by default starting in the input line, then for me Shift+F6 will toggle focus to the operator panel and F6 back to the input line.

"In the operator list, I cannot select mathematical symbols with the arrow keys.
If I want to select a symbol in the list (for example by pressing the Enter key), my cursor still remains in the command field and the targeted symbol is not selected."

What I experience is that after Shift+F6 to move focus into the panel, focus starts on the combobox, tab brings focus into the list of symbols, at which point the arrow keys move the selection around the list of symbols and return inserts that symbol and moves focus back to the input line.
Comment 16 V Stuart Foote 2021-08-24 12:46:39 UTC
(In reply to Caolán McNamara from comment #15)

The same F10, F6, <Tab> and <Cursor> keyboard navigation, so as noted comment 2.

However on Win10 with NVDA 2020.4 enabled mouseOver events do fire for mouse pointer movements, and the Element pane combobox droplist will sound category selection--but on Tab into the pane a11y no longer sounding accessible events for cursor L,R,U,D movmements between entries on the Elements pane. 

The descriptions are there--they sound on mouse movements and tooltips are exposed--but elements no longer respond with keyboard focus.

Same result with recent builds of trunk and

Version: 7.2.0.4 (x64) / LibreOffice Community
Build ID: 9a9c6381e3f7a62afc1329bd359cc48accb6435b
CPU threads: 8; OS: Windows 10.0 Build 19043; UI render: Skia/Vulkan; VCL: win
Locale: en-US (en_US); UI: en-US
Calc: threaded
Comment 17 Jan-Marek Glogowski 2021-09-01 06:56:11 UTC
Stuart pinged me on my original patch for bug 65587 https://gerrit.libreoffice.org/c/core/+/73077/, but I think it's better to leave a comment here.

FYI: I did the implementation in bug 65587 on Linux and never did any test on Windows. I had to fix various bugs in accerciser back then and just tested via this tool (and was able to crash LO with it...). I just remember it took me a long time to get it working in any way.

I literally "made something up", because I didn't now how to represent a combo box, which changes the result of a list view (the grid is internally just a list, and that's how the navigation works) + some optional scrollbar.

Reading the comments, Caolan already fixed the off-by-one bugs, because apparently the custom widget layout has changed w.r.t. the scrollbar. Otherwise I have the same experience as Caolan describes in comment 15 on Linux with kf5.
Comment 18 juergenkohler23 2021-09-07 07:26:31 UTC
(In reply to Caolán McNamara from comment #15)
> "Switching between the input line and the operator commands is not possible
> with the keyboard."
> 
> Here, with focus by default starting in the input line, then for me Shift+F6
> will toggle focus to the operator panel and F6 back to the input line.
> 
> "In the operator list, I cannot select mathematical symbols with the arrow
> keys.
> If I want to select a symbol in the list (for example by pressing the Enter
> key), my cursor still remains in the command field and the targeted symbol
> is not selected."
> 
> What I experience is that after Shift+F6 to move focus into the panel, focus
> starts on the combobox, tab brings focus into the list of symbols, at which
> point the arrow keys move the selection around the list of symbols and
> return inserts that symbol and moves focus back to the input line.



Thanks Caolán for your comment! The reference to Shift+F6 was helpful; I had not tested this key combination before.

As a sighted person, using the key combinations you mentioned, the formula editor seems to be operable. If I turn on NVDA to do this, it looks different, unfortunately: NVDA reads out the subcategories in the list (e.g. "unary/binary operators", "functions" etc.), but as soon as I move to the selection of relations, functions etc. using the shift key, NVDA remains mute.
Whether this is due to NVDA or to the formula editor, I can't say...

Many greetings