Created attachment 116166 [details]
one text to confirm the problem.

Setting new, but believe this remains an issue with NVDAs support for parsing IAccessible2 based content.

This has an open issue on NVDA support: http://community.nvda-project.org/ticket/4119 or with Symphony's early IA2 based support NVDA in ticket 910. But issues with SayAll with ODF and OpenOffice existed back to the JavaAccessBridge/JRE Java Accessibility era--see NVDA tickets 422, 96

@Jamie, any thoughts on implementing SayAll for LibreOffice/Apache OpenOffice or, should I just close this not our bug?

Stuart

Say all for Symphony (NVDA ticket #910) was fixed years ago. Any remaining issues were due to bugs in Symphony.

The problem with say all in OO/LO is due to the fact that they only expose text that is currently on screen. As a result, once NVDA say all gets to the last paragraph, it asks for the next paragraph and is told that there isn't one, so say all can't go any further. There are hacks to get around this (e.g. send control+downArrow or scroll in some other way when LO reports no more paragraphs), but this is fairly inelegant and potentially unreliable. Whether this is a bug is debatable, but it's certainly a problem for us and is not what happens in other implementations. In Symphony, you could get at paragraphs that weren't yet on screen (barring weird bugs), which is why say all worked in Symphony.

Note that you don't have to generate accessibles for every paragraph in the document ahead of time. You could just generate them on demand when the flowsTo relation is required.

@Jamie, *,

This gets beyond my skill set pretty quickly...

But, looking at i27138 - we already have an XAccessibleContext providing flowsFrom and flowsTo in the internal UAA. And that should be providing the IAccessible2 relationships.

But, seems that  for our Windows implementation--UAccCOM--we may be defining just the parent and child relationships, and not actually defining IA2_RELATION_FLOWS_FROM and IA2_RELATION_FLOWS_TO (see refs).

Anyone have a better feel for the this?

@Jamie,  if the IA2_RELATION_FLOWS_FROM and IA2_RELATION_FLOWS_TO are fully instrumented, would that then facilitate the NVDA SayAll function?

=-refs-=
Core
http://opengrok.libreoffice.org/xref/core/accessibility/source/extended/textwindowaccessibility.cxx#1447
http://opengrok.libreoffice.org/xref/core/sw/source/core/access/accpara.hxx#265
http://opengrok.libreoffice.org/xref/core/sw/source/core/access/accpara.cxx#959
http://opengrok.libreoffice.org/xref/core/editeng/source/accessibility/AccessibleEditableTextPara.cxx#813

Windows IA2 UAccCOM
http://opengrok.libreoffice.org/xref/core/winaccessibility/source/UAccCOMIDL/UAccCOM.idl
http://opengrok.libreoffice.org/xref/core/winaccessibility/source/UAccCOM/MAccessible.cxx
http://opengrok.libreoffice.org/xref/core/winaccessibility/source/UAccCOMIDL/ia2_api_all.idl#354

LO *does* expose flowsFrom and flowsTo via IA2. The issue isn't that they aren't exposed. The issue is that they don't expose paragraphs that are currently off-screen in some cases.

So, steps to reproduce with the attached example document:
1. Open the document. The cursor will be at the first line.
2. Get the accessible for the first paragraph.
3. Call IAccessible::nRelations.
Result (correct): 1
4. Call IAccessible2::relations, passing 1 for maxRelations.
5. Get the first (and only) IAccessibleRelation returned.
6. On the relation from step 5, call IAccessibleRelation::relationType.
Result (correct): "flowsTo"
7. On the relation from step 5, call IAccessibleRelation::nTargets.
Result (correct): 1
8. On the relation from step 5, call IAccessibleRelation::target, passing 0 for targetIndex.
Expected: An IUnknown for the second paragraph.
Actual: E_FAIL
9. In the document, press control+downArrow to move to the second paragraph.
10. On the relation from step 5, call IAccessibleRelation::target, passing 0 for targetIndex.
Result (correct): An IUnknown for the second paragraph.

Interestingly, if you then press control+home to move to the top of the document and repeat step 10, you still get the second paragraph. So, it seems that if you've retrieved the paragraph once already, you can retrieve it again even if it goes off-screen.

This issue also impacts things like Orca's ability to do structural navigation (navigate amongst headings, for instance).

As for the flows-to relations, things get more interesting depending on what you are trying to accomplish. Consider something like:

Page 1 (which is visible)
* Paragraph 1
* Paragraph 2
Page 2 (not currently visible)
* Paragraph 3
* Paragraph 4

At least on my platform, if you get the last child of the accessible document, you get "Paragraph 2". If you then get the flows-to target of that child you get "Paragraph 3". So far so good. But if you get the parent of "Paragraph 3" you get the accessible document which contains "Paragraph 1" and "Paragraph 2". In other words, the parent of "Paragraph 3" does not have "Paragraph 3" as a child. That's a broken tree.

I understand the desire to not automatically expose the entire tree for all pages at once because that potentially would be a huge and non-performant solution. But in addition to a non-broken tree, I'd really like to have structural navigation for document content in Orca. And not everything has an immediate flows-to (tables, form fields). So what about something like:

For each page in the document, create an accessible (ATK_ROLE_PAGE on my platform; dunno about IA2). Regardless of the role, these page accessibles are all immediate children of the document and are all created upon document load. Also load the accessible children for a given page when that page is accessed. Lazy load the rest of the children of each page.

(In reply to Joanmarie Diggs from comment #6)

> I understand the desire to not automatically expose the entire tree for all
> pages at once because that potentially would be a huge and non-performant
> solution. But in addition to a non-broken tree, I'd really like to have
> structural navigation for document content in Orca. And not everything has
> an immediate flows-to (tables, form fields). So what about something like:
> 
> For each page in the document, create an accessible (ATK_ROLE_PAGE on my
> platform; dunno about IA2). Regardless of the role, these page accessibles
> are all immediate children of the document and are all created upon document
> load. Also load the accessible children for a given page when that page is
> accessed. Lazy load the rest of the children of each page.

IAccessible2 has the same... IA2_ROLE_PAGE, but it looks like NSAccessibility does not currently map it in any useful fashion, refs.

=-refs-=
IA2_ROLE_PAGE
http://opengrok.libreoffice.org/xref/core/winaccessibility/source/UAccCOMIDL/ia2_api_all.idl#901
http://opengrok.libreoffice.org/xref/core/winaccessibility/source/service/AccObject.cxx

NSAccessibility
http://opengrok.libreoffice.org/xref/core/vcl/osx/a11yrolehelper.mm#119
https://developer.apple.com/library/mac/documentation/AppKit/Reference/NSAccessibility_Protocol_Reference/index.html#//apple_ref/doc/constant_group/Roles

The accessible role PAGE seems to exist in LibreOffice internal FW [1], but I've done some grepping and I think it's never used.

[1] http://api.libreoffice.org/docs/idl/ref/namespacecom_1_1sun_1_1star_1_1accessibility_1_1AccessibleRole.html#a29bb572b50678c1b43fb73f8e0855d65

(In reply to Joanmarie Diggs from comment #6)
> 
> For each page in the document, create an accessible (ATK_ROLE_PAGE on my
> platform; dunno about IA2). Regardless of the role, these page accessibles
> are all immediate children of the document and are all created upon document
> load. Also load the accessible children for a given page when that page is
> accessed. Lazy load the rest of the children of each page.

Unfortunately, this approach seems to be difficult to implement. There is no concept of "pages" in the document model; pages are an artifact of the view that wraps the contents. There are contents that can be split between two pages, so we would still have to do some hacks to model the contents in a proper tree. Finally, the flow of paragraphs to/from the next/previous page when editions happen would have to trigger a rebuild of parts of the accessibility tree, this should be doable but it's definitely inconvenient and would probably generate a lot of event noise.


> Page 1 (which is visible)
> * Paragraph 1
> * Paragraph 2
> Page 2 (not currently visible)
> * Paragraph 3
> * Paragraph 4
> 
> At least on my platform, if you get the last child of the accessible
> document, you get "Paragraph 2". If you then get the flows-to target of that
> child you get "Paragraph 3". So far so good. But if you get the parent of
> "Paragraph 3" you get the accessible document which contains "Paragraph 1"
> and "Paragraph 2". In other words, the parent of "Paragraph 3" does not have
> "Paragraph 3" as a child. That's a broken tree.

When the paragraph 3 is got through the flows-to relation, is the parent component (the document) refreshed? Would that work to implement "say all"?

> This issue also impacts things like Orca's ability to do structural
> navigation (navigate amongst headings, for instance).

Regarding this topic, would it help if we implemented flows to/from relations among header objects?

This is still an issue, no change supporting structural navigation by page top to bottom with current release or on master. NVDA screen reader exhausts buffer and stalls.

Here are links to some related historical reading from the Gnome ATK/AT-SPI side on providing support for the Accessibility::Collection IDL (AtkDocument/AtkCollection) as a possible solution to our limited window into a complex document that disrupts screen reading.

https://bugzilla.gnome.org/show_bug.cgi?id=652548
https://bugzilla.gnome.org/show_bug.cgi?id=345750

(In reply to QA Administrators from comment #14)
This is still an issue, no change supporting structural navigation by page top to bottom with current release or on master. NVDA screen reader exhausts buffer and stalls.

(In reply to QA Administrators from comment #16)

This is still an issue, no change supporting structural navigation by page top to bottom with current release or on master. NVDA screen reader exhausts buffer and stalls.

Here are links to some related historical reading from the Gnome ATK/AT-SPI side on providing support for the Accessibility::Collection IDL (AtkDocument/AtkCollection) as a possible solution to our limited window into a complex document that disrupts screen reading.

https://bugzilla.gnome.org/show_bug.cgi?id=652548
https://bugzilla.gnome.org/show_bug.cgi?id=345750

(In reply to QA Administrators from comment #18)

This is still an issue, no change supporting structural navigation by page top to bottom with current release or on master. NVDA screen reader exhausts buffer and stalls.

Version: 7.6.4.1 (X86_64) / LibreOffice Community
Build ID: e19e193f88cd6c0525a17fb7a176ed8e6a3e2aa1
CPU threads: 8; OS: Windows 10.0 Build 19045; UI render: Skia/Vulkan; VCL: win
Locale: en-US (en_US); UI: en-US
Calc: CL threaded

or

Version: 24.8.0.0.alpha0+ (X86_64) / LibreOffice Community
Build ID: 566bb271b8fe5882f24fef230e06c2af4ea12b33
CPU threads: 8; OS: Windows 10.0 Build 19045; UI render: Skia/Vulkan; VCL: win
Locale: en-US (en_US); UI: en-US
Calc: CL threaded