Bug 91739 - nvda can not read documents from top to bottom.
Summary: nvda can not read documents from top to bottom.
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: UI (show other bugs)
Version:
(earliest affected)
unspecified
Hardware: Other Windows (All)
: medium normal
Assignee: Michael Weghorn
URL:
Whiteboard: target:25.2.0
Keywords: accessibility
Depends on:
Blocks: a11y-Windows
  Show dependency treegraph
 
Reported: 2015-05-30 07:17 UTC by zahra
Modified: 2024-09-12 05:08 UTC (History)
9 users (show)

See Also:
Crash report or crash signature:


Attachments
one text to confirm the problem. (38.62 KB, text/richtext)
2015-05-30 07:40 UTC, zahra
Details
Another multi-page sample doc, from https://github.com/nvaccess/nvda/issues/4115 (31.27 KB, application/vnd.oasis.opendocument.text)
2024-08-21 16:01 UTC, Michael Weghorn
Details
Sample file from https://github.com/nvaccess/nvda/issues/8152#issuecomment-2342167620 (11.74 KB, application/vnd.oasis.opendocument.text)
2024-09-11 15:46 UTC, Michael Weghorn
Details

Note You need to log in before you can comment on or make changes to this bug.
Description zahra 2015-05-30 07:17:50 UTC
nvda cant read documents from top to bottom by incert down. 
when we use microsoft office nvda can read entire document when we press incert down. 
but in libreoffice and open office when we press incert down it reads only many lines and when we want to continue to read we should press incert down again and nvda reads only many lines and again and again. its very difficult specially in large documents and reading bookks. 
and also when we navigate in document word by word using control left and right to read previous and next word after ending sentences after . when we press control left and right it reads blank. in this case we should read charactor by charactor to understand if its realy a blank line or not.
Comment 1 zahra 2015-05-30 07:40:49 UTC
Created attachment 116166 [details]
one text to confirm the problem.
Comment 2 V Stuart Foote 2015-05-30 13:00:16 UTC
Setting new, but believe this remains an issue with NVDAs support for parsing IAccessible2 based content.

This has an open issue on NVDA support: http://community.nvda-project.org/ticket/4119 or with Symphony's early IA2 based support NVDA in ticket 910. But issues with SayAll with ODF and OpenOffice existed back to the JavaAccessBridge/JRE Java Accessibility era--see NVDA tickets 422, 96

@Jamie, any thoughts on implementing SayAll for LibreOffice/Apache OpenOffice or, should I just close this not our bug?

Stuart
Comment 3 James Teh 2015-05-30 23:11:36 UTC
Say all for Symphony (NVDA ticket #910) was fixed years ago. Any remaining issues were due to bugs in Symphony.

The problem with say all in OO/LO is due to the fact that they only expose text that is currently on screen. As a result, once NVDA say all gets to the last paragraph, it asks for the next paragraph and is told that there isn't one, so say all can't go any further. There are hacks to get around this (e.g. send control+downArrow or scroll in some other way when LO reports no more paragraphs), but this is fairly inelegant and potentially unreliable. Whether this is a bug is debatable, but it's certainly a problem for us and is not what happens in other implementations. In Symphony, you could get at paragraphs that weren't yet on screen (barring weird bugs), which is why say all worked in Symphony.

Note that you don't have to generate accessibles for every paragraph in the document ahead of time. You could just generate them on demand when the flowsTo relation is required.
Comment 4 V Stuart Foote 2015-05-31 02:06:48 UTC
@Jamie, *,

This gets beyond my skill set pretty quickly...

But, looking at i27138 - we already have an XAccessibleContext providing flowsFrom and flowsTo in the internal UAA. And that should be providing the IAccessible2 relationships.

But, seems that  for our Windows implementation--UAccCOM--we may be defining just the parent and child relationships, and not actually defining IA2_RELATION_FLOWS_FROM and IA2_RELATION_FLOWS_TO (see refs).

Anyone have a better feel for the this?

@Jamie,  if the IA2_RELATION_FLOWS_FROM and IA2_RELATION_FLOWS_TO are fully instrumented, would that then facilitate the NVDA SayAll function?

=-refs-=
Core
http://opengrok.libreoffice.org/xref/core/accessibility/source/extended/textwindowaccessibility.cxx#1447
http://opengrok.libreoffice.org/xref/core/sw/source/core/access/accpara.hxx#265
http://opengrok.libreoffice.org/xref/core/sw/source/core/access/accpara.cxx#959
http://opengrok.libreoffice.org/xref/core/editeng/source/accessibility/AccessibleEditableTextPara.cxx#813

Windows IA2 UAccCOM
http://opengrok.libreoffice.org/xref/core/winaccessibility/source/UAccCOMIDL/UAccCOM.idl
http://opengrok.libreoffice.org/xref/core/winaccessibility/source/UAccCOM/MAccessible.cxx
http://opengrok.libreoffice.org/xref/core/winaccessibility/source/UAccCOMIDL/ia2_api_all.idl#354
Comment 5 James Teh 2015-05-31 23:44:46 UTC
LO *does* expose flowsFrom and flowsTo via IA2. The issue isn't that they aren't exposed. The issue is that they don't expose paragraphs that are currently off-screen in some cases.

So, steps to reproduce with the attached example document:
1. Open the document. The cursor will be at the first line.
2. Get the accessible for the first paragraph.
3. Call IAccessible::nRelations.
Result (correct): 1
4. Call IAccessible2::relations, passing 1 for maxRelations.
5. Get the first (and only) IAccessibleRelation returned.
6. On the relation from step 5, call IAccessibleRelation::relationType.
Result (correct): "flowsTo"
7. On the relation from step 5, call IAccessibleRelation::nTargets.
Result (correct): 1
8. On the relation from step 5, call IAccessibleRelation::target, passing 0 for targetIndex.
Expected: An IUnknown for the second paragraph.
Actual: E_FAIL
9. In the document, press control+downArrow to move to the second paragraph.
10. On the relation from step 5, call IAccessibleRelation::target, passing 0 for targetIndex.
Result (correct): An IUnknown for the second paragraph.

Interestingly, if you then press control+home to move to the top of the document and repeat step 10, you still get the second paragraph. So, it seems that if you've retrieved the paragraph once already, you can retrieve it again even if it goes off-screen.
Comment 6 Joanmarie Diggs 2015-06-11 04:47:10 UTC
This issue also impacts things like Orca's ability to do structural navigation (navigate amongst headings, for instance).

As for the flows-to relations, things get more interesting depending on what you are trying to accomplish. Consider something like:

Page 1 (which is visible)
* Paragraph 1
* Paragraph 2
Page 2 (not currently visible)
* Paragraph 3
* Paragraph 4

At least on my platform, if you get the last child of the accessible document, you get "Paragraph 2". If you then get the flows-to target of that child you get "Paragraph 3". So far so good. But if you get the parent of "Paragraph 3" you get the accessible document which contains "Paragraph 1" and "Paragraph 2". In other words, the parent of "Paragraph 3" does not have "Paragraph 3" as a child. That's a broken tree.

I understand the desire to not automatically expose the entire tree for all pages at once because that potentially would be a huge and non-performant solution. But in addition to a non-broken tree, I'd really like to have structural navigation for document content in Orca. And not everything has an immediate flows-to (tables, form fields). So what about something like:

For each page in the document, create an accessible (ATK_ROLE_PAGE on my platform; dunno about IA2). Regardless of the role, these page accessibles are all immediate children of the document and are all created upon document load. Also load the accessible children for a given page when that page is accessed. Lazy load the rest of the children of each page.
Comment 7 V Stuart Foote 2015-06-11 13:28:50 UTC
(In reply to Joanmarie Diggs from comment #6)

> I understand the desire to not automatically expose the entire tree for all
> pages at once because that potentially would be a huge and non-performant
> solution. But in addition to a non-broken tree, I'd really like to have
> structural navigation for document content in Orca. And not everything has
> an immediate flows-to (tables, form fields). So what about something like:
> 
> For each page in the document, create an accessible (ATK_ROLE_PAGE on my
> platform; dunno about IA2). Regardless of the role, these page accessibles
> are all immediate children of the document and are all created upon document
> load. Also load the accessible children for a given page when that page is
> accessed. Lazy load the rest of the children of each page.

IAccessible2 has the same... IA2_ROLE_PAGE, but it looks like NSAccessibility does not currently map it in any useful fashion, refs.

=-refs-=
IA2_ROLE_PAGE
http://opengrok.libreoffice.org/xref/core/winaccessibility/source/UAccCOMIDL/ia2_api_all.idl#901
http://opengrok.libreoffice.org/xref/core/winaccessibility/source/service/AccObject.cxx

NSAccessibility
http://opengrok.libreoffice.org/xref/core/vcl/osx/a11yrolehelper.mm#119
https://developer.apple.com/library/mac/documentation/AppKit/Reference/NSAccessibility_Protocol_Reference/index.html#//apple_ref/doc/constant_group/Roles
Comment 8 Jacobo Aragunde Pérez 2015-12-03 14:24:04 UTC
The accessible role PAGE seems to exist in LibreOffice internal FW [1], but I've done some grepping and I think it's never used.

[1] http://api.libreoffice.org/docs/idl/ref/namespacecom_1_1sun_1_1star_1_1accessibility_1_1AccessibleRole.html#a29bb572b50678c1b43fb73f8e0855d65
Comment 9 Jacobo Aragunde Pérez 2015-12-04 16:39:18 UTC
(In reply to Joanmarie Diggs from comment #6)
> 
> For each page in the document, create an accessible (ATK_ROLE_PAGE on my
> platform; dunno about IA2). Regardless of the role, these page accessibles
> are all immediate children of the document and are all created upon document
> load. Also load the accessible children for a given page when that page is
> accessed. Lazy load the rest of the children of each page.

Unfortunately, this approach seems to be difficult to implement. There is no concept of "pages" in the document model; pages are an artifact of the view that wraps the contents. There are contents that can be split between two pages, so we would still have to do some hacks to model the contents in a proper tree. Finally, the flow of paragraphs to/from the next/previous page when editions happen would have to trigger a rebuild of parts of the accessibility tree, this should be doable but it's definitely inconvenient and would probably generate a lot of event noise.


> Page 1 (which is visible)
> * Paragraph 1
> * Paragraph 2
> Page 2 (not currently visible)
> * Paragraph 3
> * Paragraph 4
> 
> At least on my platform, if you get the last child of the accessible
> document, you get "Paragraph 2". If you then get the flows-to target of that
> child you get "Paragraph 3". So far so good. But if you get the parent of
> "Paragraph 3" you get the accessible document which contains "Paragraph 1"
> and "Paragraph 2". In other words, the parent of "Paragraph 3" does not have
> "Paragraph 3" as a child. That's a broken tree.

When the paragraph 3 is got through the flows-to relation, is the parent component (the document) refreshed? Would that work to implement "say all"?

> This issue also impacts things like Orca's ability to do structural
> navigation (navigate amongst headings, for instance).

Regarding this topic, would it help if we implemented flows to/from relations among header objects?
Comment 10 Robinson Tryon (qubit) 2015-12-10 03:37:52 UTC Comment hidden (obsolete)
Comment 11 QA Administrators 2017-01-03 19:40:19 UTC Comment hidden (obsolete)
Comment 12 V Stuart Foote 2017-01-03 19:46:38 UTC
This is still an issue, no change supporting structural navigation by page top to bottom with current release or on master. NVDA screen reader exhausts buffer and stalls.
Comment 13 V Stuart Foote 2017-10-11 19:10:49 UTC
Here are links to some related historical reading from the Gnome ATK/AT-SPI side on providing support for the Accessibility::Collection IDL (AtkDocument/AtkCollection) as a possible solution to our limited window into a complex document that disrupts screen reading.

https://bugzilla.gnome.org/show_bug.cgi?id=652548
https://bugzilla.gnome.org/show_bug.cgi?id=345750
Comment 14 QA Administrators 2018-10-12 03:05:11 UTC Comment hidden (obsolete)
Comment 15 V Stuart Foote 2018-10-12 12:32:54 UTC
(In reply to QA Administrators from comment #14)
This is still an issue, no change supporting structural navigation by page top to bottom with current release or on master. NVDA screen reader exhausts buffer and stalls.
Comment 16 QA Administrators 2019-10-13 02:34:23 UTC Comment hidden (obsolete)
Comment 17 V Stuart Foote 2019-10-13 13:32:13 UTC
(In reply to QA Administrators from comment #16)

This is still an issue, no change supporting structural navigation by page top to bottom with current release or on master. NVDA screen reader exhausts buffer and stalls.

Here are links to some related historical reading from the Gnome ATK/AT-SPI side on providing support for the Accessibility::Collection IDL (AtkDocument/AtkCollection) as a possible solution to our limited window into a complex document that disrupts screen reading.

https://bugzilla.gnome.org/show_bug.cgi?id=652548
https://bugzilla.gnome.org/show_bug.cgi?id=345750
Comment 18 QA Administrators 2023-10-15 03:16:20 UTC Comment hidden (obsolete)
Comment 19 V Stuart Foote 2024-01-26 20:02:05 UTC
(In reply to QA Administrators from comment #18)

This is still an issue, no change supporting structural navigation by page top to bottom with current release or on master. NVDA screen reader exhausts buffer and stalls.

Version: 7.6.4.1 (X86_64) / LibreOffice Community
Build ID: e19e193f88cd6c0525a17fb7a176ed8e6a3e2aa1
CPU threads: 8; OS: Windows 10.0 Build 19045; UI render: Skia/Vulkan; VCL: win
Locale: en-US (en_US); UI: en-US
Calc: CL threaded

or

Version: 24.8.0.0.alpha0+ (X86_64) / LibreOffice Community
Build ID: 566bb271b8fe5882f24fef230e06c2af4ea12b33
CPU threads: 8; OS: Windows 10.0 Build 19045; UI render: Skia/Vulkan; VCL: win
Locale: en-US (en_US); UI: en-US
Calc: CL threaded
Comment 20 Michael Weghorn 2024-08-21 16:01:46 UTC
Created attachment 195932 [details]
Another multi-page sample doc, from https://github.com/nvaccess/nvda/issues/4115
Comment 21 Michael Weghorn 2024-08-21 16:11:56 UTC
The problem about paragraphs not being available via the flows-to relationship already occurs for on-screen paragraphs in my tests with both sample docs, attachment 116166 [details] and attachment 195932 [details].

Further analysis:
https://github.com/nvaccess/nvda/issues/4115#issuecomment-2299084993

Pending fix:
https://gerrit.libreoffice.org/c/core/+/172215

With that in place, reading the whole docs from top to bottom using NVDA works for both of the sample docs, which is what this bug report is about.

Other aspects mentioned in comments of this bug report should be handled separately. There's already tdf#137955 ("Browse Mode in LibreOffice Writer"). If anything else is missing, please create separate bug reports.
Comment 22 Commit Notification 2024-08-21 18:13:08 UTC
Michael Weghorn committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/4ac1015b58a676324443de54188c16abf28d642f

tdf#91739 wina11y: Ensure IAccessible for relation target exists

It will be available in 25.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 23 Michael Weghorn 2024-08-22 12:51:41 UTC
The pending change series up to https://gerrit.libreoffice.org/c/core/+/172258 now also implements `IAccessible2_2::get_relationTargetsOfType`, NVDA's preferred way to retrieve the next paragraph via the flowsTo relationship, so this also works without the fallback code path in NVDA (s. commit msg for details).
Comment 24 Commit Notification 2024-08-23 05:40:29 UTC
Michael Weghorn committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/002a29b8dd0d4c61194227e4151b4847480cb8b1

tdf#91739 wina11y: Add skeleton IAccessible2_2 implementation

It will be available in 25.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 25 Commit Notification 2024-08-23 05:40:32 UTC
Michael Weghorn committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/2318b823732df3820557943e56dd5243381dc558

tdf#91739 wina11y: Implement IAccessible2_2::get_relationTargetsOfType

It will be available in 25.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 26 Michael Weghorn 2024-09-11 15:46:06 UTC
Created attachment 196385 [details]
Sample file from https://github.com/nvaccess/nvda/issues/8152#issuecomment-2342167620

Attached sample file from the referenced other NVDA issue currently triggers an assert in a debug build when starting SayAll right away, i.e. without moving through the document first. I have a local fix that I'll submit to Gerrit.
Comment 27 Michael Meeks 2024-09-11 16:01:28 UTC
Nice work Michael ! =)
Comment 28 Michael Weghorn 2024-09-11 16:13:33 UTC
(In reply to Michael Meeks from comment #27)
> Nice work Michael ! =)

Thanks, Michael! :)


(In reply to Michael Weghorn from comment #26)
> Created attachment 196385 [details]
> Sample file from
> https://github.com/nvaccess/nvda/issues/8152#issuecomment-2342167620
> 
> Attached sample file from the referenced other NVDA issue currently triggers
> an assert in a debug build when starting SayAll right away, i.e. without
> moving through the document first. I have a local fix that I'll submit to
> Gerrit.

-> Pending fix: https://gerrit.libreoffice.org/c/core/+/173224
Comment 29 Commit Notification 2024-09-12 05:08:17 UTC
Michael Weghorn committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/071b626079243908cfa85a5bd49620235fdf48cf

tdf#91739 wina11y: Create AccObject for parent as necessary

It will be available in 25.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.